Category Archives: OpenStack Design Summit

Ballistic Release Cycles: Tracking the Trajectory of OpenStack Milestones

Posted on by
9

I’ve been watching a pattern emerge on the semiannual OpenStack release cycles for a while now. There is a hidden but crucial development phase that accelerates projects faster than many observers realize. In fact, I believe that substantial work is happening outside of the “normal” design cycle during what I call “free fall” development.

Understanding when the cool, innovative stuff happens is essential to getting (and giving) the most from OpenStack.

The published release cycle looms like a 6 stage ballistic trajectory. Launching at the design summit, the release features change and progress the most in the first 3 milestones. At the apogee of the release, maximum velocity is reached just as we start having to decide which features are complete enough to include in the release. Since many are not ready, we have to jettison (really, defer) partial work to ensure that we can land the release on schedule.

I think of the period where we lose potential features as free fall because thing can go in any direction. The release literally reverses course: instead of expanding, it is contracting. This process is very healthy for OpenStack. It favors code stability and “long” hardening times. For operators, this means that the code stops changing early enough that we have more time to test and operationalize the release.

But what happens to the jettisoned work? In free fall, objects in motion stay in motion. The code does not just disappear! It continues on its original upward trajectory.

The developers who invested time in the code do not simply take a 3 month sabbatical, nor do they stop their work and start testing the code that was kept. No, after the short in/out sorting pause, the free fall work continues onward with rockets blasting. The challenge is that it is now getting outside of the orbit of the release plan and beyond the radar of many people who are tracking the release.

The consequence of this ongoing development is that developers (and the features they are working on) show up at the summit with 3 extra months of work completed. It also means that OpenStack starts each release cycle with a bucket of operationally ready code. Wow, that’s a huge advantage for the project in terms of delivered work, feature velocity and innovation. Even better, it means that the design summit can focus on practical discussions of real prototypes and functional features.

Unfortunately, this free fall work has hidden costs:

  • It is relatively hidden because it is outside of the normal release cycle.
  • It makes true design discussions less productive because the implemented code is more likely to make the next release cycle
  • Integration for the work is postponed because it continues before branching
  • Teams that are busy hardening a core feature can be left out of work on the next iteration of the same feature
  • Forking can make it hard to capture bugs caught during hardening

I think OpenStack greatly benefits from free fall development; consequently, I think we need to acknowledge and embrace it to reduce its costs. A more explicit mid-release design synchronization when or before we fork may help make this hidden work more transparent.

I am seeking your vote(s) for the OpenStack Board

Posted on by
4

If registered, you have 8 votes to allocate as you wish.  You will get a link via email – you must use that link.

Joseph B George and I are cross-blogging this post because we are jointly seeking your vote(s) for individual member seats on the OpenStack Foundation board.  This is key point in the OpenStack journey and we strongly encourage eligible voters to participate no matter who you vote for!  As we have said before, success of the Foundation governance process matters just as much as the code because it ensures equal access and limits forking.

We think that OpenStack succeeds because it is collaboratively developed.  It is essential that we select board members who have a proven record of community development, a willingness to partner and have demonstrated investment in the project.

Our OpenStack vision favors production operations by being operator, user and ecosystem focused.  If elected, we will represent these interests by helping advance deployability, API specifications, open operations and both large and small scale cloud deployments.

Of the nominees, we best represent OpenStack users and operators (as opposed to developers).  We have the most diverse experience in real-world OpenStack deployments because our solution has been deployed broadly (both as Dell and through Crowbar.  We have a proven record of collaborating broadly with contributors, demonstrated skills at building the OpenStack community and doing real open source work to ensure that OpenStack is the most deployable cloud platform anywhere.

Let’s get specific about our leadership in the OpenStack project and community:

  • We have been active and vocal leaders in the OpenStack community
    • our team has established two very active user groups (Austin & Boston)
    • we have lead multiple world-wide deploy day events (March 2012  &  May 2012).
    • we have substantial experience in the field and know the challenges of running OpenStack for a wide variety of real-world deployments
    • our first solution came out on Cactus!  We’ve been delivering on Essex since OSCON 2012 (http://www.oscon.com/ ).
  • We represent a broad range of deployment scenarios ranging from hosting, government, healthcare, retail, education, media, financial and more!
  • We have broad engagements and partnerships at the infrastructure (SUSE, Canonical, Redhat), consulting (Canonical, Mirantis) and ecosystem layers (enStratus) and beyond!
  • We have a proven track record of collaboration instead of forking/disrupting – a critical skill for this project reflected by our consistent actions to preserve the integrity of the project.
  • We have led the “make OpenStack deployable” campaign with substantial investments (open source Crowbar, white papers, documentation & cookbooks.
  • We have very long and consistent history with the project starting even before the first OpenStack summit in Austin.

Of course, we’re asking for you to consider for both of us; however, if you want to focus on just one then here’s the balance between us.  Rob (bio) is a technologist with deep roots in cloud technology, data center operations and open source.  Joseph is a business professional with experience new product introduction and enterprise delivery.

Not sure if you can vote?  If you registered as an individual member then your name should be on the voting list.  In that case, you can vote between 8/20 and 8/24.

Four OpenStack Trends from Summit: Practical, Friendly, Effective and Deployable

Posted on by
1

With the next OpenStack Austin meetup on Thursday (sponsored by Puppet), I felt like it was past time for me to post my thoughts and observations about the Spring 2012 OpenStack design conference.  This was my fifth OpenStack conference (my notes about Bexar, Cactus, Diablo & Essex).  Every conference has been unique, exciting, and bigger than the previous.

My interest lies in the trend lines of OpenStack.  For details about sessions, I recommend Stefano Maffulli‘s  excellent link aggregation post for the Summit.

1. Technology Trend: Practical with Potential.

OpenStack started with a BIG vision to become the common platform for cloud API and operations.  That vision is very much alive and on-track; however, our enthusiasm for what could be is tempered by the need to build a rock solid foundation.  The drive to stability over feature expansion has had a very positive impact.  I give a lot of credit for this effort to the leadership of the project technical leads (PTLs), Canonical‘s drive to include OpenStack in the 12.04 LTS and the Rackspace Cloud drive to deploy Essex.  My team at Dell has also been part of this trend by focusing so much effort on making OpenStack production deployable (via Crowbar).

Overall, I am seeing a broad-based drive to minimize disruption.

2. Culture Trend: Friendly but some tension.

Companies at both large and small ends of the spectrum are clearly jockeying for position.  I think the market is big enough for everyone; however, we are also bumping into each other.  Overall, we are putting aside these real and imagined differences to focus on enlarging the opportunity of having a true community cloud platform.  For example, the OpenStack Foundation investment formation has moneyed competitors jostling for position to partner together.

However, it’s not just about paying into the club; OpenStack’s history is clearly about execution.  Looking back to the original Austin Summit sponsors, we’ve clearly seen that intent and commitment are different.

3. Discussion Trend: Small Groups Effective

The depth & quality of discussions inside sessions was highly variable.  Generally, I saw that large group discussions stayed at a very high level.  The smaller sessions required deep knowledge of the code to participate and seemed more productive.  We continue to have a juggle between discussions that are conceptual or require detailed knowledge of the code.  If conceptual, it’s too far removed.  If code, it becomes inaccessible to many people.

This has happened at each Summit and I now accept that it is natural.  We are using vision sessions to ensure consensus and working sessions to coordinate deliverables for the release.

I cannot over emphasize importance of small groups and delivery driven execution interactions: I spent most of my time in small group discussions with partners aligning efforts.

4. Deployment Trend: Testing and Upstreams matter

Operations for deploying OpenStack is a substantial topic at the Summit.  I find that to be a significant benefit to the community because there are a large block of us who were vocal advocates for deployability at the very formation of the project.

From my perspective at Dell, we are proud to see that wide spread acknowledgement of our open source contribution, Crowbar, as the most prominent OpenStack deployer.   Our efforts at making OpenStack installable are recognized as a contribution; however, we’re also getting feedback that we need to streamline and simplify Crowbar.  We also surprised to hear that Crowbar is “opinionated.”   On reflection, I agree (and am proud) of this assessment because it matches best practice coding styles.  Since our opinions also drive our test matrix there is a significant value for our OpenStack deployment is that we spend a lot of time testing (automated and manual) our preferred install process.

There’s a push to reconcile the various Chef OpenStack cookbooks into a single upstream.  This seems like a very good idea because it will allow various parties to collaborate on open operations.  The community needs leadership from Opscode to make this happen.  It appears that Puppet Labs is interested in playing a similar role for Puppet modules but these are still emerging and have not had a chance to fragment.

No matter which path we take, the deployment scripts are only as good as their level of testing.   Unreliable deployment scripts have are less than worthless.

Did Austin Stackers get what we wanted at the OpenStack Design Summit?

Posted on by
2

image

This post is a follow-up from the April 12 Austin OpenStack  (OSTAX) meeting.

Overall, we had a good meeting with strong attendance.  Unlike last meeting, the attendees were less OpenStack experienced; however, many us worked for companies that are members of the OpenStack Foundation.  I work for Dell (a gold sponsor).

Rather than posting before the summit, I’ve scored my summit experiences against our poll to see if our priorities were met.  (note: Thanks to Greg Althaus for additional input in the commentary)

Issue OSTAX Rank Results from Summit Outlook
Stability vs. Features Prioritization & Processes 68% This was a major thread throughout the summit in multiple sessions.   My feeling of the dialog was the stability (including continuous integration) was a core requirement. Excellent
API vs Code. What does it mean to be “OpenStack” 68% This is a good news / bad news story.  As OpenStack Compute gets split into more and more independent pieces; their interactions will require a well-defined externalized API.  The continuing issue is that these APIs will be still driven by the python-based reference implementation.  In some regards, APIs will emerge and be better codified.  Newer PTLs bring additional perspective and beliefs around APIs vs Code. Mixed
Operations focus: making OpenStack easy to deploy and manage 68% This was a major topic with many sessions dedicated to operationalizing OpenStack.  Special focus was given to shared Puppet and Chef deployment code.There were specific sessions around High Availability and what that means.  From this session, consensus was built for infrastructure HA documentation using Pacemaker for Folsom.  There was NOT consensus for instance-level HA. Trending Positive
Documentation Standards and improved user guides 59% Anne Gentle is championing this and had a presence throughout the summit. Strong
Driving for Hypervisor feature parity (KVM, Xen and also VMware/HyperV) 57% While Libvirt/KVM continues to dominate.  Citrix was present to support XenServer and Microsoft made commitments for (returning) HyperV support. Uneven Progress
Improving collaboration (get beyond listserv & IRC) so information is more persistent 56% I was not involved in discussions around this topic. No Comment
Have more operations discussion / design at the Design Summit 54% We had many sessions about operations tooling but little about specific considerations for operations.  Perhaps we need to take a step towards shared deployment scripts. Action with Fragmentation
Nova-volume to split out and/or more API driven (less integrated) 51% This was a major topic in multiple sessions.  There are a number of parties that are signing up to create block storage as a stand-alone project.Cinder will be the block storage service.  Not just good sessions were held, but good plans were built for constructing and improving the project.  The project will start as a clone of the current nova project with unique chunks living in Cinder and common pieces of both projects move to the openstack-common project.The Cinder working group is very cross company and had a strong desire to maintain a minimal specification (current API replacement) with only one additional feature required for Folsom (boot from volume).  The boot from volume feature is really a Nova feature, but the Cinder team will most likely drive it to ensure Cinder/Nova separation. Surprisingly Active
OpenStack on Windows & HyperV 50% This is two topics.  Microsoft is committing for OpenStack to support HyperV as a Nova Compute node.  Running the rest of the suite on Windows does not appear to be a priority (or practical?) Promising Potential
Orchestration. More projects like Donabe? 48% There are a number of ecosystem projects emerging.  Now that Essex has emerged as a solid release, I expect to see an acceleration projects.  At this time, they are still incubating.There was also the acknowledgement that there are two levels of orchestration, instance orchestration (think nova scheduler) and workload orchestration (think Donabe or VAPP).  Instance orchestration had many good discussions and improvements suggested and started (host aggregates, filter scheduler extensions, …) Building Slowly
Making Nova into smaller components 46% This was a thread in several sessions and it part of the ongoing stabilization work to improve collaboration.  One important component of this is moving common code into a shared library. In process, needs focus
How should invitations be handed out to Summit? Was the last process to Dev focused? 40% I was not aware of any discussion of this at the summit.  Looks like we all need to go out and commit some code! No Comment

Overall, I think that the Austin Stacker priorities were well positioned at the Design Summit.

After the split, I’m posted the twitter feed from the meeting (in post  order):

Continue reading →

Dell Team at the OpenStack Spring 2012 Summit

Posted on by
Reply

It’s OpenStack Summit time again for my team at Dell and there’s deployment in the air. It’s been an amazing journey from the first Austin summit to Folsom today. Since those first heady days, the party has gotten a lot more crowded, founding members have faded away, recruiters became enriched as employees changed email TLDs and buckets of code was delivered.

Throughout, Dell has stayed the course: our focus from day-one has been ensuring OpenStack can be deployed into production in a way that was true to the OpenStack mission of community collaboration and Apache-2-licensed open source.

We’ve delivered on the making OpenStack deployable vision by collaborating broadly on the OpenStack components of the open source Crowbar project. I believe that our vision for sustainable open operations based on DevOps principles is the most complete strategy for production cloud deployments.

We are at the Folsom Summit in force and we’re looking forward to discussions with the OpenStack community. Here are some of the ways to engage with us:

  • Demos
    • During the summit (M-W), we’ll have our Crowbar OpenStack Essex deployments running. We kicked off Essex development with a world-wide event in early March and we want more people to come and join in.
    • During the conference (W-F), we’ll be showing off application deployments using enStratus and Chef against our field proven Diablo release.
  • Speakers
    • Thursday 1:00pm, OpenStack Gains Momentum: Customers are Speaking Up by Kamesh Pemmaraju (Dell)
    • Friday 9:50am, Deploy Apps on OpenStack using Dashboard, Chef and enStratus by Rob Hirschfeld (Dell), Matt Ray (Opscode) and Keith Hudgins (enStratus).
    • Friday 11:30am, Expanding the Community Panel
      including Joseph George (Dell)
    • This fun round trip road trip from Rackspace & Dell HQs in Austin to the summit and home again promises to be an odyssey of inclusion. Dell OpenStack/Crowbar engineer Andi Abes (@a_abes). Follow @RoadstackRV to follow along as they return home and share their thoughts about the summit!
  • Parties
    • Monday 6pm Mirantis Welcome Party, co-sponsored with Dell, at Sens Restaurant (RSVP)
    • Tuesday 5pm “Demos & Drinks” Happy Hour, co-hosted by Dell, Mirantis, Morphlabs, Canonical at the Hyatt Regency Hospitality Room off the Atrium

My team has been in the field talking to customers and doing OpenStack deployments. We are proud to talk about it and our approach.

Mostly importantly, we want to collaborate with you on our Essex deployments using Crowbar.  Get on our list, download/build crowbar, run the “essex-hack” branch and start banging on the deploy.  Let’s work together to make this one rock solid Essex deploy.

OpenStack Meetup 4/12: Austin at Summit, DevStack Essex

Posted on by
Reply

Austin Stackers!  This Thursday is our April meetup at the Austin TechRanch.

Please RSVP so that we know how much food to get!  SUSE is this Month’s sponsor for food and my team at Dell continues to pickup the room rental.  We have 35 RSVPs as of Monday noon – this will be another popular meeting (last meeting minutes).

Topics for the meetup are:

With the Summit next week, I think it is very important that we pre-discuss Summit topics and priorities as a community.  It will help us be more productive individually and for our collective interests when we engage the larger community next week.

Open Source Cloud Bootstrapping Revisised

Posted on by
Reply

At the OpenStack last design conference, Greg Althaus and I presented about updates (presentation here) we were making to a Nov 2010 cloud architecture white paper.

The revised “Bootstrapping Open Source Clouds” white paper has been out for a few months so I thought it was past time to throw out a link.

I’m really pleased about this update because it reflects real world experience my team has working with customers and partners on OpenStack (and Hadoop) deployments.

Executive Summary
Bringing a cloud infrastructure online can be a daunting bootstrapping challenge. Before
hanging out a shingle as a private or public cloud service provider, you must select a platform,
acquire hardware, configure your network, set up operations services, and integrate it to work
well together. That is a lot of moving parts before you have even installed a sellable application.
This white paper walks you through the decision process to get started with an open source
cloud infrastructure based on OpenStack™ and Dell™ PowerEdge™ C servers. At the end, you’ll
be ready to design your own trial system that will serve as the foundation of your hyperscale
cloud.
2011 Revision Notes
In the year since the the original publication of this white paper, we worked with many
customers building OpenStack clouds. These clouds range in size from small six-node lab
systems to larger production deployments. Based on these experiences, we updated this white
paper to reflect lessons learned.

OpenStack Austin: What we’d like to see at the Design Summit

Posted on by
2

Last week, the OpenStack Austin user group discussed what we’d like to see at the upcoming OpenStack Design Summit. We had a strong turnout (48?!).

  1. To get the meeting started, Marc Padovani from HP (this month’s sponsor) provided some lessons learned from the HP OpenStack-Powered Cloud. While Marc noted that HP has not been able to share much of their development work on OpenStack; he was able to show performance metrics relating to a fix that HP contributed back to the OpenStack community. The defect related to the scheduler’s ability to handle load. The pre-fix data showed a climb and then a gap where the scheduler simply stopped responding. Post-fix, the performance curve is flat without any “dead zones.” (sharing data like this is what I call “open operations“)
  2. Next, I (Rob Hirschfeld) gave a brief overview of the OpenStack Essex Deploy Day (my summary) that Dell coordinated with world-wide participation. The Austin deploy day location was in the same room as the meetup so several of the OSEDD participants were still around.
  3. The meat of the meetup was a freeform discussion about what the group would like to see discussed at the Design Summit. My objective for the discussion was that the Austin OpenStack community could have a broader voice is we showed consensus for certain topics in advance of the meeting.

At Jim Plamondon‘s suggestion, we captured our brain storming on the OpenStack etherpad. The Etherpad is super cool – it allows simultaneous editing by multiple parties, so the notes below were crowd sourced during the meeting as we discussed topics that we’d like to see highlighted at the conference. The etherpad preserves editors, but I removed the highlights for clarity.

The next step is for me to consolidate the list into a voting page and ask the membership to rank the items (poll online!) below.

Brain storm results (unedited)

Stablity vs. Features

API vs. Code

  • What is the measurable feature set?
  • Is it an API, or an implementation?
  • Is the Foundation a formal-ish standards body?
  • Imagine the late end-game: can Azure/VMWare adopt OPenStack’s APIs and data formats to deliver interop, without running OpenStack’s code? Is this good? Are there conversations on displacing incumbents and spurring new adoption?
  • Logo issues

Documentation Standards

  • Dev docs vs user docs
  • Lag of update/fragmentation (10 blogs, 10 different methods, 2 “work”)
  • Per release getting started guide validated and available prior or at release.

Operations Focus

  • Error messages and codes vs python stack traces
  • Alternatively put, “how can we make error messages more ops-friendly, without making them less developer-friendly?”
  • Upgrade and operations of rolling updates and upgrades. Hot migrations?

If OpenStack was installable on Windows/Hyper-V as a simple MSI/Service installer – would you try it as a node?

  • Yes.

Is Nova too big?  How does it get fixed?

  • libraries?
  • sections?
  • make it smaller sub-projects
  • shorter release cycles?

nova-volume

  • volume split out?
  • volume expansion of backend storage systems
  • Is nova-volume the canonical control plane for storage provisioning?  Regardless of transport? It presently deals in block devices only… is the following blueprint correctly targeted to nova-volume?

https://blueprints.launchpad.net/nova/+spec/filedriver

Orchestration

  • Is the Donabe project dead?

Discussion about invitations to Summit

  • What is a contribution that warrants an invitation
  • Look at Launchpad’s Karma system, which confers karma for many different “contributory” acts, including bug fixes and doc fixes, in addition to code commitments

Summit Discussions

  • Is there a time for an operations summit?
  • How about an operators’ track?
  • Just a note: forums.openstack.org for users/operators to drive/show need and participation.

How can we capture the implicit knowledge (of mailing list and IRC content) in explicit content (documentation, forums, wiki, stackexchange, etc.)?

Hypervisors: room for discussion?

  • Do we want hypervisor featrure parity?
  • From the cloud-app developer’s perspective, I want to “write once, run anywhere,” and if hypervisor features preclude that (by having incompatible VM images, foe example)
  • (RobH: But “write once, run anywhere” [WORA] didn’t work for Java, right?)
  • (JimP: Yeah, but I was one of Microsoft’s anti-Java evangelists, when we were actively preventing it from working — so I know the dirty tricks vendors can use to hurt WORA in OpenStack, and how to prevent those trick from working.)

CDMI

Swift API is an evolving de facto open alternative to S3… CDMI is SNIA standards track.  Should Swift API become CDMI compliant?  Should CDMI exist as a shim… a la the S3 stuff.

OpenStack Essex Events (Austin & Boston 3/8, WW Hack Day 3/1, Docs 3/6)

Posted on by
1

The excitement over the OpenStack Essex release is building!  While my team has been making plans around the upcoming design summit in SF,  there is more immediate action afoot.

Tomorrow (3/1), numerous sites are gathering around a World Wide Essex Hack Day on 3/1.  If you want to participate or even host a hack venue, get on the list and IRC channel (details).

My team at Dell is organizing a community a follow-up OpenStack Essex Install Day next week (3/8) in both Austin and Boston.  Just like the Hack Day, the install fest will focus on Essex release code with both online and local presence.  Unlike the Hack Day, our focus will be on deployments.  For the Dell team, that means working on the Essex deployment for Crowbar.  We’re still working on a schedule and partner list so stay tuned.  I’m trying to webcast Crowbar & OpenStack training sessions during the install day.

The hack day will close with the regularly scheduled 3/8 OpenStack Austin Meetup (6:30pm at Austin TechRanch).  The topic for the meetup will be, …. wait for it …., the Essex Release.  Thanks go to HP and Dell for sponsoring!

It’s important to note that Anne Gentle is also coordinating an OpenStack Essex Doc Day on 3/6.

To recap:

Wow… that should satisfy your Essex cravings.

Austin OpenStack Meetup (January Minutes) + OpenStack Foundation Web Cast!

Posted on by
2

Sorry for the brevity… At the last Austin OpenStack meetup, we had >60 stackers!  Some from as far away as Portland and Boston (as in Oregon and Massachusetts).

Notes:

  • Suse introduced their OpenStack beta and talked about their Suse Studio that can deploy images against the OpenStack APIs
  • I showed off DevStack.org code that can setup the truck of OpenStack (now Essex) in about 10 minutes on a single node.  Great for developers!
  • I showed an OpenStack Diablo Final deployment from Crowbar.  I focused mainly on Dashboard and used our reference architecture (see below) as illustration of the many parts.
  • Matt Ray suggested everyone watch the webcasts about the OpenStack Foundation (Thurs 6pm central  & Friday 9am central)
  • We planned the next few meetups.
    • For February, we’ll talk about Swift and Dashboard.
    • For March, we’ll talk about Essex and DevStack to prep for the next design summit (in SF).
    • For April, we’ll debrief the conference

Thank you Suse and Dell (my employer) for sponsoring!   The next meetup is sponsored by Canonical.