Category Archives: OpenStack

Crowbar releases for OpenStack Essex and Folsom

Posted on by
6

On the eve of the OpenStack design summit, it’s worth noting that the Crowbar team at Dell cut our final Essex release (aka Betty) last week. We’ve also committed the initial Folsom deployment scripts to the 1x development trunk under “feature/folsom” if you are doing Crowbar builds from DevTool (see bit.ly/crowbardevtool).

If you don’t want to build it, I’ve cut ISOs from the open source bits and posted them to http://crowbar.zehicle.com.

Andi Abes is presenting about the new Pull From Source (pfs) feature at the Summit on Monday. There’s a feature branch for that too and I’m going to check with him and try to post and ISO for that too.

20121014-132246.jpg

OpenStack Summit: Let’s talk DevOps, Fog, Upgrades, Crowbar & Dell

Posted on by
7

If you are coming to the OpenStack summit in San Diego next week then please find me at the show! I want to hear from you about the Foundation, community, OpenStack deployments, Crowbar and anything else.  Oh, and I just ordered a handful of Crowbar stickers if you wanted some CB bling.

Matt Ray (Opscode), Jason Cannavale (Rackspace) and I were Ops track co-chairs. If you have suggestions, we want to hear. We managed to get great speakers and also some interesting sessions like DevOps panel and up streaming deploy working sessions. It’s only on Monday and Tuesday, so don’t snooze or you’ll miss it.

My team from Dell has a lot going on, so there are lots of chances to connect with us:

At the Dell booth, Randy Perryman will be sharing field experience about hardware choices. We’ve got a lot of OpenStack battle experience and we want to compare notes with you.

I’m on the board meeting on Monday so likely occupied until the Mirantis party.

See you in San Diego!

PS: My team is hiring for Dev, QA and Marketing. Let me know if you want details.

Continuous Release combats disruptions of “Free Fall” development

Posted on by
Reply

Since I posted the “Free Fall” development post, I’ve been thinking a bit about the pros and cons of this type of off-release development.

The OpenStack Swift project does not do free fall because they are on a constant “ship ready” state for the project and only loosely flow the broader OpenStack release track.  My team at Dell also has minimal free fall development because we have a more frequent release clock and choose to have the team focus together through dev/integrate/harden cycles as much as possible.

From a Lean/Agile/CI perspective, I would work to avoid hidden development where possible.  New features are introduced by split test (they are in the code, but not active for most users) so that the all changes in incremental.  That means that refactoring, rearchitecture and new capabilities appear less disruptively.  While it may this approach appears to take more effort in the short term; my experience is that it accelerates delivery because we are less likely to over develop code.

Unfortunately, free fall development has the opposite effect.  Having code that appears in big blocks is contrary to best practices in my opinion.  Further, it rewards groups that work asynchronously and

While I think that OpenStack benefits from free fall work, I think that it is ultimately counter-productive.

Ballistic Release Cycles: Tracking the Trajectory of OpenStack Milestones

Posted on by
9

I’ve been watching a pattern emerge on the semiannual OpenStack release cycles for a while now. There is a hidden but crucial development phase that accelerates projects faster than many observers realize. In fact, I believe that substantial work is happening outside of the “normal” design cycle during what I call “free fall” development.

Understanding when the cool, innovative stuff happens is essential to getting (and giving) the most from OpenStack.

The published release cycle looms like a 6 stage ballistic trajectory. Launching at the design summit, the release features change and progress the most in the first 3 milestones. At the apogee of the release, maximum velocity is reached just as we start having to decide which features are complete enough to include in the release. Since many are not ready, we have to jettison (really, defer) partial work to ensure that we can land the release on schedule.

I think of the period where we lose potential features as free fall because thing can go in any direction. The release literally reverses course: instead of expanding, it is contracting. This process is very healthy for OpenStack. It favors code stability and “long” hardening times. For operators, this means that the code stops changing early enough that we have more time to test and operationalize the release.

But what happens to the jettisoned work? In free fall, objects in motion stay in motion. The code does not just disappear! It continues on its original upward trajectory.

The developers who invested time in the code do not simply take a 3 month sabbatical, nor do they stop their work and start testing the code that was kept. No, after the short in/out sorting pause, the free fall work continues onward with rockets blasting. The challenge is that it is now getting outside of the orbit of the release plan and beyond the radar of many people who are tracking the release.

The consequence of this ongoing development is that developers (and the features they are working on) show up at the summit with 3 extra months of work completed. It also means that OpenStack starts each release cycle with a bucket of operationally ready code. Wow, that’s a huge advantage for the project in terms of delivered work, feature velocity and innovation. Even better, it means that the design summit can focus on practical discussions of real prototypes and functional features.

Unfortunately, this free fall work has hidden costs:

  • It is relatively hidden because it is outside of the normal release cycle.
  • It makes true design discussions less productive because the implemented code is more likely to make the next release cycle
  • Integration for the work is postponed because it continues before branching
  • Teams that are busy hardening a core feature can be left out of work on the next iteration of the same feature
  • Forking can make it hard to capture bugs caught during hardening

I think OpenStack greatly benefits from free fall development; consequently, I think we need to acknowledge and embrace it to reduce its costs. A more explicit mid-release design synchronization when or before we fork may help make this hidden work more transparent.

OSCBM Seeking Community Input for Long Board Meetings and Candlelight Coding Sessions

Posted on by
1

Happy OpenStack Foundation Launch Day! I’m a little breathless at OpenStack’s sponsored sprint to foundationhood but very proud to be part of the process (you can be too!).   Just looking at the numbers it’s clear that we’re building something important.

While it’s important that OpenStack is innovative, stable and useful cloud infrastructure, it’s equally important the project in collaborative developed.

Collaborative development makes it safe for so many diverse commercial interests to participate.  The Foundation, with a gold and platinum war chest,  is a reflection of the need for the project to remain both openly collaborative and commercially successful for our community.  We must ensure a level technology playing field while we work to ensure members of the community can be commercially successful while contributing.  This balance is one of our core challenges.

As an OpenStack Community Board Member (OSCBM), I want to hear what you think the OpenStack Foundation should be doing for OpenStack!

It is vital that I get input from the OpenStack community!  Unlike 2/3 of the Board, my seat is decided by the community and (re)elected by the community on an annual basis; consequently, it is my responsibility to voice Stackers’ interest, not my employer’s (Dell).

Frankly, the project is still hugely dominated by developers with users/operators only just gaining influence. The Foundation’s primary purpose is to help safe guard its independence. As a board member, my job today is to oversee building  up the critical infrastructure (like having a staff) to perform that mission.

Of course, you also need to know my priorities.

  1. Consensus governance helps ensure minority views get heard while we still act as a unified body.  Consensus includes formalized agendas, rules (aka Robert’s rules) , clarifying motions and simple actions to make it easier for the community to follow.
  2. More community integration in the form of work done in subcomittees that can bring in external voices and integration with technical and user committees.
  3. Make activities more consistent, visible and accessible.  While our actions are open, our practices (and audio bridges)  make it difficult for the community to follow along.  That includes faster turn around on minutes so that board actions are not subject to twitter extrapolation.

The board is still very young and I’m impressed with what we’ve accomplished so far.

OpenStack Board needs Consensus Governance

Posted on by
6

I am humbled by the community support for my election (I finished first in the results) and have been surprised to realize that one of my unlisted credentials, 5 years as Secretary of our local public Community Development Corporation, could also be an important asset. Dealing with Texas Open Government laws around parliamentary minutia such as open discussions, voting, minutes and agendas turns out to translate directly to open source governance (which affects everyone!).

I believe that the OpenStack Board should operate by Consensus rules.

Boards can choose to operate either by Consensus or Majority. A Consensus board typically passes all resolutions unanimously while a Majority board does not need agreement on decisions (see table below).

On first blush, majority process seems more efficient; unfortunately, split votes are divisive and polarizing. The consequence of split votes is the minority positions will seek longer debate, resort to back room politics and procedural overhead. This type of behavior would be destructive for our community.

A Consensus board, which only happens by implied agreement of the members and leadership, works to ensure that decisions can be supported by all the members. This does not mean that all the members agree with the board positions, hold-hands during meetings, or participate in Polynesian drum circles! It does mean that the board as a whole considers minority positions and their motivation before calling a vote. If there is too much difference in opinion, then the majority may defer voting or minority members may abstain from voting. One common aspect of Consensus boards is that members may appear to argue against their own positions to ensure that minority views have been represented.

While the consensus model takes discipline for our Directors, it also takes patience and cooperation from the community that we serve. Board actions may take longer or be less direct than members of the community desire.

I believe that committing to a Consensus board is essential for OpenStack because our board is large (24 members!), our community is diverse and the financial impacts to members are high. So far, I’m proud that we’ve been following that model and will try to ensure we maintain that tradition.

Post Script Table: Consensus vs. Majority Governance Snapshot

Consensus Majority
Voting Unanimous Split
Process Flexible Strict
Position in Discussions Ambiguous Polarized
Controversy Avoided / Postponed Forced / Decisive Wins
Community Encouraged Divided
Minority Interests Incorporated Excluded
Board Unity High Low

I am seeking your vote(s) for the OpenStack Board

Posted on by
4

If registered, you have 8 votes to allocate as you wish.  You will get a link via email – you must use that link.

Joseph B George and I are cross-blogging this post because we are jointly seeking your vote(s) for individual member seats on the OpenStack Foundation board.  This is key point in the OpenStack journey and we strongly encourage eligible voters to participate no matter who you vote for!  As we have said before, success of the Foundation governance process matters just as much as the code because it ensures equal access and limits forking.

We think that OpenStack succeeds because it is collaboratively developed.  It is essential that we select board members who have a proven record of community development, a willingness to partner and have demonstrated investment in the project.

Our OpenStack vision favors production operations by being operator, user and ecosystem focused.  If elected, we will represent these interests by helping advance deployability, API specifications, open operations and both large and small scale cloud deployments.

Of the nominees, we best represent OpenStack users and operators (as opposed to developers).  We have the most diverse experience in real-world OpenStack deployments because our solution has been deployed broadly (both as Dell and through Crowbar.  We have a proven record of collaborating broadly with contributors, demonstrated skills at building the OpenStack community and doing real open source work to ensure that OpenStack is the most deployable cloud platform anywhere.

Let’s get specific about our leadership in the OpenStack project and community:

  • We have been active and vocal leaders in the OpenStack community
    • our team has established two very active user groups (Austin & Boston)
    • we have lead multiple world-wide deploy day events (March 2012  &  May 2012).
    • we have substantial experience in the field and know the challenges of running OpenStack for a wide variety of real-world deployments
    • our first solution came out on Cactus!  We’ve been delivering on Essex since OSCON 2012 (http://www.oscon.com/ ).
  • We represent a broad range of deployment scenarios ranging from hosting, government, healthcare, retail, education, media, financial and more!
  • We have broad engagements and partnerships at the infrastructure (SUSE, Canonical, Redhat), consulting (Canonical, Mirantis) and ecosystem layers (enStratus) and beyond!
  • We have a proven track record of collaboration instead of forking/disrupting – a critical skill for this project reflected by our consistent actions to preserve the integrity of the project.
  • We have led the “make OpenStack deployable” campaign with substantial investments (open source Crowbar, white papers, documentation & cookbooks.
  • We have very long and consistent history with the project starting even before the first OpenStack summit in Austin.

Of course, we’re asking for you to consider for both of us; however, if you want to focus on just one then here’s the balance between us.  Rob (bio) is a technologist with deep roots in cloud technology, data center operations and open source.  Joseph is a business professional with experience new product introduction and enterprise delivery.

Not sure if you can vote?  If you registered as an individual member then your name should be on the voting list.  In that case, you can vote between 8/20 and 8/24.

I’m seeking Nominations for OpenStack Board

Posted on by
Reply

The OpenStack Foundation is currently seeking nominations for community representatives for the board and I am asking for you to nominate me for that position.  Candidates are required to have ten (10) nominations to be considered for the election.  To nominate, you can join and nominate from here.  (I’m nominated, thanks!)

Rob Background:

As the OpenStack technology lead within Dell and long-time cloud deployer and developer, I made OpenStack deployability a top concern for Dell and the community.  My leadership has changed the dialog around OpenStack to be balanced between Ops and Dev.  I have also been pivotal in bringing open collaboration to OpenStack operations through our Crowbar project.   Through my role at Dell, I am actively engaged with numerous field deployments and uniquely positioned to represent OpenStack’s developer, provider and enterprise user bases.  I bring substantial process experience (Agile/Lean/CI) into my decision making.  My focus will be on ensuring OpenStack is deployable and ready for use.

More?  Ready my previous background post for previous OpenStack elections.

Crowbar 2.0 Design Summit Notes (+ open weekly meetings starting)

Posted on by
1

I could not be happier with the results Crowbar collaborators and my team at Dell achieved around the 1st Crowbar design summit. We had great discussions and even better participation.

The attendees represented major operating system vendors, configuration management companies, OpenStack hosting companies, OpenStack cloud software providers, OpenStack consultants, OpenStack private cloud users, and (of course) a major infrastructure provider. That’s a very complete cross-section of the cloud community.

I knew from the start that we had too little time and, thankfully, people were tolerant of my need to stop the discussions. In the end, we were able to cover all the planned topics. This was important because all these features are interlocked so discussions were iterative. I was impressed with the level of knowledge at the table and it drove deep discussion. Even so, there are still parts of Crowbar that are confusing (networking, late binding, orchestration, chef coupling) even to collaborators.

In typing up these notes, it becomes even more blindingly obvious that the core features for Crowbar 2 are highly interconnected. That’s no surprise technically; however, it will make the notes harder to follow because of knowledge bootstrapping. You need take time and grok the gestalt and surf the zeitgeist.

Collaboration Invitation: I wanted to remind readers that this summit was just the kick-off for a series of open weekly design (Tuesdays 10am CDT) and coordination (Thursdays 8am CDT) meetings. Everyone is welcome to join in those meetings – information is posted, recorded, folded, spindled and mutilated on the Crowbar 2 wiki page.

These notes are my reflection of the online etherpad notes that were made live during the meeting. I’ve grouped them by design topic.

Introduction

  • Contributors need to sign CLAs
  • We are refactoring Crowbar at this time because we have a collection of interconnected features that could not be decoupled
  • Some items (Database use, Rails3, documentation, process) are not for debate. They are core needs but require little design.
  • There are 5 key topics for the refactor: online mode, networking flexibility, OpenStack pull from source, heterogeneous/multi operating systems, being CDMB agnostic
  • Due to time limits, we have to stop discussions and continue them online.
  • We are hoping to align Crowbar 2 beta and OpenStack Folsom release.

Online / Connected Mode

  • Online mode is more than simply internet connectivity. It is the foundation of how Crowbar stages dependencies and components for deploy. It’s required for heterogeneous O/S, pull from source and it has dependencies on how we model networking so nodes can access resources.
  • We are thinking to use caching proxies to stage resources. This would allow isolated production environments and preserves the run everything from ISO without a connection (that is still a key requirement to us).
  • Suse’s Crowbar fork does not build an ISO, instead it relies on RPM packages for barclamps and their dependencies.
  • Pulling packages directly from the Internet has proven to be unreliable, this method cannot rely on that alone.

Install From Source

  • This feature is mainly focused on OpenStack, it could be applied more generally. The principals that we are looking at could be applied to any application were the source code is changing quickly (all of them?!). Hadoop is an obvious second candidate.
  • We spent some time reviewing the use-cases for this feature. While this appears to be very dev and pre-release focused, there are important applications for production. Specifically, we expect that scale customers will need to run ahead of or slightly adjacent to trunk due to patches or proprietary code. In both cases, it is important that users can deploy from their repository.
  • We discussed briefly our objective to pull configuration from upstream (not just OpenStack, but potentially any common cookbooks/modules). This topic is central to the CMDB agnostic discussion below.
  • The overall sentiment is that this could be a very powerful capability if we can manage to make it work. There is a substantial challenge in tracking dependencies – current RPMs and Debs do a good job of this and other configuration steps beyond just the bits. Replicating that functionality is the real obstacle.

CMDB agnostic (decoupling Chef)

  • This feature is confusing because we are not eliminating the need for a configuration management database (CMDB) tool like Chef, instead we are decoupling Crowbar from the a single CMDB to a pluggable model using an abstraction layer.
  • It was stressed that Crowbar does orchestration – we do not rely on convergence over multiple passes to get the configuration correct.
  • We had strong agreement that the modules should not be tightly coupled but did need a consistent way (API? Consistent namespace? Pixie dust?) to share data between each other. Our priority is to maintain loose coupling and follow integration by convention and best practices rather than rigid structures.
  • The abstraction layer needs to have both import and export functions
  • Crowbar will use attribute injection so that Cookbooks can leverage Crowbar but will not require Crowbar to operate. Crowbar’s database will provide the links between the nodes instead of having to wedge it into the CMDB.
  • In 1.x, the networking was the most coupled into Chef. This is a major part of the refactor and modeling for Crowbar’s database.
  • There are a lot of notes captured about this on the etherpad – I recommend reviewing them

Heterogeneous OS (bare metal provisioning and beyond)

  • This topic was the most divergent of all our topics because most of the participants were using some variant of their own bare metal provisioning project (check the etherpad for the list).
  • Since we can’t pack an unlimited set of stuff on the ISO, this feature requires online mode.
  • Most of these projects do nothing beyond OS provisioning; however, their simplicity is beneficial. Crowbar needs to consider users who just want a stream-lined OS provisioning experience.
  • We discussed Crowbar’s late binding capability, but did not resolve how to reconcile that with these other projects.
  • Critical use cases to consider:
    • an API for provisioning (not sure if it needs to be more than the current one)
    • pick which Operating Systems go on which nodes (potentially with a rules engine?)
    • inventory capabilities of available nodes (like ohai and factor) into a database
    • inventory available operating systems

Crowbar’s early twins: Cloudera Hadoop & OpenStack Essex

Posted on by
Reply

I’m proud to see my team announce the twin arrival of the Dell | Cloudera Apache Hadoop (Manager v4) and Dell OpenStack-Powered Cloud (Essex) solutions.

Not only are we simultaneously releasing both of these solutions, they reflect a significant acceleration in pace of delivery.  Both solutions had beta support for their core technologies (Cloudera 4 & OpenStack Essex) when the components were released and we have dramatically reduced the lag from component RC to solution release compared to past (3.7 & Diablo) milestones.

As before, the core deployment logic of these open source based solutions was developed in the open on Crowbar’s github.  You are invited to download and try these solutions yourself.   For Dell solutions, we include validated reference architectures, hardware configuration extensions for Crowbar, services and support.

The latest versions of Hadoop and OpenStack represent great strides for both solutions.   It’s great to be able have made them more deployable and faster to evaluate and manage.