Category Archives: Events

Crowbar releases for OpenStack Essex and Folsom

Posted on by
6

On the eve of the OpenStack design summit, it’s worth noting that the Crowbar team at Dell cut our final Essex release (aka Betty) last week. We’ve also committed the initial Folsom deployment scripts to the 1x development trunk under “feature/folsom” if you are doing Crowbar builds from DevTool (see bit.ly/crowbardevtool).

If you don’t want to build it, I’ve cut ISOs from the open source bits and posted them to http://crowbar.zehicle.com.

Andi Abes is presenting about the new Pull From Source (pfs) feature at the Summit on Monday. There’s a feature branch for that too and I’m going to check with him and try to post and ISO for that too.

20121014-132246.jpg

OpenStack Summit: Let’s talk DevOps, Fog, Upgrades, Crowbar & Dell

Posted on by
7

If you are coming to the OpenStack summit in San Diego next week then please find me at the show! I want to hear from you about the Foundation, community, OpenStack deployments, Crowbar and anything else.  Oh, and I just ordered a handful of Crowbar stickers if you wanted some CB bling.

Matt Ray (Opscode), Jason Cannavale (Rackspace) and I were Ops track co-chairs. If you have suggestions, we want to hear. We managed to get great speakers and also some interesting sessions like DevOps panel and up streaming deploy working sessions. It’s only on Monday and Tuesday, so don’t snooze or you’ll miss it.

My team from Dell has a lot going on, so there are lots of chances to connect with us:

At the Dell booth, Randy Perryman will be sharing field experience about hardware choices. We’ve got a lot of OpenStack battle experience and we want to compare notes with you.

I’m on the board meeting on Monday so likely occupied until the Mirantis party.

See you in San Diego!

PS: My team is hiring for Dev, QA and Marketing. Let me know if you want details.

Join in Blog Action Day on 10/15

Posted on by
Reply

You’ll have to wait for the reveal on 10/15 but I wanted to throw out the link for blog action day and encourage fellow bloggers to participate in the event.

I participated in this event while I blogging about electric cars (I converted 96 RAV4 to EV) and energy issues.  It’s an interesting perspective to have a large focus on a single topic.  The breadth of discussion is impressive.

This year, I’m going to be topical to my latest interests and offering insights from the floor of the OpenStack summit.

Ballistic Release Cycles: Tracking the Trajectory of OpenStack Milestones

Posted on by
9

I’ve been watching a pattern emerge on the semiannual OpenStack release cycles for a while now. There is a hidden but crucial development phase that accelerates projects faster than many observers realize. In fact, I believe that substantial work is happening outside of the “normal” design cycle during what I call “free fall” development.

Understanding when the cool, innovative stuff happens is essential to getting (and giving) the most from OpenStack.

The published release cycle looms like a 6 stage ballistic trajectory. Launching at the design summit, the release features change and progress the most in the first 3 milestones. At the apogee of the release, maximum velocity is reached just as we start having to decide which features are complete enough to include in the release. Since many are not ready, we have to jettison (really, defer) partial work to ensure that we can land the release on schedule.

I think of the period where we lose potential features as free fall because thing can go in any direction. The release literally reverses course: instead of expanding, it is contracting. This process is very healthy for OpenStack. It favors code stability and “long” hardening times. For operators, this means that the code stops changing early enough that we have more time to test and operationalize the release.

But what happens to the jettisoned work? In free fall, objects in motion stay in motion. The code does not just disappear! It continues on its original upward trajectory.

The developers who invested time in the code do not simply take a 3 month sabbatical, nor do they stop their work and start testing the code that was kept. No, after the short in/out sorting pause, the free fall work continues onward with rockets blasting. The challenge is that it is now getting outside of the orbit of the release plan and beyond the radar of many people who are tracking the release.

The consequence of this ongoing development is that developers (and the features they are working on) show up at the summit with 3 extra months of work completed. It also means that OpenStack starts each release cycle with a bucket of operationally ready code. Wow, that’s a huge advantage for the project in terms of delivered work, feature velocity and innovation. Even better, it means that the design summit can focus on practical discussions of real prototypes and functional features.

Unfortunately, this free fall work has hidden costs:

  • It is relatively hidden because it is outside of the normal release cycle.
  • It makes true design discussions less productive because the implemented code is more likely to make the next release cycle
  • Integration for the work is postponed because it continues before branching
  • Teams that are busy hardening a core feature can be left out of work on the next iteration of the same feature
  • Forking can make it hard to capture bugs caught during hardening

I think OpenStack greatly benefits from free fall development; consequently, I think we need to acknowledge and embrace it to reduce its costs. A more explicit mid-release design synchronization when or before we fork may help make this hidden work more transparent.

I am seeking your vote(s) for the OpenStack Board

Posted on by
4

If registered, you have 8 votes to allocate as you wish.  You will get a link via email – you must use that link.

Joseph B George and I are cross-blogging this post because we are jointly seeking your vote(s) for individual member seats on the OpenStack Foundation board.  This is key point in the OpenStack journey and we strongly encourage eligible voters to participate no matter who you vote for!  As we have said before, success of the Foundation governance process matters just as much as the code because it ensures equal access and limits forking.

We think that OpenStack succeeds because it is collaboratively developed.  It is essential that we select board members who have a proven record of community development, a willingness to partner and have demonstrated investment in the project.

Our OpenStack vision favors production operations by being operator, user and ecosystem focused.  If elected, we will represent these interests by helping advance deployability, API specifications, open operations and both large and small scale cloud deployments.

Of the nominees, we best represent OpenStack users and operators (as opposed to developers).  We have the most diverse experience in real-world OpenStack deployments because our solution has been deployed broadly (both as Dell and through Crowbar.  We have a proven record of collaborating broadly with contributors, demonstrated skills at building the OpenStack community and doing real open source work to ensure that OpenStack is the most deployable cloud platform anywhere.

Let’s get specific about our leadership in the OpenStack project and community:

  • We have been active and vocal leaders in the OpenStack community
    • our team has established two very active user groups (Austin & Boston)
    • we have lead multiple world-wide deploy day events (March 2012  &  May 2012).
    • we have substantial experience in the field and know the challenges of running OpenStack for a wide variety of real-world deployments
    • our first solution came out on Cactus!  We’ve been delivering on Essex since OSCON 2012 (http://www.oscon.com/ ).
  • We represent a broad range of deployment scenarios ranging from hosting, government, healthcare, retail, education, media, financial and more!
  • We have broad engagements and partnerships at the infrastructure (SUSE, Canonical, Redhat), consulting (Canonical, Mirantis) and ecosystem layers (enStratus) and beyond!
  • We have a proven track record of collaboration instead of forking/disrupting – a critical skill for this project reflected by our consistent actions to preserve the integrity of the project.
  • We have led the “make OpenStack deployable” campaign with substantial investments (open source Crowbar, white papers, documentation & cookbooks.
  • We have very long and consistent history with the project starting even before the first OpenStack summit in Austin.

Of course, we’re asking for you to consider for both of us; however, if you want to focus on just one then here’s the balance between us.  Rob (bio) is a technologist with deep roots in cloud technology, data center operations and open source.  Joseph is a business professional with experience new product introduction and enterprise delivery.

Not sure if you can vote?  If you registered as an individual member then your name should be on the voting list.  In that case, you can vote between 8/20 and 8/24.

Crowbar 2.0 Design Summit Notes (+ open weekly meetings starting)

Posted on by
1

I could not be happier with the results Crowbar collaborators and my team at Dell achieved around the 1st Crowbar design summit. We had great discussions and even better participation.

The attendees represented major operating system vendors, configuration management companies, OpenStack hosting companies, OpenStack cloud software providers, OpenStack consultants, OpenStack private cloud users, and (of course) a major infrastructure provider. That’s a very complete cross-section of the cloud community.

I knew from the start that we had too little time and, thankfully, people were tolerant of my need to stop the discussions. In the end, we were able to cover all the planned topics. This was important because all these features are interlocked so discussions were iterative. I was impressed with the level of knowledge at the table and it drove deep discussion. Even so, there are still parts of Crowbar that are confusing (networking, late binding, orchestration, chef coupling) even to collaborators.

In typing up these notes, it becomes even more blindingly obvious that the core features for Crowbar 2 are highly interconnected. That’s no surprise technically; however, it will make the notes harder to follow because of knowledge bootstrapping. You need take time and grok the gestalt and surf the zeitgeist.

Collaboration Invitation: I wanted to remind readers that this summit was just the kick-off for a series of open weekly design (Tuesdays 10am CDT) and coordination (Thursdays 8am CDT) meetings. Everyone is welcome to join in those meetings – information is posted, recorded, folded, spindled and mutilated on the Crowbar 2 wiki page.

These notes are my reflection of the online etherpad notes that were made live during the meeting. I’ve grouped them by design topic.

Introduction

  • Contributors need to sign CLAs
  • We are refactoring Crowbar at this time because we have a collection of interconnected features that could not be decoupled
  • Some items (Database use, Rails3, documentation, process) are not for debate. They are core needs but require little design.
  • There are 5 key topics for the refactor: online mode, networking flexibility, OpenStack pull from source, heterogeneous/multi operating systems, being CDMB agnostic
  • Due to time limits, we have to stop discussions and continue them online.
  • We are hoping to align Crowbar 2 beta and OpenStack Folsom release.

Online / Connected Mode

  • Online mode is more than simply internet connectivity. It is the foundation of how Crowbar stages dependencies and components for deploy. It’s required for heterogeneous O/S, pull from source and it has dependencies on how we model networking so nodes can access resources.
  • We are thinking to use caching proxies to stage resources. This would allow isolated production environments and preserves the run everything from ISO without a connection (that is still a key requirement to us).
  • Suse’s Crowbar fork does not build an ISO, instead it relies on RPM packages for barclamps and their dependencies.
  • Pulling packages directly from the Internet has proven to be unreliable, this method cannot rely on that alone.

Install From Source

  • This feature is mainly focused on OpenStack, it could be applied more generally. The principals that we are looking at could be applied to any application were the source code is changing quickly (all of them?!). Hadoop is an obvious second candidate.
  • We spent some time reviewing the use-cases for this feature. While this appears to be very dev and pre-release focused, there are important applications for production. Specifically, we expect that scale customers will need to run ahead of or slightly adjacent to trunk due to patches or proprietary code. In both cases, it is important that users can deploy from their repository.
  • We discussed briefly our objective to pull configuration from upstream (not just OpenStack, but potentially any common cookbooks/modules). This topic is central to the CMDB agnostic discussion below.
  • The overall sentiment is that this could be a very powerful capability if we can manage to make it work. There is a substantial challenge in tracking dependencies – current RPMs and Debs do a good job of this and other configuration steps beyond just the bits. Replicating that functionality is the real obstacle.

CMDB agnostic (decoupling Chef)

  • This feature is confusing because we are not eliminating the need for a configuration management database (CMDB) tool like Chef, instead we are decoupling Crowbar from the a single CMDB to a pluggable model using an abstraction layer.
  • It was stressed that Crowbar does orchestration – we do not rely on convergence over multiple passes to get the configuration correct.
  • We had strong agreement that the modules should not be tightly coupled but did need a consistent way (API? Consistent namespace? Pixie dust?) to share data between each other. Our priority is to maintain loose coupling and follow integration by convention and best practices rather than rigid structures.
  • The abstraction layer needs to have both import and export functions
  • Crowbar will use attribute injection so that Cookbooks can leverage Crowbar but will not require Crowbar to operate. Crowbar’s database will provide the links between the nodes instead of having to wedge it into the CMDB.
  • In 1.x, the networking was the most coupled into Chef. This is a major part of the refactor and modeling for Crowbar’s database.
  • There are a lot of notes captured about this on the etherpad – I recommend reviewing them

Heterogeneous OS (bare metal provisioning and beyond)

  • This topic was the most divergent of all our topics because most of the participants were using some variant of their own bare metal provisioning project (check the etherpad for the list).
  • Since we can’t pack an unlimited set of stuff on the ISO, this feature requires online mode.
  • Most of these projects do nothing beyond OS provisioning; however, their simplicity is beneficial. Crowbar needs to consider users who just want a stream-lined OS provisioning experience.
  • We discussed Crowbar’s late binding capability, but did not resolve how to reconcile that with these other projects.
  • Critical use cases to consider:
    • an API for provisioning (not sure if it needs to be more than the current one)
    • pick which Operating Systems go on which nodes (potentially with a rules engine?)
    • inventory capabilities of available nodes (like ohai and factor) into a database
    • inventory available operating systems

Open Community Access to Crowbar 2 Efforts

Posted on by
Reply

We’re moving along on the Crowbar2 refactoring work (`/release/rails3anddb/master` branch for now) and it’s time to start making it easier for you to participate if you are interested.

We are planning to start having TWO weekly community sprint meetings.

NOTE: Times can still shift depending on community input! We are trying to a truly global community so we need your input.

The weekly design discussions will be on Tuesdays @ 10am Central (GMT -6). The topics will be relevant to the coming sprint and we expect dialog. The topics will be determined by Greg Althaus based on progress on the refactor. You’re welcome to contact him or the list with suggestions.

The purpose of these meetings is to

  • discuss/resolve design related to the refactor
  • Document use-cases
  • identify issues that need to be addressed in the next sprint

The weekly coordination meetings will be on Thursdays @ 8am Central (GMT -6). We want to respect everyone’s time and will strictly limit these to 1 hour. This meeting is in between our internal sprint review and planning so we have flexibility to adjust our plans based on your input. It is important to us that we make it possible for you to contribute and we need your input to make sure that you are not blocked!

The coordination meetings will be structured as follows (times approximate)

  • Voice & Screencast: https://join.me/dellcrowbar
  • 25 minutes for review of current progress
  • 10 minutes for feedback/adjustments on process and workflow
  • 25 minutes for planning of next sprint
  • Online discussion & notes on the http://crowbar.sync.in/sprintMMDD etherpads
  • Identification of working groups for further discussion and coding collaboration

These meetings are the primary way that we will be making sure the community is not blocked by our development efforts.

The purpose of these meetings is

  • to synchronize quickly so that we can connect the people who should be collaborating
  • eliminate blocking items for Crowbar contributors

Of course, we will attempt to record and post all of these meetings.

Stay tuned! We will likely announce additional meetings for community collaboration.

PS: These are design meetings, they are NOT Crowbar training meetings. Please consult http://bit.ly/crowbarwiki for links and videos about learning Crowbar.

Community Participation in Crowbar 2.0 Efforts

Posted on by
3

I’ve laid out the Dell Crowbar team‘s technical objectives for Crowbar 2.0. Now it’s time to lay out how we plan to involve the Crowbar community. We welcome community participation and contributions but, I’m not making any pretense here, that refactoring must also satisfy needs expressed by the “Dell OpenStack powered Cloud” and “Dell | Cloudera Apache Hadoop” solutions. There is a happy alignment between Dell’s needs and our community’s since both of these solutions are openly developed.

Our community played a substantial role in the definition of Crowbar 2.0. We have been watching and listening carefully and the majority of the changes will leverage recommendations and address concerns raised by Crowbar users.

The Dell Crowbar team has been working towards Crowbar 2 for quite a while. Networking and operating system code is already in place and our team began design requirements for the refactoring mid-June.

Community Communication

We have setup the following meeting for community engagement around the Crowbar refactoring. Our purpose is to enable development collaboration during the refactoring – these will not be “learning Crowbar” sessions.

  • Design Brief (2 hour meeting)
    • Objective: Review by of design and refactoring plans in progress by Dell Crowbar team. Provide basis for community input and discussion during week.
    • When: Monday 7/16 at 10am CDT
    • Where: Online Session
    • Who: Anyone (this is a listening session, not a discussion event)
  • Design Summit (4+2 hour meeting)
    • Objective: Provide input and suggestions about design proposal (see preparation). Identify members of collaboration teams for refactoring effort.
    • Preparation: Attend 7/16 Design Brief & Complete Expert Homework.
    • When: Friday 7/20
    • Where: OSCON 2012 in Portland (not on-site at OSCON). To facilitate face-to-face dialog, there will be no interactive online component.
    • Who: 20 people, invitation only due to space limitations. Attendees representing DevOps tools, hosting companies, operating system vendors, Hadoop & OpenStack contributors
    • Notes will be posted
  • Follow-up sessions
    • Objective: Coordinate delivery of refactored code
    • When: Likely weekly following Summit
    • Where: Online (Skype or Google Hangout)
    • Who: Crowbar Developers

Of course, we will continue to engage on the Crowbar list and Skype channels.

More on the 7/20 Design Summit

The purpose of the summit is strictly limited to discussing and organizing efforts around Crowbar 2.0 refactoring. To maintain a tight focus on this effort, we made the decision to limit the audience of this event by making it in-person and invitation only.

The Summit is not a broad discussion about Crowbar’s future with an open space format. We have a specific need and agenda – coordinating the development of 2.0. For that reason, we made to decision to restrict invitations to partners who are ready to commit substantial effort towards Crowbar development in the next 6 months. This is a technical session only: attendees must have first-hand experience coding Crowbar, be able to build the code and take on development or test commitments.

Because of the time limitations, we are holding a mandatory pre-summit brief about the design changes. This session will be open for the community and we welcome comments.

The agenda for the Design Summit is as follows:

  • 8:30 (45 mins): Crowbar 2.0 Challenges & Objectives
  • 9:15 (30 mins): Packaging and Delivery
  • 9:45 (15 mins) Break
  • 10:00 (30 mins): Networking
  • 10:30 (30 mins): Data Representation and Framework
  • 11:00 (30 mins): Next Steps, Future meetings & work assignments
  • 11:30 official content ends & lunch
  • 12:00 (30 mins) breakout sessions based on discussions above, TBD during summit
  • 12:30 (30 mins) breakout sessions based on discussions above, TBD during summit
  • 1:00 (30 mins) breakout sessions based on discussions above, TBD during summit
  • 1:30 (30 mins) breakout sessions based on discussions above, TBD during summit
  • 2:00 meeting ends

Post Script: A Community Design Summit

We have already begun discussing a true community summit with an open agenda. That summit will likely be a dedicated event in Austin, Texas with 2 (or more!) days of content. Please let me know if this is of interest to you so we can begin building the business case for hosting it!

Stop the Presses! Austin OpenStack Meetup 7/12 features docs, bugs & cinder

Posted on by
Reply

Don’t miss the 7/12 OpenStack Austin meetup!  We’ve got a great agenda lined up.

This meetup is sponsored by HP (Mark Padovani will give the intro).

Topics will include

  1. 6:30 pre-meeting OpenStack intro & overview for N00bs.
  2. Anne Gentle, OpenStack Technical Writer at Rackspace Hosting, talking about How to contribute to docs & the areas needed. *
  3. Report on the Folsom.3 bug squash day (http://wiki.openstack.org/BugDays/20120712BugSquashing)
  4. (tentative) Greg Althaus, Dell, talking about the “Cinder” Block Storage project
  5. White Board – Next Meeting Topics

* if you contribute to docs then you’ll get an invite to the next design summit!   It’s a great way to support OpenStack even if you don’t write code.

Crowbar Celebrates 1st Anniversary

Posted on by
1

Nearly a year ago at OSCON 2011, my team at Dell opened sourced “Crowbar, an OpenStack installer.” That first Github commit was a much more limited project than Crowbar today: there was no separation into barclamps, no distinct network configuration, one operating system option and the default passwords were all “openstack.” We simply did not know if our effort would create any interest.

The response to Crowbar has been exciting and humbling. I most appreciate those who looked at Crowbar and saw more than a bare metal installer. They are the ones who recognized that we are trying to solve a bigger problem: it has been too difficult to cope with change in IT operations.

During this year, we have made many changes. Many have been driven by customer, user and partner feedback while others support Dell product delivery needs. Happily, these inputs are well aligned in intent if not always in timing.

  • Introduction of barclamps as modular components
  • Expansion into multiple applications (most notably OpenStack and Apache Hadoop)
  • Multi-Operating System
  • Working in the open (with public commits)
  • Collaborative License Agreements

Dell‘s understanding of open source and open development has made a similar transformation. Crowbar was originally Apache 2 open sourced because we imagined it becoming part of the OpenStack project. While that ambition has faded, the practical benefits of open collaboration have proven to be substantial.

The results from this first year are compelling:

  • For OpenStack Diablo, coordination with the Rackspace Cloud Builder team enabled Crowbar to include the Keystone and Dashboard projects into Dell’s solution
  • For OpenStack Essex, the community focused work we did for the March Essex Hackday are directly linked to our ability to deliver Dell’s OpenStack-Powered Essex solution over two months earlier than originally planned.
  • For Apache Hadoop distributions for 3.x and 4.x with implementation of Cloudera Manager and eco system components.
  • We’ve amassed hundreds of mail subscribers and Github followers
  • Support for multiple releases of RHEL, Centos & Ubuntu including Ubuntu 12.04 while it was still in beta.
  • SuSE does their own port of Crowbar to SuSE with important advances in Crowbar’s install model (from ISO to package).

We stand on the edge of many exciting transformations for Crowbar’s second year. Based on the amount of change from this year, I’m hesitant to make long term predictions. Yet, just within next few months there are significant plans based on Crowbar 2.0 refactor. We have line of site to changes that expand our tool choices, improve networking, add operating systems and become more even production ops capable.

That’s quite a busy year!