What makes OpenStack meaningful to the market?

THIS POST IS #2 IN A SERIES ABOUT “WHAT IS CORE.”

What gives a project a strong core? 

A strong project has utility, community, and longevity.

TelescopeUtility, community and longevity are the fundamental objectives of any project or product.  It must do something that people find useful (utility).  It’s not enough for one person to like the project, there must be a market (community).  And that useful and popular work must be sustainable over multiple “generations” (longevity).

These goals are basic.  The challenge is finding the right rules to keep OpenStack in the sustainable project zone.  Unfortunately, as an open source project, the OpenStack Foundation ultimately has very little real power (like hiring flocks of developers) to enforce use or maintenance of the code base.

The Foundation’s tools are velocity, culture, and brand.  Understanding “what is core” hones these tools to ensure they are effective.

Velocity – the rate of progress and quality of the code base.  A project at sufficient velocity is not worth forking or duplicating.  The fact that >1000 developers companies are contributing and 100s of companies are deploying OpenStack makes it profitable to remain in our community.   Make no mistake: being part of a community takes effort so there must be a return on that investment.  The foundation must ensure that commercial entities find an ROI from their participation.

Culture – open source culture strongly encourages sharing and collaboration.  I have seen that culture as a more potent force than the legalese and licenses.  While a strong culture reinforces itself, a toxic culture will rot a project like ice cream in the summerCulture maintenance is a chief foundation objective and includes fostering new users, documentation, orderly interactions and co-opetitive collaboration.

Brand – when all else fails, OpenStack can use legal means to define our brand.  This is the weakest of all the tools because the strength of the defense is only as good as the brand.  If we allow the OpenStack brand (sometimes we say it’s mark) to become weak or diluted then people have little reason to support velocity or culture.

An important insight when looking at these three control levers is that they are very different between individuals and corporations.  While individuals may highly motivated by culture they are not as motivated by brand; conversely, corporations are highly motivated by brand and compliance and minimally by culture.

As the OpenStack Foundation Board takes up the “what is core” question, we must be cognizant of the duality between individual and corporate interests.  OpenStack must be both meaningful culturally to individuals and strong brand-wise to corporations.   Both are needed to sustain OpenStack’s velocity.

READ POST IS #2: SPIDER CHART

Kicking off discussion about OpenStack Core

What's in?I’ve been leading an effort with Alan Clark to define “what is OpenStack core” for the Foundation Board.  Now that I am sitting here at OSCON and celebrating OpenStack’s third birthday, I think it’s a great time to bring the general community into the discussion.

There is significant history behind this topic.  According to Foundation governance, the Technical Committee (TC) defines which incubated projects are integrated and the Board of Directors (I am one) determines which of the integrated projects are labeled as core.

When it comes to the core label, the stakes are high.

Defining core is a convoluted topic.  To make it digestible, I’m breaking it down into multiple blog posts over a series of weeks:

  1. Why do we care about core?
  2. Decomposing the problem (“the spider”)
  3. Insights from decomposition (healthy tensions in OpenStack)
  4. 12 Positions: A Common Framework (I recommend ready the list of 10 instead)
  5. Community Feedback at OSCON 
  6. “What is Core” the visualization
  7. Where I think this is going, OpenStack’s Test Driven Core
  8. Core Positions Refined: the 10 positions behind the core visualization (above).
  9. Videos (most >90 mins).  The online meetups are easier to follow.
    1. 9/5 OpenStack Core Meetup in San Antonio
    2. 9/22 OpenStack Core Meetup in NYC
    3. online: 10/16 Online Meetup (daytime) and 10/22 Online Meetup (evening)
  10. Thinking about how to Implement OpenStack Core Definition

Too much reading?  At OSCON, Rafael Knuth shot a video of me talking about “what is core.”

Crowbar 2.0 Design Summit Notes (+ open weekly meetings starting)

I could not be happier with the results Crowbar collaborators and my team at Dell achieved around the 1st Crowbar design summit. We had great discussions and even better participation.

The attendees represented major operating system vendors, configuration management companies, OpenStack hosting companies, OpenStack cloud software providers, OpenStack consultants, OpenStack private cloud users, and (of course) a major infrastructure provider. That’s a very complete cross-section of the cloud community.

I knew from the start that we had too little time and, thankfully, people were tolerant of my need to stop the discussions. In the end, we were able to cover all the planned topics. This was important because all these features are interlocked so discussions were iterative. I was impressed with the level of knowledge at the table and it drove deep discussion. Even so, there are still parts of Crowbar that are confusing (networking, late binding, orchestration, chef coupling) even to collaborators.

In typing up these notes, it becomes even more blindingly obvious that the core features for Crowbar 2 are highly interconnected. That’s no surprise technically; however, it will make the notes harder to follow because of knowledge bootstrapping. You need take time and grok the gestalt and surf the zeitgeist.

Collaboration Invitation: I wanted to remind readers that this summit was just the kick-off for a series of open weekly design (Tuesdays 10am CDT) and coordination (Thursdays 8am CDT) meetings. Everyone is welcome to join in those meetings – information is posted, recorded, folded, spindled and mutilated on the Crowbar 2 wiki page.

These notes are my reflection of the online etherpad notes that were made live during the meeting. I’ve grouped them by design topic.

Introduction

  • Contributors need to sign CLAs
  • We are refactoring Crowbar at this time because we have a collection of interconnected features that could not be decoupled
  • Some items (Database use, Rails3, documentation, process) are not for debate. They are core needs but require little design.
  • There are 5 key topics for the refactor: online mode, networking flexibility, OpenStack pull from source, heterogeneous/multi operating systems, being CDMB agnostic
  • Due to time limits, we have to stop discussions and continue them online.
  • We are hoping to align Crowbar 2 beta and OpenStack Folsom release.

Online / Connected Mode

  • Online mode is more than simply internet connectivity. It is the foundation of how Crowbar stages dependencies and components for deploy. It’s required for heterogeneous O/S, pull from source and it has dependencies on how we model networking so nodes can access resources.
  • We are thinking to use caching proxies to stage resources. This would allow isolated production environments and preserves the run everything from ISO without a connection (that is still a key requirement to us).
  • Suse’s Crowbar fork does not build an ISO, instead it relies on RPM packages for barclamps and their dependencies.
  • Pulling packages directly from the Internet has proven to be unreliable, this method cannot rely on that alone.

Install From Source

  • This feature is mainly focused on OpenStack, it could be applied more generally. The principals that we are looking at could be applied to any application were the source code is changing quickly (all of them?!). Hadoop is an obvious second candidate.
  • We spent some time reviewing the use-cases for this feature. While this appears to be very dev and pre-release focused, there are important applications for production. Specifically, we expect that scale customers will need to run ahead of or slightly adjacent to trunk due to patches or proprietary code. In both cases, it is important that users can deploy from their repository.
  • We discussed briefly our objective to pull configuration from upstream (not just OpenStack, but potentially any common cookbooks/modules). This topic is central to the CMDB agnostic discussion below.
  • The overall sentiment is that this could be a very powerful capability if we can manage to make it work. There is a substantial challenge in tracking dependencies – current RPMs and Debs do a good job of this and other configuration steps beyond just the bits. Replicating that functionality is the real obstacle.

CMDB agnostic (decoupling Chef)

  • This feature is confusing because we are not eliminating the need for a configuration management database (CMDB) tool like Chef, instead we are decoupling Crowbar from the a single CMDB to a pluggable model using an abstraction layer.
  • It was stressed that Crowbar does orchestration – we do not rely on convergence over multiple passes to get the configuration correct.
  • We had strong agreement that the modules should not be tightly coupled but did need a consistent way (API? Consistent namespace? Pixie dust?) to share data between each other. Our priority is to maintain loose coupling and follow integration by convention and best practices rather than rigid structures.
  • The abstraction layer needs to have both import and export functions
  • Crowbar will use attribute injection so that Cookbooks can leverage Crowbar but will not require Crowbar to operate. Crowbar’s database will provide the links between the nodes instead of having to wedge it into the CMDB.
  • In 1.x, the networking was the most coupled into Chef. This is a major part of the refactor and modeling for Crowbar’s database.
  • There are a lot of notes captured about this on the etherpad – I recommend reviewing them

Heterogeneous OS (bare metal provisioning and beyond)

  • This topic was the most divergent of all our topics because most of the participants were using some variant of their own bare metal provisioning project (check the etherpad for the list).
  • Since we can’t pack an unlimited set of stuff on the ISO, this feature requires online mode.
  • Most of these projects do nothing beyond OS provisioning; however, their simplicity is beneficial. Crowbar needs to consider users who just want a stream-lined OS provisioning experience.
  • We discussed Crowbar’s late binding capability, but did not resolve how to reconcile that with these other projects.
  • Critical use cases to consider:
    • an API for provisioning (not sure if it needs to be more than the current one)
    • pick which Operating Systems go on which nodes (potentially with a rules engine?)
    • inventory capabilities of available nodes (like ohai and factor) into a database
    • inventory available operating systems

Crowbar Celebrates 1st Anniversary

Nearly a year ago at OSCON 2011, my team at Dell opened sourced “Crowbar, an OpenStack installer.” That first Github commit was a much more limited project than Crowbar today: there was no separation into barclamps, no distinct network configuration, one operating system option and the default passwords were all “openstack.” We simply did not know if our effort would create any interest.

The response to Crowbar has been exciting and humbling. I most appreciate those who looked at Crowbar and saw more than a bare metal installer. They are the ones who recognized that we are trying to solve a bigger problem: it has been too difficult to cope with change in IT operations.

During this year, we have made many changes. Many have been driven by customer, user and partner feedback while others support Dell product delivery needs. Happily, these inputs are well aligned in intent if not always in timing.

  • Introduction of barclamps as modular components
  • Expansion into multiple applications (most notably OpenStack and Apache Hadoop)
  • Multi-Operating System
  • Working in the open (with public commits)
  • Collaborative License Agreements

Dell‘s understanding of open source and open development has made a similar transformation. Crowbar was originally Apache 2 open sourced because we imagined it becoming part of the OpenStack project. While that ambition has faded, the practical benefits of open collaboration have proven to be substantial.

The results from this first year are compelling:

  • For OpenStack Diablo, coordination with the Rackspace Cloud Builder team enabled Crowbar to include the Keystone and Dashboard projects into Dell’s solution
  • For OpenStack Essex, the community focused work we did for the March Essex Hackday are directly linked to our ability to deliver Dell’s OpenStack-Powered Essex solution over two months earlier than originally planned.
  • For Apache Hadoop distributions for 3.x and 4.x with implementation of Cloudera Manager and eco system components.
  • We’ve amassed hundreds of mail subscribers and Github followers
  • Support for multiple releases of RHEL, Centos & Ubuntu including Ubuntu 12.04 while it was still in beta.
  • SuSE does their own port of Crowbar to SuSE with important advances in Crowbar’s install model (from ISO to package).

We stand on the edge of many exciting transformations for Crowbar’s second year. Based on the amount of change from this year, I’m hesitant to make long term predictions. Yet, just within next few months there are significant plans based on Crowbar 2.0 refactor. We have line of site to changes that expand our tool choices, improve networking, add operating systems and become more even production ops capable.

That’s quite a busy year!

OSCON preso: how Dell Crowbar brings DevOps to OpenStack Cloud (“No Soup for You!”)

Today I presented about how Crowbar + DevOps + OpenStack = CloudOps.   The highlight of the presentation (to me, anyway) is the Images vs Layers analogy of Soup vs Sandwiches.  I hope it helps explain why we believe that a DevOps approach to Cloud is essential to success.

Here’s the preso: OSCON 07 2011

I’ll add a link to the videos when they are available.

Videos about Crowbar, CloudOps, and Dell OpenStack Cloud

I’m not usually a big fan of launch videos (too much markitecture); however, these turned out to be nice and meaty.  The meaty part explains why it looks like I’m about to eat a big sandwich in the last video.  yum!

  • What is Crowbar: Dell Crowbar Software Overview  

Continue reading

Crowbar source released, includes OpenStack Cloud install

I’m delighted to announce (official version) that my team at Dell has opened the Crowbar source under the Apache 2 license. This action is part of the broader Dell OpenStack Cloud Solution which includes OpenStack install packages, Crowbar, reference hardware architectures, and services/consulting to support deployments.

There are two important components to this news:

  1. Dell is officially offering our OpenStack Solution and helping advance the community’s ability to implement OpenStack quickly and consistently.
  2. Dell is releasing the Crowbar code (which is included in the solution) as open source.

Both are significant items; however, my focus here is on the Crowbar release.

Crowbar started as a Dell OpenStack installer project and then grew beyond that in scope.  Now it can be extended to work with other vendors’ kits and other solutions bits.

We are contributing Crowbar to the community because we believe that everyone benefits by sharing in the operational practices that Crowbar embodies. These are rooted in Opscsode Chef (which Crowbar tightly integrates with) and the cloud & hyper-scale proven DevOps practices that are reflected in our deployment model.

Where to get it?

What’s included?

  • A comprehensive set of barclamps to set up an OpenStack cloud.
  • Crowbar UI and Remote APIs to make it easy to set up your cloud
  • Automated testing scripts for community members doing continuous integration with OpenStack.
  • Build scripts so you can create your own Crowbar install ISO
  • Switch discovery so you can create Chef Cookbooks that are network aware.
  • Open source Chef server that powers much of Crowar’s functionality

What’s not included?

  • Non-open source license components (BIOS+RAID config) that we could not distribute under the Apache 2 license.  We are working to address this and include them in our release.  They are available in the Dell Licensed version of Crowbar.
  • Dell Branded Components (skin + overview page).   Crowbar has an OpenSource skin with identical functionality.
  • Pre-built ISOs with install images (you must download the open source components yourself, we cannot redistribute them to you as a package)

Important notes:

  • Crowbar uses Chef Server as its database and relies on cookbooks for node deployments.  It is installed (using Chef Solo) automatically as part of the Crowbar install.
  • Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as barclamps.
  • Each barclamp has its own Chef configuration, UI sub-component, deployment configuration, and documentation.

On the project roadmap:

  • Hadoop support
  • Additional operating system support (specifically RHEL)
  • Barclamp version repository
  • Network configuration
  • We’d like suggestions!  Please comment!

Sites for more information: Joseph George, Barton George (launch day), Dell

OpenStack at OSCON schedule & event signup

If you’re at OSCON, here’s where to find OpenStack content:

OpenStack Wednesday Evening Event (RSVP REQUIRED):

Wednesday, July 27, 7-9 pm, at Spirit of 77 (right across from the Oregon
Convention Center at the close of the day).  Join us to toast the first
anniversary of the fastest-growing open source project! Please register here and
help promote the event: http://openstack-one-year.eventbrite.com

Speaking Sessions, Wednesday, July 27


Introduction to OpenStack, Eric Day

Wednesday, 1:40 pm http://www.oscon.com/oscon2011/public/schedule/detail/19146

Using OpenStack APIs, Present and Future, Mike Mayo
Wednesday, 4:10 pm http://www.oscon.com/oscon2011/public/schedule/detail/18550

OpenStack Fundamentals Training Part 1, Swift, John Dickinson
Wednesday, 4:10 pm http://www.oscon.com/oscon2011/public/schedule/detail/21287

OpenStack Fundamentals Training Part 2, Nova, Jason Cannavale
Wednesday, 5:00 pm http://www.oscon.com/oscon2011/public/schedule/detail/21347

OpenStack One-Year Anniversary Party, Spirit of 77
Wednesday, 7-9 pm http://openstack-one-year.eventbrite.com/

Speaking Sessions, Thursday, July 28

See why Rob says “No Soup for You” about Cloud Deployments.

Prying Open the Cloud with Dell Crowbar and OpenStack, Joseph George, Rob Hirschfeld
Thursday, 10:40 am http://www.oscon.com/oscon2011/public/schedule/detail/21206

OpenStack + Ceph, Ben Cherian, Jonathan Bryce
Thursday, 1:40 pm http://www.oscon.com/oscon2011/public/schedule/detail/21174

Achieving Hybrid Cloud Mobility with OpenStack and XCP, Paul Voccio, Ewan Mellor
Thursday, 2:30 pm http://www.oscon.com/oscon2011/public/schedule/detail/18726

I’ll be at OSCON 7/25-29/11 (Dell=sponsor & speaking w/ @jbgeorge)

As part of our commitment to open source, Dell is a sponsor of OSCON 2011.  The Dell OpenStack Cloud team will have a booth presence with our well-travelled Crowbar Install rack (now with BOTH PowerEdge C6100 & C6105s).  We’re doing our famous 30 minute OpenStack installs and handing out goodies including USB keys. 

Joseph George (@jbgeorge) and I are speaking:

We’ll be giving specifics about how Crowbar works to deliver the Dell OpenStack Cloud Solution including a narrated demo and details about how the community can extend Crowbar using barclamps.

Note:  Stephen Spector (opnstk_com_mgr), the amazing OpenStack community manager, wanted me to remind everyone that we’re celebrating OpenStack’s 1 year anniversary with activites at OSCON.  He’s asking for video commentary about OpenStack and RSVPs if you can attend the events.  More at OpenStack Blog.