OpenStack: Five Challenges & Conference Observations

I was part of the Dell contingent at the OpenStack design conference earlier in the month.  The conference opened a new chapter for the project because the number of contributing companies reached critical mass.  That means that the core committers are no longer employed by just one or two entities; instead, there are more moneyed interests rubbing elbows and figuring out how to collaborate.

From my perspective (from interview with @Cote ), this changed the tone of the conference from more future looking to pragmatic.

That does not mean that everything is sunshine and rainbows for OpenStack clouds, there are real issues to be resolved.  IMHO, the top issues for OpenStack are:

  1. API implementation vs specification
  2. Building up coverage on continuous integration
  3. Ensuring that we can deploy consistently in multi-node systems
  4. Getting contributions from new members
  5. Figuring out which projects are core, satellite, missing or junk.  [xref 2014 DefCore]

Of these issues, I’ve been reconsidering my position favoring API via Implementation over specification (past position).  This has been a subject of debate on my team at Dell and I like Greg Althaus’ succinct articulation of the problem with implementation driven API: “it is not fair.”  This also ends up being a branding issues for OpenStack because governance needs to figure out which is a “real” OpenStack cloud deployment that can use the brand.  Does it have to be 100% of the source?  What about extensions?  What if it uses the API with an alternate implementation?

Of the other issues, most are related to maturity.  I think #2 needs pressure by and commitment from the larger players (Dell very much included).  Crowbar and the deployment blueprint is our answer #3.  Shouting the “don’t fork it up” chorus from the roof tops addresses contributions while #5 will require some strong governance and inevitably create some hurt feelings.

Cote & Rob interview: Crowbar+OpenStack Summit/Conference Reflections (40 mins)

I’m working on a larger post about the OpenStack Summit around API Implementation vs. Specification. You can have a preview of that AND A LOT OF OTHER STUFF (OpenStack, Crowbar, lunch) in this 40 minute interview w/ Michael Cote.

Setting: Dell World
Interview w/ @Cote at the Hilton Hotel Lobby on 6th street in Austin.

I know that Cote’s post does not have a time marker for easy navigation; however, I added them to help guide your navigation in the interview (link for audio) if you want to jump around.

  • 0:00 Introductions
  • 1:00 OpenStack
    • 1:00 Essex Conference – what is it, naming conventions
    • 2:45 Diablo is adding projects from incubation (Keystone, Dashboard,Quantum,
    • 5:30 OpenStack vs. Amazon – “OpenStack has ambitions.” We see it as a “platform for innovation.”
    • 6:30 OpenStack is a competitor for Amazon. It implements the EC2 APIs.
    • 7:30 How are people managing the evolving nature?
    • 8:20 We’re going to see OpenStack in production for the next release based on what we see in our deal flow.
    • 9:00 Every user that comes on adds momentum
    • 9:30 Rackspace setting up the OpenStack foundation is a reflection of the speed of adoption
    • 11:15 Our message is “we’re doing it, we’re in the field.” We are very hands on
    • 11:15 We chose early on to focus on helping deployment to help drive adoption
    • 12:00 “Our first test for partners is: Are you contributing back to the community?”
    • 12:44 The community told us “if you are participating then you are going to open source.” Our commits for OpenStack are live and in the open on our github.
    • 13:40 Why Github? We’re happy with it.
    • 14:20 OpenStack is using Gerrit because they have a gated trunk. They are migrated to Github
    • 15:20 APIs have been a big topic for OpenStack
    • 16:00 Do you track who is forking and following? Yes. We also have a listserv. We are trying to do a better job managing the Crowbar community. We know we need to do a better job.
    • 17:30 OpenStack is defined by its Implementation. That’s “an effective way to move the project forward quickly;” however, we’re getting to a point where people want to use alternate implementations.
    • 19:20 Implementation vs Specification is like the SOAP vs REST debate
    • 20:05 This is something the community needs to wrestle with
    • 21:45 Specification would allow the efforts to scale. The more people consume the API, the more people care about how it operates
    • 22:30 “Bugs can become the API”
    • 23:10 Asia and Europe are very active. We are seeing a ton of activity overseas.
  • 23:30 Crowbar
    • 24:00 Crowbar arose out of our need to deploy cloud software regardless of customer infrastructure
    • 24:45 We would show up and the customer needed all this cloud infrastructure. We created Crowbar because we always needed this
    • 26:00 We extended Chef because we had to do the initial bring-up including BIOS and RAID
    • 26:45 We added a state machine and an orchestration layer
    • 27:45 Updating the system is a huge component. Every month you may be upgrading the infrastructure!
    • 28:30 In our lab, we build whole clouds multiple times a day
    • 29:45 Crowbar is the “cloud unboxer”
    • 30:00 We modularized Crowbar with barclamps. Hadoop and OpenStack are a series of barclamps. Over 5 for each
    • 31:00 Barclamps are applied as layers. We are using that as a term to define DevOps
    • 31:15 We are using Crowbar to help message that we understand DevOps
    • 31:45 Soup vs Sandwich analogy – Images are like soup while DevOps is like a sandwich.
    • 32:45 If you don’t want something in a 1000 server deployment, DevOps lets you make a small change. Gives you flexibility.
    • 33:45 We added Cloud Foundry
    • 34:00 We’ve made it so easy with barclamps that partners are coming to us with ideas for barclamps. It’s like “changing the meat for the sandwich.”
    • 34:30 Dreamhost Ceph team created a barclamp and was actually running a majority of the Crowbar demos at the OpenStack conference
  • 35:25 What’s the future for Crowbar?
    • 35:30 More aspects of the infrastructure as open source
    • 35:45 More Hardware
    • 36:00 Multiple operating systems at the same time (XenServer, ESX, etc)
    • 36:30 Larger scale
    • 36:50 More types of infrastructure: storage & network
    • 37:40 Scalr shout out
    • 38:00 We know we need to collaborate more with our community
    • 38:30 The first step is to download it and try. Read my blog and sign up for the list serve
    • 39:00 CROWBAR IS NOT DELL SPECIFIC – we are working with people who want to create support for other vendor’s hardware. This benefits Dell.
    • 39:40 We don’t pretend that our customers are single vendor


OpenStack Hardware Sessions Slides & Updates to “Bootstrapping Hyperscale Clouds”

Disclaimer: I work for Dell and we make very fine cloud optimized hardware.  This presentation is NOT slanted towards Dell hardware, it simply reflects lessons learned from listening to our customers.  Of course, people buying Dell servers, networking and storage products pays my salary and WE LOVE OUR CLOUD CUSTOMERS!

This presentation OpenStack Hardware Oct 2011 was the outline for our revisiting of the hyperscale white paper from last fall.  I’m in the process of updating this to reflect lessons learned.

Here are some highlights:

  1. Fault zones are not as critical as expected – customers are putting in more redundancy instead of striping apps
  2. 10 GbE is more popular than expected
  3. Logically segmented redundant networking (teamed) is more popular than physical isolate
  4. Customers are starting small (AnyScale instead of HyperScale)

Crowbar OpenStack deployment video (15 mins): Diablo + Keystone + Dashboard

This week at the OpenStack Design Summit and Conference in Boston, my team unveiled the Diablo+ Crowbar deployments. The OpenStack deployment that’s included with Crowbar reflects a collaborative effort between Dell, Opscode, and Rackspace and pulls packages from the Rackspace repository. It was important for us to use the Rackspace repos so that we could include integrated Keystone and Dashboard components that were omitted from the Diablo (current) release. Our decision to include these Essex (coming) components is based on customer feedback.

Since some of you cannot make it to the show and see the demo in person, we’ve captured it as a video for your enjoyment. The OpenStack deployment is available in our open source distribution. We are currently in QA for the overall solution so expect additional refinement as we progress towards our next OpenStack solution release.

REMINDER: Dell Hardware is NOT required to use Crowbar for OpenStack.  The open source version has everything you need – the BIOS and RAID barclamps are optional (but handy).

Shout, chat and whisper with Dell at OpenStack Design Summit & Conference

My team at Dell has been very (very) busy delivering a lot of great materials for the Fall 2011 OpenStack Design Summit & Conference in Boston MA next week.

Our motto for this conference is “DOING IS DOING” or, perhaps, “DIABLO IS DOING.”

You can count on Dell to be walking the walk with deliverables that advance OpenStack.  In fact, you can watch what we’re doing because it’s posted live as we work with the community to build it.

First, we’ll have our Crowbar demo rack showcasing LIVE MULTI-NODE DIABLO DEPLOYMENTS and some IMPORTANT FEATURE AND COMMUNITY ADDITIONS.  No spoilers here – you’ll have to come by.  Of course, it’s in the git hub too, but we’ve put a bow on it.

Second, there’s a DEPLOYMENT BLUE PRINT discussion about getting better interlocks between OpenStack development and deployment.  We really need to reduce the pain and lag between adding great features and using those features.

Next, we’ve got a limited audience CONCEPT SNEAK PEEK for something from our labs that we think is very interesting and we’d like to get input about.  Unfortunately, we’re very limited with space & time for this whisper session so you’ll need to contact OpenStack@Dell.com to request an invitation.

Finally, at the Conference, you can see OUR TEAM IN ACTION:

  • Thurs 11:30 – Dell Keynote by John Igoe
  • Thurs 3:30 – Private Cloud Panel w/ Rob Hirschfeld
  • Thurs 4:30 – Hardware Infrastructure w/ Rob Hirschfeld & Greg Althaus
  • Friday 11:00 – Deployments w/ Greg Althaus
  • Friday 3:15 – Crowbar!! w/ Scott Jensen (yes, he does it with the !!)
  • Friday 4:15 – KVM & OpenStack

More conference posts: JB Gorge & Barton George.

Dell Crowbar to deploy OpenStack Diablo Cloud

Direction in the Cloud

Photo by JB George

This week, some of the Crowbar/Dell OpenStack-Powered Cloud team, plus Matt Ray from Opscode, have been working with our partners at Rackspace in San Antonio (see Opscode post about collaboration). Our target is to have Crowbar deliver a core Diablo deployment by the October 2011 design conference (sponsored in part by Dell). This is a collaborative effort and we invite community participation – we are trying to be open and communicative (via the Crowbar listserv) while also respecting that there is a mountain of work if we are to meet deadlines.

We are doing the work in the open on the Crowbar Github so you have access to the very latest capabilities and it also means that the head the Crowbar may be unstable while we add capabilities. We feel like this is an important trade off because it allows us to keep up with the rapid pace of development in OpenStack (and other projects). This is the motivation for the recent modularization work and will continue to be a feature driver for Crowbar enhancements because it allows Crowbar users to easily bring in updated bits.

 

Don’t fork it up. OpenStack needs community collaboration

Cant we just be friends?

We’re standing on the eve of the OpenStack 4th Design summit (aka Essex) and I’m watching a frenzy of IT Goliaths (Dell, Citrix, Cisco, HP, Rackspace) and some Cloud Davids (Nebula, Stackops) try to tangle revenue streams from an open source cloud project.

I was pleased to read GigaOM‘s Derrick Harris validation of Dell’s strategy which featured my team’s contributions (Crowbar, OpenStack & Hadoop).  We are working hard to bring these technologies to our customers in an open and collaborative way.

Dell has substantial IT assets to bring to bear on cloud solutions.  All of them are ultimately tied to products that generate revenue for Dell; however, that does not prevent our being able to collaborate and share.  On the contrary, we benefiting from input from our partners, customers and community to determine which features are needed to accelerate adoption.  Our recent decision to accelerate Crowbar modularization is a clear example of that process.

It is essential to understand that this is not just about cloud technologies!  It is about the collaborative way we are promoting them and the processes we are using to deliver them.

With Dell’s cloud moving at hurricane speed, it has been interesting to watch how other companies are setting their own OpenStack initiatives.  It seems to me that many of these efforts involve forks from OpenStack that cannot/will not be contributed back the community.  One (but not the only) example is from HP’s Emil Sayegh who says that “HP developers … ideas will be shared…”  He does not commit to sharing HP’s code in his post.  I hope that is an oversight and not their plan.

In time, forking may be needed.  Right now, we need to focus on building a strong foundation.  Open contributions of code are the engine of that success.

Crowbar modularization work begins

I shared the following with the Crowbar listserv and wanted to post it for the larger audience.  If you want the latest on Crowbar then subscribe!

We’ve been getting questions and defects (thanks Matt Ray) about how we are going to allow you to update and add barclamps to Crowbar.  We’re working on that exact issue right now – you can watch me on the “modules” branch of the github.

NOTE TO CROWBAR FOLLOWERS: we are moving some items around in the repo!  There are “cactus” and “v1.0” tags in place so you can still build the current trees after we start the refactor.

We’ve got some big plans that I’ll outline on the list and earlier posts.

Right now, we’re working to modularize barclamps so that each one is in its own github repo.   This will allow you to pull in barclamps at build time or live on site. We’re also creating import/update routines that work for live systems to make it easier to develop barclamps.  Once again, that’s on the github modules branch. These will be exposed as rake barclamp:create[“foo”] and rake barclamp:install[“../foo”] type commands and I’ve committed to create some “how to make barclamps” videos.

That work is a prelude for a hard push on OpenStack Diablo before the design conference.  All that work will also be done in the github but the Diablo barclamps will be in independent repos from the Crowbar framework.

If you want to get started early.  80% of a barclamp effort is around the Chef Cookbooks.  Keith Hudgins with DTO did a great job writing up barclamps here: http://kb.dtosolutions.com/wiki/Deploying_the_cloudfoundry_barclamp.  We’re changing some of it to make it much easier and more modular.

OpenStack Design Summit regististration open for DEVS! See you in October

Registration for the OpenStack Conference (which my team at Dell is co-sponsoring), which is now a separate event (from the User Conference the same week) with a separate registration mechanism.

From the OpenStack List:

The Essex Design Summit is an event for the OpenStack developers community (existing and prospective coders) to gather and discuss the upcoming development cycle. It is highly technical in nature and is not targeted to the general OpenStack public, which will find the OpenStack Conference much more enjoyable.

If you think you belong in the former category, please join us. The Summit is free to attend, but with limited attendance, so we require registration.

More details at: http://wiki.openstack.org/Summit/Essex

Crowbar near-term features: increasing DevOps mojo and brewing Diablo

We’ve been so busy working on getting RHEL support ready to drop into the Crowbar repos that I have not had time to post about what’s coming next for Crowbar. The RHEL addition has required a substantial amount of work to accommodate different packaging models and capabilities. This change moves Crowbar closer to being able allow nodes’ operating systems (the allocated TFTP Boot Image) to be unique per node.

I will post more forward looking details soon but wanted to prime the pump and invite suggestions from our community.

We are tracking two major features for delivery by the OpenStack October Design Conference

  1. OpenStack Diablo Barclamps. Expect to see individual barclamps for various components like Keystone, Dashboard, Glace, Nova, Swift, etc)
  2. Barclamp versioning / connected imports. This feature will enable Crowbar to pull in the latest components for barclamps from remote repositories. I consider this a critical feature for Crowbar’s core DevOps/CloudOps capabilities and to support more community development for barclamps.

We are also working on some UI enhancements

  • Merging together the barclamps/proposals/active views into a single view
  • Enabling bulk actions for nodes (description, BIOS types, and allocate)
  • Allowing users to set node names and showing the names throughout the UI
  • More clarity on state of proposal application process (stretch goal)

I am planning to post more about our design ideas as work begins.

If you want to help with Diablo barclamps, these will be worked in the open and we’d be happy to collaborate. We’re also open to suggestion for what’s next.