Connecting the dots: Dell stays course on OpenStack private

Posted on May 23, 2013 by Rob H

When it comes to OpenStack, I don’t just work for Dell: I’m the technical lead for our OpenStack-powered private Cloud Solution and an elected director to the OpenStack Foundation board.

Frankly, the announcement of our change in public cloud strategy overshadowed our increasing level of investment in OpenStack-powered private cloud solutions (we are hiring!). Sam Greenblatt, Dell Product Group VP and Chief Architect, is very specific that the recent announcements are about increasing investment where Dell is already successful plus accelerating with new features (such as leadership in HyperV enablement).

The fact that we focused on our decision to pivot away from Dell hosted public cloud distracted from the strategic choices that we’ve been making. In the lean process that we use, pivots are a positive sign of listening and self-honesty. Sadly, that distraction led to confusion, misleading comments, and implications that Dell was dropping OpenStack or questioning OpenStack sustainability and market success.

For the record, Dell was one of the first companies to support OpenStack with supporting quotes from Forrest Norrod (Dell GM for Servers and my direct boss) way back in July 2010. Our private OpenStack based cloud, built on open source Crowbar, was the first to market 2 years ago (deploying Cactus!). We’ve been investing steadily in both fundamental improvements to OpenStack deployment and being early supporting the Grizzly release.

I am not implying that OpenStack’s future is certain (we have a lot of work to do) or that Dell OpenStack strategy will not change again; however, I know first-hand that both are on much firmer footing than some reports have implied.

Crowbar cuts OpenStack Grizzly (“pebbles”) branch & seeks community testing

Posted on May 17, 2013 by Rob H

The Crowbar team (I work for Dell) continues to drive towards “zero day” deployment readiness. Our Hadoop deployments are tracking Dell | Cloudera Hadoop-powered releases within a month and our OpenStack releases harden within three months.

During the OpenStack summit, we cut our Grizzly branch (aka “pebbles”) and switched over to the release packages. Just a reminder, we basically skipped Folsom. While we’re still tuning out issues on OpenStack Networking (OVS+GRE) setup, we’re also looking for community to start testing and tuning the Chef deployment recipes.

We’re just sprints from release; consequently, it’s time for the Crowbar/OpenStack community to come and play! You can learn Grizzly and help tune the open source Ops scripts.

While the Crowbar team has been generating a lot of noise around our Crowbar 2.0 work, we have not neglected progress on OpenStack Grizzly. We’ve been building Grizzly deploys on the 1.x code base using pull-from-source to ensure that we’d be ready for the release. For continuity, these same cookbooks will be the foundation of our CB2 deployment.

Features of Crowbar’s OpenStack Grizzly Deployments

We’ve had ~~Nova~~ Compute, ~~Glance~~ Image, ~~Keystone~~ Identity, ~~Horizon~~ Dashboard, ~~Swift~~ Object and Tempest for a long time. Those, of course, have been updated to Grizzly.
Added Block Storage
- importable Ceph Barclamp & OpenStack Block Plug-in
- Equalogic OpenStack Block Plug-in
Added ~~Quantum~~ OpenStack Network Barclamp
- Uses OVS + GRE for deployment
10 GB networking configuration
Rabbit MQ as its own barclamp
~~Swift~~ Object Barclamps made a lot of progress in Folsom that translates to Grizzly
- Apache Web Service
- Rack awareness
- HA configuration
- Distribution Report
“Under the covers” improvements for Crowbar 1.x
- Substantial improvements in how we configure host networking
- Numerous bug fixes and tweaks
Pull from Source via the Git barclamp
- Grizzly branch was switched to use Ubuntu & SUSE packages

We’ve made substantial progress, but there are still gaps. We do not have upgrade paths from Essex or Folsom. While we’ve been adding fault-tolerance features, full automatic HA deployments are not included.

Please build your own Crowbar ISO or check our new SoureForge download site then join the Crowbar List and IRC to collaborate with us on OpenStack (or Hadoop or Crowbar 2). Together, we will make this awesome.

Thanks! I’m enjoying my conversation with you

Posted on May 15, 2013 by Rob H

I write because I love to tell stories and to think about how actions we take today will impact tomorrow. Ultimately, everything here is about a dialog with you because you are my sounding board and my critic. I appreciate when people engage me about posts here and extend the conversation into other dimensions. Feel free to call me on points and question my position – that’s what this is all about.

Thank you for being at part of my blog and joining in. I’m looking forward to hearing more from you.

During the OpenStack Summit, I got to lead and participate in some excellent presentations and panels. While my theme for this summit was interoperability, there are many other items discussed.

I hope you enjoy them.

Did one of these topics stand out? Is there something I missed? Please let me know!

Parable of Lions and Elephants

Posted on May 3, 2013 by Rob H

There was once a family with two children: Barney and Bailum. Both wanted wanted to start a circus and did exactly that when they came into their inheritance. Being highly competitive, they each wanted to have the greatest show the world has ever seen.

Always ambitious, Barney wanted to start big and decided to start with elephants. To have an elephant act, Barney has a lot of planning to do. Even before acquiring the actual elephants, he had to get permits, hire handlers, arrange transport and arrange special feeding. He really had to get busy and make some plans even before he could start on the tusks of selling tickets.

Bailum, more humble, decided to start by training some stray dogs into an animal act. While not nearly as exciting as elephants, she was able to procure dogs immediately and start training them. Instead of having to host her own shows, she was able to bring the dogs into other people’s shows. That let her gain critical experience, get a reputation and even have positive cash flow.

Barney was merciless about Bailum’s flea bag circus. Barney was 100% confident that his vision of a grand circus was the right plan because that’s what he saw from going to other shows. Based on her behind the scenes experience, Bailum was starting to learn that running a circus was a lot more than the animal shows. Some of those tasks, like booking venues, selling ads and clown discipline, made cleaning up after elephants look like a circus highlight.

As time went on, Bailum extended her expertise with dogs into lions, horses and even bipedal simians. Her business was thriving as a specialist for other circuses to such an extent that she ~~abandoned~~ adjusted her original ringmaster vision and embraced a new plan as an animal specialist. Based on her discussions with her circus partners, her limited scope as a lion trainer was more profitable than their lives in the spotlight.

Meanwhile, Barney was still working out the issues with his elephants. It seemed like every time he turned around there was a new complication. After spending every penny on getting his glorious African pachyderms he discovered that his cages were sized for Indian elephants (which are smaller). Out of money and unable to operate, Barney had to abandon his vision and go back to clown school.

It’s hard to eat an elephant, but if you start with something you can handle then you can learn to tame lions.

We need better Gold Member criteria to help building OpenStack culture

Posted on May 3, 2013 by Rob H

During last OpenStack board meeting, we started a dialog that will be continued over the rest of the year. It concerns how/if we should apply our criteria to measure the contributions of companies that are applying to become Gold members.

I believe that we should see many contribution “footprints” for companies in Foundation leadership positions. These footprints do not have to be code in github: there are many visible ways to contribute to OpenStack including internal installs, delivered product, community meetups, open source support around code, service to the community through speaking and sponsoring and, of course, code too.

At this point in the OpenStack evolution, there is so much going on that it is easy to leave footprints because there are so many ways to engage. Footprints are tangible evidence of community leadership and the currency of collaboration. OpenStack thrives because we are committed to working together, being transparent in our actions and providing service to the project beyond our own needs.

I believe OpenStack Foundation’s new gold members will are great additions to our growing community; however, we need to be increasingly deliberate in accepting new Gold members to make sure that they have a history of demonstrating a culture of open source leadership and contribution.

These applications deserve careful consideration for several reasons:

there are a limited number of gold level positions (16 of the 24 are now occupied)
there is no practical way to remove a gold member (but only 8 are elected to the board)
there is a perception (by the applicants) that they gain additional credibility through gold membership
gold and platinum members are the leaders of our community so everyone will models their behavior

It is important to remember that there is no limit or barrier (beyond $) to joining at the corporate sponsors level. So, being a gold member means that companies are seeking a broader leadership role in the project.

Over the next months, Simon Anderson (committee chair, Dreamhost) will be leading me and several other board members in an effort to refine of our Gold member review criteria. I’ll post own list shortly and I’m interested in hearing from you about what type of “footprints” we should be considered in this process.

7 takeaways from DevOps Days Austin

Posted on May 2, 2013 by Rob H

I spent Tuesday and Wednesday at DevOpsDays Austin and continue to be impressed with the enthusiasm and collaborative nature of the DOD events. We also managed to have a very robust and engaged twitter backchannel thanks to an impressive pace set by Gene Kim!

I’ve still got a 5+ post backlog from the OpenStack summit, but wanted to do a quick post while it’s top of mind.

My takeaways from DevOpsDays Austin:

DevOpsDays spends a lot of time talking about culture. I’m a huge believer on the importance of culture as the foundation for the type of fundamental changes that we’re making in the IT industry; however, it’s also a sign that we’re still in the minority if we have to talk about culture evangelism.
Process and DevOps are tightly coupled. It’s very clear that Lean/Agile/Kanban are essential for DevOps success (nice job by Dominica DeGrandis). No one even suggested DevOps+Waterfall as a joke (but Patrick Debois had a picture of a xeroxed butt in his preso which is pretty close).
Still need more Devs people to show up! My feeling is that we’ve got a lot of operators who are engaging with developers and fewer developers who are engaging with operators (the “opsdev” people).
Chef Omnibus installer is very compelling. This approach addresses issues with packaging that were created because we did not have configuration management. Now that we have good tooling we separate the concerns between bits, configuration, services and dependencies. This is one thing to watch and something I expect to see in Crowbar.
The old mantra still holds: If something is hard, do it more often.
Eli Goldratt’s The Goal is alive again thanks to Gene Kims’s smart new novel, The Phoenix project, about DevOps and IT (I highly recommend both, start with Kim).
Not DevOps, but 3D printing is awesome. This is clearly a game changing technology; however, it takes some effort to get right. Dell brought a Solidoodle 3D printer to the event to try and print OpenStack & Crowbar logos (watch for this in the future).

I’d be interested in hearing what other people found interesting! Please comment here and let me know.

OpenStack steps toward Interopability with Temptest, RAs & RefStack.org

Posted on April 26, 2013 by Rob H

I’m a cautious supporter of OpenStack leading with implementation (over API specification); however, it clearly has risks. OpenStack has the benefit of many live sites operating at significant scale. The short term cost is that those sites were not fully interoperable (progress is being made!). Even if they were, we are lack the means to validate that they are.

The interoperability challenge was a major theme of the Havana Summit in Portland last week (panel I moderated) . Solving it creates significant benefits for the OpenStack community. These benefits have significant financial opportunities for the OpenStack ecosystem.

This is a journey that we are on together – it’s not a deliverable from a single company or a release that we will complete and move on.

There were several themes that Monty and I presented during Heat for Reference Architectures (slides). It’s pretty obvious that interop is valuable (I discuss why you should care in this earlier post) and running a cloud means dealing with hardware, software and ops in equal measures. We also identified lots of important items like Open Operations, Upstreaming, Reference Architecture/Implementation and Testing.

During the session, I think we did a good job stating how we can use Heat for an RA to make incremental steps. and I had a session about upgrade (slides).

Even with all this progress, Testing for interoperability was one of the largest gaps.

The challenge is not if we should test, but how to create a set of tests that everyone will accept as adequate. Approach that goal with standardization or specification objective is likely an impossible challenge.

Joshua McKenty & Monty Taylor found a starting point for interoperability FITS testing: “let’s use the Tempest tests we’ve got.”

We should question the assumption that faithful implementation test specifications (FITS) for interoperability are only useful with a matching specification and significant API coverage. Any level of coverage provides useful information and, more importantly, visibility accelerates contributions to the test base.

I can speak from experience that this approach has merit. The Crowbar team at Dell has been including OpenStack Tempest as part of our reference deployment since Essex and it runs as part of our automated test infrastructure against every build. This process does not catch every issue, but passing Tempest is a very good indication that you’ve got the a workable OpenStack deployment.

Rob Hirschfeld

Agile in the Clouds