Introducing the OpenStack Spider Graph: untangling our web of interdependencies

This post is #2 in a series about “what is core.”

peaceOne of the OpenStack Foundation’s most fundamental responsibilities is to define “what is core” for the OpenStack project.  A simple yet urgent question with profound implications for the project’s success; consequently, it’s no surprise that finding a consensus answer has been a thorny problem.   In fact, the answer (it’s 42, of course) is so convoluted that we had to step back establish common ground before we could proceed. I’m proud to say that we’ve made substantial progress towards establishing a framework for the discussion.   

This post (#2) is about the process we followed.  My next post (#3) is about what we learned.  Next, we want to engage the community in discussing the baseline positions (post #4) that make up the framework.   The challenge in discussion was that the Board (and Community) lacked common baseline.  When one person talked about commercial extensions to core another would insist that core always be open.  There are compatible statements; yet, we’d often talk past each other because the question was simply too big and fuzzy.

spider boardAlan Clark, Foundation Board Chairman, and I took on the task of breaking the logjam by decomposing the problem.  We used a multilayered mind map that I called the “spider” chart (yes, it’s a misnomer).  In constructing the spider, we first flagged out issue areas (the nodes) and added goals (red text) for each issue area.  Next we linked connected nodes together to form a graph.  We added blue text on connections where goals were opposed or in tension and green text where goals were aligned.

Note: Some of our text is controversial – it was our objective to find the tension.  If you are looking at the image, we also created some grid and flow charts to test the map.

We did not time capture the spider so it’s impossible to sweep back and see how the ideas progressed.  In general, we started at the center from “IncUp Discussion” which was the TC and Board’s collaboration, led by Alan, to define the process by which projects are promoted from incubation to integrated projects.

Some topics were easy like “API vs Implementation” and “Trademark” while “Implementation Led” and “Plug-ins” were more difficult.  Some clarification on those two items is worthwhile:

Implementation Led describes the OpenStack community for starting from working implementations rather than designing an API.  In my experience, this pattern serves the community well as long as we accept that APIs and implementations in incubated projects will take time to mature.   Using this approach gets us out of prolonged design by committee spirals.  It has serious pitfalls also – there’s no milk and honey approach. Plug-ins represented a big pot of politics in the community that was part of the challenge in defining what is core.  The obvious issue is that not all core projects use plug-ins and the ones that do take different approaches.  Even so, we felt that the concept of multiple implementations with a single API was a significant part of the opportunity (and challenge) for OpenStack core.

Ultimately, we decomposed the spider graph into the following table that shows some of our thoughts in raw form.  I’ll discuss the actual insights in my next post (#3) Continue reading

Crowbar lays it all out: RAID & BIOS configs officially open sourced

MediaToday, Dell (my employer) announced a plethora of updates to our open source derived solutions (OpenStack and Hadoop). These solutions include the latest bits (Grizzly and Cloudera) for each project. And there’s another important notice for people tracking the Crowbar project: we’ve opened the remainder of its provisioning capability.

Yes, you can now build the open version of Crowbar and it has the code to configure a bare metal server.

Let me be very specific about this… my team at Dell tests Crowbar on a limited set of hardware configurations. Specifically, Dell server versions R720 + R720XD (using WSMAN and iIDRAC) and C6220 + C8000 (using open tools). Even on those servers, we have a limited RAID and NIC matrix; consequently, we are not positioned to duplicate other field configurations in our lab. So, while we’re excited to work with the community, caveat emptor open source.

Another thing about RAID and BIOS is that it’s REALLY HARD to get right. I know this because our team spends a lot of time testing and tweaking these, now open, parts of Crowbar. I’ve learned that doing hard things creates value; however, it’s also means that contributors to these barclamps need to be prepared to get some silicon under their fingernails.

I’m proud that we’ve reached this critical milestone and I hope that it encourages you to play along.

PS: It’s worth noting is that community activity on Crowbar has really increased. I’m excited to see all the excitement.

What makes OpenStack meaningful to the market?

THIS POST IS #2 IN A SERIES ABOUT “WHAT IS CORE.”

What gives a project a strong core? 

A strong project has utility, community, and longevity.

TelescopeUtility, community and longevity are the fundamental objectives of any project or product.  It must do something that people find useful (utility).  It’s not enough for one person to like the project, there must be a market (community).  And that useful and popular work must be sustainable over multiple “generations” (longevity).

These goals are basic.  The challenge is finding the right rules to keep OpenStack in the sustainable project zone.  Unfortunately, as an open source project, the OpenStack Foundation ultimately has very little real power (like hiring flocks of developers) to enforce use or maintenance of the code base.

The Foundation’s tools are velocity, culture, and brand.  Understanding “what is core” hones these tools to ensure they are effective.

Velocity – the rate of progress and quality of the code base.  A project at sufficient velocity is not worth forking or duplicating.  The fact that >1000 developers companies are contributing and 100s of companies are deploying OpenStack makes it profitable to remain in our community.   Make no mistake: being part of a community takes effort so there must be a return on that investment.  The foundation must ensure that commercial entities find an ROI from their participation.

Culture – open source culture strongly encourages sharing and collaboration.  I have seen that culture as a more potent force than the legalese and licenses.  While a strong culture reinforces itself, a toxic culture will rot a project like ice cream in the summerCulture maintenance is a chief foundation objective and includes fostering new users, documentation, orderly interactions and co-opetitive collaboration.

Brand – when all else fails, OpenStack can use legal means to define our brand.  This is the weakest of all the tools because the strength of the defense is only as good as the brand.  If we allow the OpenStack brand (sometimes we say it’s mark) to become weak or diluted then people have little reason to support velocity or culture.

An important insight when looking at these three control levers is that they are very different between individuals and corporations.  While individuals may highly motivated by culture they are not as motivated by brand; conversely, corporations are highly motivated by brand and compliance and minimally by culture.

As the OpenStack Foundation Board takes up the “what is core” question, we must be cognizant of the duality between individual and corporate interests.  OpenStack must be both meaningful culturally to individuals and strong brand-wise to corporations.   Both are needed to sustain OpenStack’s velocity.

READ POST IS #2: SPIDER CHART

Kicking off discussion about OpenStack Core

What's in?I’ve been leading an effort with Alan Clark to define “what is OpenStack core” for the Foundation Board.  Now that I am sitting here at OSCON and celebrating OpenStack’s third birthday, I think it’s a great time to bring the general community into the discussion.

There is significant history behind this topic.  According to Foundation governance, the Technical Committee (TC) defines which incubated projects are integrated and the Board of Directors (I am one) determines which of the integrated projects are labeled as core.

When it comes to the core label, the stakes are high.

Defining core is a convoluted topic.  To make it digestible, I’m breaking it down into multiple blog posts over a series of weeks:

  1. Why do we care about core?
  2. Decomposing the problem (“the spider”)
  3. Insights from decomposition (healthy tensions in OpenStack)
  4. 12 Positions: A Common Framework (I recommend ready the list of 10 instead)
  5. Community Feedback at OSCON 
  6. “What is Core” the visualization
  7. Where I think this is going, OpenStack’s Test Driven Core
  8. Core Positions Refined: the 10 positions behind the core visualization (above).
  9. Videos (most >90 mins).  The online meetups are easier to follow.
    1. 9/5 OpenStack Core Meetup in San Antonio
    2. 9/22 OpenStack Core Meetup in NYC
    3. online: 10/16 Online Meetup (daytime) and 10/22 Online Meetup (evening)
  10. Thinking about how to Implement OpenStack Core Definition

Too much reading?  At OSCON, Rafael Knuth shot a video of me talking about “what is core.”

In scale-out infrastructure, tools & automation matter

WiseScale out platforms like Hadoop have different operating rules.  I heard an interesting story today in which the performance of the overall system was improved 300% (run went from 15 mins down to 5 mins) by the removal of a node.

In a distributed system that coordinates work between multiple nodes, it only takes one bad node to dramatically impact the overall performance of the entire system.

Finding and correcting this type of failure can be difficult.  While natural variability, hardware faults or bugs cause some issues, the human element is by far the most likely cause.   If you can turn down noise injected by human error then you’ve got a chance to find the real system related issues.

Consequently, I’ve found that management tooling and automation are essential for success.  Management tools help diagnose the cause of the issue and automation creates repeatable configurations that reduce the risk of human injected variability.

I’d also like to give a shout out to benchmarks as part of your tooling suite.  Without having a reasonable benchmark it would be impossible to actually know that your changes improved performance.

Teaming Related Post Script: In considering the concept of system performance, I realized that distributed human systems (aka teams) have a very similar characteristic.  A single person can have a disproportionate impact on overall team performance.

SUSE Cloud powered by OpenStack > Demo using Crowbar

OpenStack booth at HostingConAs much as I love talking about Crowbar and OpenStack, it’s even more fun to cheer for other people doing it!

SUSE’s be a great development partner for Crowbar and an active member of the OpenStack community.  I’m excited to see them giving a live demo today about their OpenStack technology stack (which includes Crowbar and Ceph).

Register for the Live Demo on Wed 06-26-2013 at 3.00 – 4.00 pm GMT to “learn about SUSE’s OpenStack distribution: SUSE Cloud with Dell Crowbar as the deployment mechanism and advanced features such as Ceph unified storage platform for object, block and file storage in the cloud.”

The presenter, Rick Ashford, lives in Austin and is a regular at the OpenStack Austin Meetups.  He has been working with Linux and open-source software since 1998 and currently specializes in the OpenStack cloud platform and the SUSE ecosystem surrounding it.

my lean & open source reading list – recommendations welcome!

Cube Seat

I think it’s worth pulling together a list of essential books that I think should be required reading for people on Lean & open source teams (like mine):

  • Basis for the team values that we practice: The Five Dysfunctions of a Team: A Leadership Fable Patrick Lencioni (amazon)
  • This is a foundational classic for team building:  Peopleware: Productive Projects and Teams (Second Edition) Tom DeMarco (amazon)
  • This novel is good primer for lean and devops The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win Gene Kim and George Spafford & Kevin Behr (amazon)
  • Business Focus on Lean: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses Eric Ries (amazon)
  • Foundational (and easy) reading about Lean: The Goal: A Process of Ongoing Improvement Eliyahu M. Goldratt (amazon)
  • One of my favorites on Lean / Agile: Implementing Lean Software Development: From Concept to Cash Mary Poppendieck (amazon)
  • Should be required reading for open source (as close to “Open Source for Dummies as you can get): The Cathedral & the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary Eric S. Raymond (amazon)
  • Culture Change Liquid Leadership: From Woodstock to Wikipedia–Multigenerational Management Ideas That Are Changing the Way We Run Things Brad Szollose (amazon)
  • More Team Building – this one is INTERACTIVE! http://www.strengthsfinder.com/home.aspx

There are some notable additions, but I think this is enough for now.  I’m always looking for recommendations!  Please post your favorites in the comments!

OpenStack leaders learning by humility, doing and being good partners

With the next OpenStack Board meeting on Thursday (5/30/13 agenda) and Mark McLoughlin’s notes crossing my desk, I was reminded of still open discussion topics around OpenStack leadership.  Reminder: except for executive sessions, OpenStack Board Meetings are open (check agenda for details).

2013-03-11_20-01-50_458

Many of the people and companies involved in OpenStack are new to open source projects. Before OpenStack, I had no direct experience building a community like we’ve built together with OpenStack or I’ve been leading with Crowbar. There is no Collaborative Open Source Communities for Dummies book (I looked).

I am not holding myself, OpenStack or Crowbar up as shining examples of open source perfection. Just the opposite, we’ve had to learn the hard way about what works and what fails. I attribute our successes to humility to accept feedback and willingness to ask for help.

But being successful in the small (like during OpenStack Cactus) is different than where we are heading.  In the small, everyone was an open source enthusiast and eager collaborator.  In the large, we should be asking the question “how will we teach people to join and build an open source community?”

The answer is that collaboration must be modeled by the OpenStack leadership.

At the Summit, I was talking with fellow board director Sean Roberts (Yahoo!) and I think he made this point very simply:

“Being in open source is a partnership. If you don’t bring something to the partnership then you’re a user not a partner. We love users but we need to acknowledge the difference.” (Sean Roberts, OpenStack Director)

OpenStack will succeed by building a large base of users; consequently, we need our leaders to be partners in the community.

Connecting the dots: Dell stays course on OpenStack private

rob pdx drivingWhen it comes to OpenStack, I don’t just work for Dell: I’m the technical lead for our OpenStack-powered private Cloud Solution and an elected director to the OpenStack Foundation board.

Frankly, the announcement of our change in public cloud strategy overshadowed our increasing level of investment in OpenStack-powered private cloud solutions (we are hiring!).  Sam Greenblatt, Dell Product Group VP and Chief Architect, is very specific that the recent announcements are about increasing investment where Dell is already successful plus accelerating with new features (such as leadership in HyperV enablement).

The fact that we focused on our decision to pivot away from Dell hosted public cloud distracted from the strategic choices that we’ve been making.  In the lean process that we use, pivots are a positive sign of listening and self-honesty.  Sadly, that distraction led to confusion, misleading comments, and implications that Dell was dropping OpenStack or questioning OpenStack sustainability and market success.

For the record, Dell was one of the first companies to support OpenStack with supporting quotes from Forrest Norrod (Dell GM for Servers and my direct boss) way back  in July 2010.  Our private OpenStack based cloud, built on open source Crowbar, was the first to market 2 years ago (deploying Cactus!).  We’ve been investing steadily in both fundamental improvements to OpenStack deployment and being early supporting the Grizzly release.

I am not implying that OpenStack’s future is certain (we have a lot of work to do) or that Dell OpenStack strategy will not change again; however, I know first-hand that both are on much firmer footing than some reports have implied.

Crowbar cuts OpenStack Grizzly (“pebbles”) branch & seeks community testing

Pebbles CutThe Crowbar team (I work for Dell) continues to drive towards “zero day” deployment readiness. Our Hadoop deployments are tracking Dell | Cloudera Hadoop-powered releases within a month and our OpenStack releases harden within three months.

During the OpenStack summit, we cut our Grizzly branch (aka “pebbles”) and switched over to the release packages. Just a reminder, we basically skipped Folsom. While we’re still tuning out issues on OpenStack Networking (OVS+GRE) setup, we’re also looking for community to start testing and tuning the Chef deployment recipes.

We’re just sprints from release; consequently, it’s time for the Crowbar/OpenStack community to come and play! You can learn Grizzly and help tune the open source Ops scripts.

While the Crowbar team has been generating a lot of noise around our Crowbar 2.0 work, we have not neglected progress on OpenStack Grizzly.  We’ve been building Grizzly deploys on the 1.x code base using pull-from-source to ensure that we’d be ready for the release. For continuity, these same cookbooks will be the foundation of our CB2 deployment.

Features of Crowbar’s OpenStack Grizzly Deployments

  • We’ve had Nova Compute, Glance Image, Keystone Identity, Horizon Dashboard, Swift Object and Tempest for a long time. Those, of course, have been updated to Grizzly.
  • Added Block Storage
    • importable Ceph Barclamp & OpenStack Block Plug-in
    • Equalogic OpenStack Block Plug-in
  • Added Quantum OpenStack Network Barclamp
    • Uses OVS + GRE for deployment
  • 10 GB networking configuration
  • Rabbit MQ as its own barclamp
  • Swift Object Barclamps made a lot of progress in Folsom that translates to Grizzly
    • Apache Web Service
    • Rack awareness
    • HA configuration
    • Distribution Report
  • “Under the covers” improvements for Crowbar 1.x
    • Substantial improvements in how we configure host networking
    • Numerous bug fixes and tweaks
  • Pull from Source via the Git barclamp
    • Grizzly branch was switched to use Ubuntu & SUSE packages

We’ve made substantial progress, but there are still gaps. We do not have upgrade paths from Essex or Folsom. While we’ve been adding fault-tolerance features, full automatic HA deployments are not included.

Please build your own Crowbar ISO or check our new SoureForge download site then join the Crowbar List and IRC to collaborate with us on OpenStack (or Hadoop or Crowbar 2). Together, we will make this awesome.