Introducing the OpenStack Spider Graph: untangling our web of interdependencies

This post is #2 in a series about “what is core.”

peaceOne of the OpenStack Foundation’s most fundamental responsibilities is to define “what is core” for the OpenStack project.  A simple yet urgent question with profound implications for the project’s success; consequently, it’s no surprise that finding a consensus answer has been a thorny problem.   In fact, the answer (it’s 42, of course) is so convoluted that we had to step back establish common ground before we could proceed. I’m proud to say that we’ve made substantial progress towards establishing a framework for the discussion.   

This post (#2) is about the process we followed.  My next post (#3) is about what we learned.  Next, we want to engage the community in discussing the baseline positions (post #4) that make up the framework.   The challenge in discussion was that the Board (and Community) lacked common baseline.  When one person talked about commercial extensions to core another would insist that core always be open.  There are compatible statements; yet, we’d often talk past each other because the question was simply too big and fuzzy.

spider boardAlan Clark, Foundation Board Chairman, and I took on the task of breaking the logjam by decomposing the problem.  We used a multilayered mind map that I called the “spider” chart (yes, it’s a misnomer).  In constructing the spider, we first flagged out issue areas (the nodes) and added goals (red text) for each issue area.  Next we linked connected nodes together to form a graph.  We added blue text on connections where goals were opposed or in tension and green text where goals were aligned.

Note: Some of our text is controversial – it was our objective to find the tension.  If you are looking at the image, we also created some grid and flow charts to test the map.

We did not time capture the spider so it’s impossible to sweep back and see how the ideas progressed.  In general, we started at the center from “IncUp Discussion” which was the TC and Board’s collaboration, led by Alan, to define the process by which projects are promoted from incubation to integrated projects.

Some topics were easy like “API vs Implementation” and “Trademark” while “Implementation Led” and “Plug-ins” were more difficult.  Some clarification on those two items is worthwhile:

Implementation Led describes the OpenStack community for starting from working implementations rather than designing an API.  In my experience, this pattern serves the community well as long as we accept that APIs and implementations in incubated projects will take time to mature.   Using this approach gets us out of prolonged design by committee spirals.  It has serious pitfalls also – there’s no milk and honey approach. Plug-ins represented a big pot of politics in the community that was part of the challenge in defining what is core.  The obvious issue is that not all core projects use plug-ins and the ones that do take different approaches.  Even so, we felt that the concept of multiple implementations with a single API was a significant part of the opportunity (and challenge) for OpenStack core.

Ultimately, we decomposed the spider graph into the following table that shows some of our thoughts in raw form.  I’ll discuss the actual insights in my next post (#3) Continue reading

What makes OpenStack meaningful to the market?

THIS POST IS #2 IN A SERIES ABOUT “WHAT IS CORE.”

What gives a project a strong core? 

A strong project has utility, community, and longevity.

TelescopeUtility, community and longevity are the fundamental objectives of any project or product.  It must do something that people find useful (utility).  It’s not enough for one person to like the project, there must be a market (community).  And that useful and popular work must be sustainable over multiple “generations” (longevity).

These goals are basic.  The challenge is finding the right rules to keep OpenStack in the sustainable project zone.  Unfortunately, as an open source project, the OpenStack Foundation ultimately has very little real power (like hiring flocks of developers) to enforce use or maintenance of the code base.

The Foundation’s tools are velocity, culture, and brand.  Understanding “what is core” hones these tools to ensure they are effective.

Velocity – the rate of progress and quality of the code base.  A project at sufficient velocity is not worth forking or duplicating.  The fact that >1000 developers companies are contributing and 100s of companies are deploying OpenStack makes it profitable to remain in our community.   Make no mistake: being part of a community takes effort so there must be a return on that investment.  The foundation must ensure that commercial entities find an ROI from their participation.

Culture – open source culture strongly encourages sharing and collaboration.  I have seen that culture as a more potent force than the legalese and licenses.  While a strong culture reinforces itself, a toxic culture will rot a project like ice cream in the summerCulture maintenance is a chief foundation objective and includes fostering new users, documentation, orderly interactions and co-opetitive collaboration.

Brand – when all else fails, OpenStack can use legal means to define our brand.  This is the weakest of all the tools because the strength of the defense is only as good as the brand.  If we allow the OpenStack brand (sometimes we say it’s mark) to become weak or diluted then people have little reason to support velocity or culture.

An important insight when looking at these three control levers is that they are very different between individuals and corporations.  While individuals may highly motivated by culture they are not as motivated by brand; conversely, corporations are highly motivated by brand and compliance and minimally by culture.

As the OpenStack Foundation Board takes up the “what is core” question, we must be cognizant of the duality between individual and corporate interests.  OpenStack must be both meaningful culturally to individuals and strong brand-wise to corporations.   Both are needed to sustain OpenStack’s velocity.

READ POST IS #2: SPIDER CHART

Kicking off discussion about OpenStack Core

What's in?I’ve been leading an effort with Alan Clark to define “what is OpenStack core” for the Foundation Board.  Now that I am sitting here at OSCON and celebrating OpenStack’s third birthday, I think it’s a great time to bring the general community into the discussion.

There is significant history behind this topic.  According to Foundation governance, the Technical Committee (TC) defines which incubated projects are integrated and the Board of Directors (I am one) determines which of the integrated projects are labeled as core.

When it comes to the core label, the stakes are high.

Defining core is a convoluted topic.  To make it digestible, I’m breaking it down into multiple blog posts over a series of weeks:

  1. Why do we care about core?
  2. Decomposing the problem (“the spider”)
  3. Insights from decomposition (healthy tensions in OpenStack)
  4. 12 Positions: A Common Framework (I recommend ready the list of 10 instead)
  5. Community Feedback at OSCON 
  6. “What is Core” the visualization
  7. Where I think this is going, OpenStack’s Test Driven Core
  8. Core Positions Refined: the 10 positions behind the core visualization (above).
  9. Videos (most >90 mins).  The online meetups are easier to follow.
    1. 9/5 OpenStack Core Meetup in San Antonio
    2. 9/22 OpenStack Core Meetup in NYC
    3. online: 10/16 Online Meetup (daytime) and 10/22 Online Meetup (evening)
  10. Thinking about how to Implement OpenStack Core Definition

Too much reading?  At OSCON, Rafael Knuth shot a video of me talking about “what is core.”

We need better Gold Member criteria to help building OpenStack culture

bunny slippersDuring last OpenStack board meeting, we started a dialog that will be continued over the rest of the year.  It concerns how/if we should apply our criteria to measure the contributions of companies that are applying to become Gold members.

I believe that we should see many contribution “footprints” for companies in Foundation leadership positions.  These footprints do not have to be code in github: there are many visible ways to contribute to OpenStack including internal installs, delivered product, community meetups, open source support around code, service to the community through speaking and sponsoring and, of course, code too.

At this point in the OpenStack evolution, there is so much going on that it is easy to leave footprints because there are so many ways to engage.  Footprints are tangible evidence of community leadership and the currency of collaboration.  OpenStack thrives because we are committed to working together, being transparent in our actions and providing service to the project beyond our own needs.

I believe OpenStack Foundation’s new gold members will are great additions to our growing community; however, we need to be increasingly deliberate in accepting new Gold members to make sure that they have a history of demonstrating a culture of open source leadership and contribution.  

These applications deserve careful consideration for several reasons:

  1. there are a limited number of gold level positions (16 of the 24 are now occupied)
  2. there is no practical way to remove a gold member (but only 8 are elected to the board)
  3. there is a perception (by the applicants) that they gain additional credibility through gold membership
  4. gold and platinum members are the leaders of our community so everyone will models their behavior

It is important to remember that there is no limit or barrier (beyond $) to joining at the corporate sponsors level. So, being a gold member means that companies are seeking a broader leadership role in the project.

Over the next months, Simon Anderson (committee chair, Dreamhost) will be leading me and several other board members in an effort to refine of our Gold member review criteria.  I’ll post own list shortly and I’m interested in hearing from you about what type of “footprints” we should be considered in this process.

7 takeaways from DevOps Days Austin

Block Tables

I spent Tuesday and Wednesday at DevOpsDays Austin and continue to be impressed with the enthusiasm and collaborative nature of the DOD events.  We also managed to have a very robust and engaged twitter backchannel thanks to an impressive pace set by Gene Kim!

I’ve still got a 5+ post backlog from the OpenStack summit, but wanted to do a quick post while it’s top of mind.

My takeaways from DevOpsDays Austin:

  1. DevOpsDays spends a lot of time talking about culture.  I’m a huge believer on the importance of culture as the foundation for the type of fundamental changes that we’re making in the IT industry; however, it’s also a sign that we’re still in the minority if we have to talk about culture evangelism.
  2. Process and DevOps are tightly coupled.  It’s very clear that Lean/Agile/Kanban are essential for DevOps success (nice job by Dominica DeGrandis).  No one even suggested DevOps+Waterfall as a joke (but Patrick Debois had a picture of a xeroxed butt in his preso which is pretty close).
  3. Still need more Devs people to show up!  My feeling is that we’ve got a lot of operators who are engaging with developers and fewer developers who are engaging with operators (the “opsdev” people).
  4. Chef Omnibus installer is very compelling.  This approach addresses issues with packaging that were created because we did not have configuration management.  Now that we have good tooling we separate the concerns between bits, configuration, services and dependencies.  This is one thing to watch and something I expect to see in Crowbar.
  5. The old mantra still holds: If something is hard, do it more often.
  6. Eli Goldratt’s The Goal is alive again thanks to Gene Kims’s smart new novel, The Phoenix project, about DevOps and IT  (I highly recommend both, start with Kim).
  7. Not DevOps, but 3D printing is awesome.  This is clearly a game changing technology; however, it takes some effort to get right.  Dell brought a Solidoodle 3D printer to the event to try and print OpenStack & Crowbar logos (watch for this in the future).

I’d be interested in hearing what other people found interesting!  Please comment here and let me know.

OpenStack steps toward Interopability with Temptest, RAs & RefStack.org

Pipes are interoperableI’m a cautious supporter of OpenStack leading with implementation (over API specification); however, it clearly has risks. OpenStack has the benefit of many live sites operating at significant scale. The short term cost is that those sites were not fully interoperable (progress is being made!). Even if they were, we are lack the means to validate that they are.

The interoperability challenge was a major theme of the Havana Summit in Portland last week (panel I moderated) .  Solving it creates significant benefits for the OpenStack community.  These benefits have significant financial opportunities for the OpenStack ecosystem.

This is a journey that we are on together – it’s not a deliverable from a single company or a release that we will complete and move on.

There were several themes that Monty and I presented during Heat for Reference Architectures (slides).  It’s pretty obvious that interop is valuable (I discuss why you should care in this earlier post) and running a cloud means dealing with hardware, software and ops in equal measures.  We also identified lots of important items like Open OperationsUpstreamingReference Architecture/Implementation and Testing.

During the session, I think we did a good job stating how we can use Heat for an RA to make incremental steps.   and I had a session about upgrade (slides).

Even with all this progress, Testing for interoperability was one of the largest gaps.

The challenge is not if we should test, but how to create a set of tests that everyone will accept as adequate.  Approach that goal with standardization or specification objective is likely an impossible challenge.

Joshua McKenty & Monty Taylor found a starting point for interoperability FITS testing: “let’s use the Tempest tests we’ve got.”

We should question the assumption that faithful implementation test specifications (FITS) for interoperability are only useful with a matching specification and significant API coverage.  Any level of coverage provides useful information and, more importantly, visibility accelerates contributions to the test base.

I can speak from experience that this approach has merit.  The Crowbar team at Dell has been including OpenStack Tempest as part of our reference deployment since Essex and it runs as part of our automated test infrastructure against every build.  This process does not catch every issue, but passing Tempest is a very good indication that you’ve got the a workable OpenStack deployment.

“Stack Shop” cover of Macklemore’s Thrift Shop

Sometimes a meme glitters too strongly for me to resist getting pulled in… that happened to great effect that just before the OpenStack Havana summit. When my code-addled mind kept swapping “poppin’ tags” for “OpenStack” on the radio edit, I stopped fighting and rewrote the Thrift Shop lyrics for OpenStack (see below the split).

With a lot of help from summit attendees (many of them are OpenStack celebrities, CEOs, VPs and members OpenStack Foundation board), I was able to create a freaking awesome cover of Macklemore’s second hand confection (NSFW).

Frankly, I don’t know everyone in the video (what, what?)!

But here’s a list of those that I do know.  I’m happy to update so the victims actors get credit.  Singers (in order):

Rob Hirschfeld (me) & Monty Taylor, Peter Poulliot, Judd Maltin, Forrest Norrod, Josh Kleinpeter, Tristan Goode, Dan Bode, Jay Pipes, Prabhakar Gopalan, Peter Chadwick, Simon Andersen, Vish Ishaya, Wayne Walls, Alex Freedland, Niki Acosta, Ops Track Monday Session 1, Ben Cherian, Eric Windisch, Brandon Draeger, Joseph B George,  Mark Collier, Joseph Heck, Tim Bell,  Chris Kemp, Kyle McDonald & Joshua McKenty,

Continue reading

OpenStack’s next hurdle: Interoperability. Why should you care?

SXSW life size Newton's Cradle

SXSW life size Newton’s Cradle

The OpenStack Board spent several hours (yes, hours) discussing interoperability related topics at the last board meeting.  Fundamentally, the community benefits when uses can operate easily across multiple OpenStack deployments (their own and/or public clouds).

Cloud interoperability: the ability to transfer workloads between systems without changes to the deployment operations management infrastructure.

This is NOT hybrid (which I defined as a workload transparently operating in multiple systems); however it is a prereq to achieve scalable hybrid operation.

Interoperability matters because the OpenStack value proposition is all about creating a common platform.  IT World does a good job laying out the problem (note, I work for Dell).  To create sites that can interoperate, we have to some serious lifting:

At the OpenStack Summit, there are multiple chances to engage on this.   I’m moderating a panel about Interop and also sharing a session about the highly related topic of Reference Architectures with Monty Tayor.

The Interop Panel (topic description here) is Tuesday @ 5:20pm.  If you join, you’ll get to see me try to stump our awesome panelists

  • Jonathan LaCour, DreamHost
  • Troy Toman, Rackspace
  • Bernard Golden,  Enstratius
  • Monty Taylor, OpenStack Board (and HP)
  • Peter Pouliot, Microsoft

PS: Oh, and I’m also talking about DevOps Upgrades Patterns during the very first session (see a preview).

Don’t complicate my cloud! It’s just infrastructure with an API

Getty MazeI’ve been “in cloud” for over 13 years (@dmcrory and I submitted patents using it starting in 2001) and I’m continually amazed at how complicated people want to make it.

For my role at Dell, I’m continually invited to seasons of meetings to define cloud, cloud architecture and cloud strategy. The reason these meetings go on and on is that everyone wants to make cloud complicated when it’s really very simple.

Cloud is infrastructure with an API.

That’s it. Everything else is just a consequence of having infrastructure with an API because API provides the ability to provide remote control.

What else do people try to lump into cloud?  Here are some of my topic cloud obfuscators:

  • (inter)network.  Yes, networks make an API interesting.  They are just an essential component but they are not cloud.  Most technologies are interesting because of networks: can we stop turning everything networked into cloud?  Thanks to nonsensical mega-dollar marketing campaigns, I despair this is a moot point.
  • as-a-service.  That’s another way of saying “accessible via an API.”  We have many flavors of Platform, Data, Application, Love or whatever as a Service.  That means they have a API.  Infrastructure as a Service (IaaS) is a cloud.
  • virtualization.  VMs were the first good example of hardware with an API; however, we had virtual containers (on Mainframes!) long before we had “cloud.”    They make cloud easier but they are not cloud.
  • pay-as-you-go (service pricing).  This is a common cloud requirement but it’s really a business model.  If someone builds a private cloud then it is still a cloud.

  • multi-tenant.  Another common requirement where we expect a cloud to be able to isolate users.  I agree that this is a highly desirable attribute of a good API implementation; however, it’s not essential to a cloud.  For example, most public clouds do not have true network isolation model.
  • elastic demand.  IMHO, another word for API driven provisioning.
  • live migration.  This is a cool feature often implemented on top of virtualization, but it’s not cloud.  We were doing live migrate with shared storage and clusters before anyone heard of cloud.   I don’t think this is cloud at all but someone out there does so I included it in the list.
  • security.  Totally important consideration and required for deployments large and small, but not presence/lack does not make something cloud.

We start talking about these points and then forget the whole API thing.  These items are important, but they do not make it “a cloud.”  When Dave McCrory and I first discussed API Infrastructure as “cloud,” it was driven by the fact that you could hide the actual infrastructure behind the API.  The critical concept was that the API allowed a you to manage a server anywhere from anywhere.

When Amazon offered the first EC2 service, it had to be a cloud because the servers were remote. It was not a cloud because it was on the internet; plenty of other companies were offering hosted servers. It was a cloud because their offering allowed required operators to use and API to interact with the infrastructure.  I remember that EC2’s lack of UI (and SLA) causing many to predict it would be a failure; instead, it sparked a revolution.

I’m excited now because we’re entering a new generation of cloud where Infrastructure APIs include networking and storage in addition to compute.  Mix in some of the interesting data and network services and we’re going to have truly dynamic and powerful clouds.  More importantly, we going to have some truly amazing applications.

What do you think?  Is API a sufficient definition of cloud in your opinion?

PS: Yes, if you have a physical server/network/store that is completely controllable by an API then you’ve got a cloud on your hands.

Impressions from OpenStack Foundation board meeting 10/15

20121018-115319.jpgI spent the first day of the OpenStack summit doing my service as a Foundation board member. While you need to wait for a summary by Jonathan Bryce, the Foundation Executive Director, and/or the official minutes to get the details of our actions, I think that it’s reasonable to share my impressions.

In some ways, the chemistry of the board is as important as our actions at this point. We are still working thorough start up issues and, more importantly, learning how to work together. The board is not only big, 24 members, it is also has many companies that often compete (power of we).

Here are my top 5 impressions:

    1. Transparency. The bylaws are crafted to ensure a openness and our meetings are broadcast to the maximum extent possible. Even with that background, there is a consistent self-check and discussion about increasing transparency so that the community is included.
    2. Humor. It’s a great sign of progress that we are laughing together because it shows that we trust and respect each other.
    3. Frustration. Recognition that we have important decisions to make and a degree of impatience to make them. Boards are subtle: we spend a lot of time setting up the right structures that allow us to make hard decisions quickly. Also, grumbling that we’d overlapped the board meeting and the summit.
    4. Consensus. This something I committed to help build. I feel that our actions reflect both healthy discussion as individuals and desire to work together as a board.
    5. Leadership. Looking back from our first meeting, it is clear that Jonathan and Alan have fully assumed their roles. I’m also seeing how the Foundation team (Mark, Lauren, Stephano) are smoothly supporting operations even though we’re really just weeks from the Foundation launch.

I hope that gives you some insights into the board meeting and that more of you can join the broadcast for the next ones (2012 schedule TBD) including a joint tech and foundation meeting

Wednesday morning at the summit was “breakfast with the board.” Look for some notes from that too soon.