Patchwork Onion delivers stability & innovation: the graphics that explains how we determine OpenStack Core

This post was coauthored by the DefCore chairs, Rob Hirschfeld & Joshua McKenty.

The OpenStack board, through the DefCore committee, has been working to define “core” for commercial users using a combination of minimum required capabilities (APIs) and code (Designated Sections).  These minimums are decided on a per project basis so it can be difficult to visualize the impact on the overall effect on the Integrated Release.

Patchwork OnionWe’ve created the patchwork onion graphic to help illustrate how core relates to the integrated release.  While this graphic is pretty complex, it was important to find a visual way to show how different DefCore identifies distinct subsets of APIs and code from each project.  This graphic tries to show how that some projects have no core APIs and/or code.

For OpenStack to grow, we need to have BOTH stability and innovation.  We need to give clear guidance to the community what is stable foundation and what is exciting sandbox.  Without that guidance, OpenStack is perceived as risky and unstable by users and vendors. The purpose of defining “Core” is to be specific in addressing that need so we can move towards interoperability.

Interoperability enables an ecosystem with multiple commercial vendors which is one of the primary goals of the OpenStack Foundation.

Ecosystem OnionOriginally, we thought OpenStack would have “core” and “non-core” projects and we baked that expectation into the bylaws.  As we’ve progressed, it’s clear that we need a less binary definition.  Projects themselves have a maturity cycle (ecosystem -> incubated -> integrated) and within the project some APIs are robust and stable while others are innovative and fluctuating.

Encouraging this mix of stabilization and innovation has been an important factor in our discussions about DefCore.  Growing the user base requires encouraging stability and growing the developer base requires enabling innovation within the same projects.

The consequence is that we are required to clearly define subsets of capabilities (APIs) and implementation (code) that are required within each project.  Designating 100% of the API or code as Core stifles innovation because stability dictates limiting changes while designating 0% of the code (being API only) lessens the need to upstream.  Core reflects the stability and foundational nature of the code; unfortunately, many people incorrectly equate “being core” with the importance of the code, and politics ensues.

To combat the politics, DefCore has taken a transparent, principles-based approach to selecting core.   You can read about in Rob’s upcoming “Ugly Babies” post (check back on 8/14) .

Community dialogue around “What is Core” positions

THIS POST IS #5 IN A SERIES ABOUT “WHAT IS CORE.”

by Rob Hirschfeld (cc) w/ attribution

The OpenStack Foundation Board has been having a broadening conservation about this topic.  Feeling left out?  Please don’t be!  Now is the time to start getting involved: we had to start very narrowly focused to avoid having the discussion continue to go in circles.  As we’ve expanding the dialog, we have incorporated significant feedback to drive consensus.

No matter where I go, people are passionate about the subject of OpenStack Core.

Overall, there is confusion of scope covered by “what is core” because people bring in their perspective from public, private solution, ecosystem or internal deployment objectives.  In discussion, everyone sees that we have to deal with issues around the OpenStack mark and projects first, but they are impatient to get into the deep issues.  Personally, we can get consensus on core and will always have a degree of healthy tension between user types.

The following are my notes, not my opinions.  I strive to faithfully represent a wide range of positions here.  Clarifications, comments and feedback are welcome!

Consensus Topics:

  • Reference/Alternate Implementation (not plug-in): Not using “plug-ins” to describe the idea that OpenStack projects should have a shared API with required code and clearly specified areas where code is replaceable.  It is the Technical Committee (TC) that makes these decisions.  The most meaningful language around this point is to say that OpenStack will have an open reference implementation with allowable alternate implementations.
  • Alternate implementations are useful:  We want to ensure upstream contribution and collaboration on the code base.  Reference implementations ensure that there’s a reason to keep open source OpenStack strong.  Alternate Implementations are important to innovation.
  • Small vs. Large Core: This is an ongoing debate about if OpenStack should have a lot of projects as part of core.  We don’t have an answer but people feel like we’re heading in a direction that resolves this question.
  • Everyone likes tests: We’re heading towards a definition of core that relies heavily on tests.  Everyone expresses concerns that this will place a lot of stress on Tempest (or another framework) and that needs to be addressed as we move forward.

Open Topics:

  • Monolithic vs. Granular Trademark:  We did not discuss if vendors will be able to claim OpenStack trademarks on subcomponents of the whole.  This is related to core but wide considered secondary.
  • API vs. implementation tension:  We accept that OpenStack will lead with implementation.   There’s no official policy that “we are not a standards body” but we may also have to state that tests are not a specification.  There’s a danger that tests will be considered more than they are.  What are they?  “They are an implementation and a source of information.  They are not the definition.”   We expect to have a working model that drives the API not vice versa.
  • Brouhaha about EC2 APIs:  It’s not clear if defining core helps address the OpenStack API discussion.  I hope it will but have not tested it.
  • Usability as core: I had many people insist that usability and ease of use should be as requirements for core because it supports adoption.  Our current positions do not have any statements to support this view.
  • Toxic neighbors: We have not discussed if use of the mark and criteria could be limited by what else you put in your product.  Are there implementation options that we’d consider toxic and automatically violate the mark?  Right now, the positions are worded that if you pass then you play even if you otherwise stink.
  • Which tests are required?  It appears that we’re moving towards using must-pass tests to define the core.  Moving towards tests determining core, we want actual field data to drive which tests are required. That will allow actual user experience to shape which tests are important rather than having it be a theoretical decision.  There’s some interest in asking the User Committee (UC) to recommend which tests are required.  This would be an added responsibility for the UC and needs more discussion.
  • Need visualization:  With 12 positions so far, it’s getting hard to keep it all together.  I’ve taken on an action item to create a diagram that shows which statements apply to which projects against the roles of ownership.

I’ve had some great discussions about core and am looking forward to many more.  I hope these notes help bring you up to speed.   As always, comments and discussion are welcome!

READ POST #6: VISUALIZING CORE