DefCore Advances at the Core > My take on the OSCON’14 OpenStack Board Meeting

Last week’s day-long Board Meeting (Jonathan’s summary) focused on three major topics: DefCore, Contribute Licenses (CLA/DCO) and the “Win the Enterprise” initiative. In some ways, these three topics are three views into OpenStack’s top issue: commercial vs. individual interests.

But first, let’s talk about DefCore!

DefCore took a major step with the passing of the advisory Havana Capabilities (the green items are required). That means that vendors in the community now have a Board approved minimum requirements.  These are not enforced for Havana so that the community has time to review and evaluate.

Designated Sections (1)For all that progress, we only have half of the Havana core definition complete. Designated Sections, the other component of Core, will be defined by the DefCore committee for Board approval in September. Originally, we expected the TC to own this part of the process; however, they felt it was related to commercial interested (not technical) and asked for the Board to manage it.

The coming meetings will resolve the “is Swift code required” question and that topic will require a dedicated post.  In many ways, this question has been the challenge for core definition from the start.  If you want to join the discussion, please subscribe to the DefCore list.

The majority of the board meeting was spent discussion other weighty topics that are work a brief review.

Contribution Licenses revolve around developer vs broader community challenge. This issue is surprisingly high stakes for many in the community. I see two primary issues

  1. Tension between corporate (CLA) vs. individual (DCO) control and approval
  2. Concern over barriers to contribution (sadly, there are many but this one is in the board’s controls)

Win the Enterprise was born from product management frustration and a fragmented user base. My read on this topic is that we’re pushing on the donkey. I’m hearing serious rumbling about OpenStack operability, upgrade and scale.  This group is doing a surprisingly good job of documenting these requirements so that we will have an official “we need this” statement. It’s not clear how we are going to turn that statement into either carrots or sticks for the donkey.

Overall, there was a very strong existential theme for OpenStack at this meeting: are we a companies collaborating or individuals contributing?  Clearly, OpenStack is both but the proportions remain unclear.

Answering this question is ultimately at the heart of all three primary topics. I expect DefCore will be on the front line of this discussion over the next few weeks (meeting 1, 2, and 3). Now is the time to get involved if you want to play along.

OpenStack Core Online Forum, Oct 16 13:30 UTC / Oct 22 0100 UTC

Go Online!OpenStack Community, you are invited on an online discussion about OpenStack Core on October 16th at UTC 13:30 (8:30 am US Central) and October 22nd at UTC 0100 (8:00 pm US Central)

At the next OpenStack Foundation Board meeting, we will be setting a timeline for implementing an OpenStack Core Definition process that promotes a clear and implementation driven metric for deciding which projects should be considered “required.”  This is your chance to review and influence the process!

We’ll review the OpenStack Core Definition process (20 minutes) and then open up the channel for discussion using the IRC (#openstack-meeting) & Google Hangout on Air (link posted in IRC).

The forum will be coordinated through the IRC channel for links and questions.

Can’t make it?  The session was recorded > here!

Thinking about how to Implement OpenStack Core Definition

THIS POST IS #10 IN A SERIES ABOUT “WHAT IS CORE.”

Tied UpWe’ve had a number of community discussions (OSCON, SFO & SA-TX) around the process for OpenStack Core definition.  These have been animated and engaged discussions (video from SA-TX): my notes for them are below.

While the current thinking of a testing-based definition of Core adds pressure on expanding our test suite, it seems to pass the community’s fairness checks.

Overall, the discussions lead me to believe that we’re on the right track because the discussions jump from process to impacts.  It’s not too late!  We’re continuing to get community feedback.  So what’s next?

First…. Get involved: Upcoming Community Core Discussions

These discussions are expected to have online access via Google Hangout.  Watch Twitter when the event starts for a link.

Want to to discuss this in your meetup? Reach out to me or someone on the Board and we’ll be happy to find a way to connect with your local community!

What’s Next?  Implementation!

So far, the Core discussion has been about defining the process that we’ll use to determine what is core.  Assuming we move forward, the next step is to implement that process by selecting which tests are “must pass.”  That means we have to both figure out how to pick the tests and do the actual work of picking them.  I suspect we’ll also find testing gaps that will have developers scrambling in Ice House.

Here’s the possible (aggressive) timeline for implementation:

  • November: Approval of approach & timeline at next Board Meeting
  • January: Publish Timeline for Roll out (ideally, have usable definition for Havana)
  • March: Identify Havana must pass Tests (process to be determined)
  • April: Integration w/ OpenStack Foundation infrastructure

Obviously, there are a lot of details to work out!  I expect that we’ll have an interim process to select must-pass tests before we can have a full community driven methodology.

Notes from Previous Discussions (earlier notes):

  • There is still confusion around the idea that OpenStack Core requires using some of the project code.  This requirement helps ensure that people claiming to be OpenStack core have a reason to contribute, not just replicate the APIs.
  • It’s easy to overlook that we’re trying to define a process for defining core, not core itself.  We have spent a lot of time testing how individual projects may be effected based on possible outcomes.  In the end, we’ll need actual data.
  • There are some clear anti-goals in the process that we are not ready to discuss but will clearly going to become issues quickly.  They are:
    • Using the OpenStack name for projects that pass the API tests but don’t implement any OpenStack code.  (e.g.: an OpenStack Compatible mark)
    • Having speciality testing sets for flavors of OpenStack that are different than core.  (e.g.: OpenStack for Hosters, OpenStack Private Cloud, etc)
  • We need to be prepared that the list of “must pass” tests identifies a smaller core than is currently defined.  It’s possible that some projects will no longer be “core”
  • The idea that we’re going to use real data to recommend tests as must-pass is positive; however, the time it takes to collect the data may be frustrating.
  • People love to lobby for their favorite projects.  Gaps in testing may create problems.
  • We are about to put a lot of pressure on the testing efforts and that will require more investment and leadership from the Foundation.
  • Some people are not comfortable with self-reporting test compliance.   Overall, market pressure was considered enough to punish cheaters.
  • There is a perceived risk of confusion as we migrate between versions.  OpenStack Core for Havana seems to specific but there is concern that vendors may pass in one release and then skip re-certification.  Once again, market pressure seems to be an adequate answer.
  • It’s not clear if a project with only 1 must-pass test is a core project.  Likely, it would be considered core.  Ultimately, people seem to expect that the tests will define core instead of the project boundary.

What do you think?  I’d like to hear your opinions on this!

Taking OpenStack Core discussions to community

core flowTHIS POST IS #9 IN A SERIES ABOUT “WHAT IS CORE.”

We’ve been building up to a broad discussion about the OpenStack Core and I’d like to invite everyone in the OpenStack community to participate (review latest).

Alan Clark (Board Chairman) officially kicked off this open discussion with his post on the OpenStack blog last week.  And we’re trying to have face-to-face events for dialog like the Core meetup tonight in San Francisco.  Look for more to come!

Of course, this will also be a topic at the summit (Alan and I submitted two sessions about this).  The Board needs to move this forward in the November meeting, so NOW is the time to review and give us input.

refined: 10 OpenStack Core Positions

core flowTHIS POST IS #8 IN A SERIES ABOUT “WHAT IS CORE.”

Last week, I posted a streamlined visual of the core discussion that distilled the 12 positions into 10.  Here are reordered and cleaned up matching positions.  This should make it much easier to understand the context.

Note 11/3: The Core Definition is now maintained on the OpenStack Wiki.  This list may not reflect the latest changes.
  1. Implementations that are Core can use OpenStack trademark (OpenStack™)

    1. This is the legal definition of “core” and the  why it matters to  the community.

    2. We want to make sure that the OpenStack™ mark means something.

    3. The OpenStack™ mark is not the same as the OpenStack brand; however, the Board uses it’s control of the mark as a proxy to help manage the brand.

  2. Core is a subset of the whole project

    1. The OpenStack project is supposed to be a broad and diverse community with new projects entering incubation and new implementations being constantly added.  This innovation is vital to OpenStack but separate from the definition of Core.

    2. There may be other marks that are managed separately by the foundation, and available for the platform ecosystem as per the Board’s discretion

    3. “OpenStack API Compatible ” mark not part of this discussion and should be not be assumed.

  3. Core definition can be applied equally to all usage models

    1. There should not be multiple definitions of OpenStack depending on the operator (public, private, community, etc)

    2. While expected that each deployment is identical, the differences must be quantifiable

  4. Claiming OpenStack requiring use of designated upstream code

    1. Implementation’s claiming the OpenStack™ mark must use the OpenStack upstream code (or be using code submitted to upstream)

    2. You are not OpenStack, if you pass all the tests but do not use the API framework

    3. This prevents people from using the API without joining the community

    4. This also surfaces bit-rot in alternate implementations to the larger community

    5. This behavior improves interoperability because there is more shared code between implementation

  5. Projects must have an open reference implementation

    1. OpenStack will require an open source reference base plug-in implementation for projects (if not part of OpenStack, license model for reference plug-in must be compatible).

    2. Definition of a plug-in: alternate backend implementations with a common API framework that uses common _code_ to implement the API

    3. This expects that projects (where technically feasible) are expected to implement a plug-in or extension architecture.

    4. This is already in place for several projects and addresses around ecosystem support, enabling innovation

    5. Reference plug-ins are, by definition, the complete capability set.  It is not acceptable to have “core” features that are not functional in the reference plug-in

    6. This will enable alternate implementations to offer innovative or differentiated features without forcing changes to the reference plug-in implementation

    7. This will enable the reference to expand without forcing other  alternate implementations to match all features and recertify

  6. Vendors may substitute alternate implementations

    1. If a vendor plug-in passes all relevant tests then it can be considered a full substitute for the reference plug-in

    2. If a vendor plug-in does NOT pass all relevant test then the vendor is required to include the open source reference in the implementation.

    3. Alternate implementations may pass any tests that make sense

    4. Alternate implementations should add tests to validate new functionality.

    5. They must have all the must-pass tests (see #10) to claim the OpenStack mark.

  7. OpenStack Implementations are verified by open community tests

    1. Vendor OpenStack implementations must achieve 100% of must-have coverage?

    2. Implemented tests can be flagged as may-have requires list  [Joshua McKenty]

    3. Certifiers will be required to disclose their testing gaps.

    4. This will put a lot of pressure on the Tempest project

    5. Maintenance of the testing suite to become a core Foundation responsibility.  This may require additional resources

    6. Implementations and products are allowed to have variation based on publication of compatibility

    7. Consumers must have a way to determine how the system is different from reference (posted, discovered, etc)

    8. Testing must respond in an appropriate way on BOTH pass and fail (the wrong return rejects the entire suite)

  8. Tests can be remotely or self-administered

    1. Plug-in certification is driven by Tempest self-certification model

    2. Self-certifiers are required to publish their results

    3. Self-certified are required to publish enough information that a 3rd party could build the reference implementation to pass the tests.

    4. Self-certified must include the operating systems that have been certified

    5. It is preferred for self-certified implementation to reference an OpenStack reference architecture “flavor” instead of defining their own reference.  (a way to publish and agree on flavors is needed)

    6. The Foundation needs to define a mechanism of dispute resolution. (A trust but verify model)

    7. As an ecosystem partner, you have a need to make a “works against OpenStack” statement that is supportable

    8. API consumer can claim working against the OpenStack API if it works against any implementation passing all the “must have” tests(YES)

    9. API consumers can state they are working against the OpenStack API with some “may have” items as requirements

    10. API consumers are expected to write tests that validate their required behaviors (submitted as “may have” tests)

  9. A subset of tests are chosen by the Foundation as “must-pass”

    1. An OpenStack body will recommend which tests are elevated from may-have to must-have

    2. The selection of “must-pass” tests should be based on quantifiable information when possible.

    3. Must-pass tests should be selected from the existing body of “may-pass” tests.  This encourages people to write tests for cases they want supported.

    4. We will have a process by which tests are elevated from may to must lists

    5. Potentially: the User Committee will nominate tests that elevated to the board

  10. OpenStack Core means passing all “must-pass” tests

    1. The OpenStack board owns the responsibility to define ‘core’ – to approve ‘musts’

    2. We are NOT defining which items are on the list in this effort, just making the position that it is how we will define core

    3. May-have tests include items in the integrated release, but which are not core.

    4. Must haves – must comply with the Core criteria defined from the IncUp committee results

    5. Projects in Incubation or pre-Incubation are not to be included in the ‘may’ list

Visualizing the OpenStack Core discussion points

THIS POST IS #6 IN A SERIES ABOUT “WHAT IS CORE.”

As we take the OpenStack Core discussion to a larger audience, I was asked to create the summary version the discussion points.  We needed a quick visual way to understand how these consensus statements interconnect and help provide context.  To address this need, I based it on a refined 10 core positions to create the following OpenStack Core flowchart.

core flow

The flow diagram below is grouped into three main areas: core definition (green), technical requirements (blue), and testing impacts (orange).

  1. Core Definition (green) walks through the fundamental scope and premise of the “what is core” discussion.  We are looking for the essential OpenStack: the parts that everyone needs and nothing more.  While OpenStack can be something much larger, core lives at the heart of the use-case venn diagram.  It’s the magical ice cream flavor that everyone loves like Triple Unicorn Rainbow Crunch.
  2. Technical Requirements (blue) covers some of the most contentious parts of the dialog.  This section states the expectation that OpenStack™ implementations must use parts the OpenStack code (you can’t just provide a compatible API).  It goes further to expect that we will maintain an open reference implementation and also identify places where parts of the code can be substituted with alternate implementations.  Examples of alternate implementations are plug-ins, API extensions, different hypervisors, and alternate libraries.
  3. Testing Impacts (orange) reviews some of the important new thinking around Core.  These points focus on the use of OpenStack community tests (e.g.: Tempest) to validate the total code base.  We expect users to be able to self-administer these tests or rely on an external validation.  Either way, we do not expect all tests to pass for all configurations; instead, the Foundation will identify a subset of the tests as required or must-pass.  The current thinking is that these must-pass tests will become the effective definition of OpenStack™ Core.

I hope this helps connect the dots on the core discussions so far.

I’d like to clean-up the positions to match the flow chart and cross reference.  Stay tuned!  This flowchart is a work in process – updates and suggestions are welcome!

READ POST IS #7: WHERE IS THIS GOING?

Community dialogue around “What is Core” positions

THIS POST IS #5 IN A SERIES ABOUT “WHAT IS CORE.”

by Rob Hirschfeld (cc) w/ attribution

The OpenStack Foundation Board has been having a broadening conservation about this topic.  Feeling left out?  Please don’t be!  Now is the time to start getting involved: we had to start very narrowly focused to avoid having the discussion continue to go in circles.  As we’ve expanding the dialog, we have incorporated significant feedback to drive consensus.

No matter where I go, people are passionate about the subject of OpenStack Core.

Overall, there is confusion of scope covered by “what is core” because people bring in their perspective from public, private solution, ecosystem or internal deployment objectives.  In discussion, everyone sees that we have to deal with issues around the OpenStack mark and projects first, but they are impatient to get into the deep issues.  Personally, we can get consensus on core and will always have a degree of healthy tension between user types.

The following are my notes, not my opinions.  I strive to faithfully represent a wide range of positions here.  Clarifications, comments and feedback are welcome!

Consensus Topics:

  • Reference/Alternate Implementation (not plug-in): Not using “plug-ins” to describe the idea that OpenStack projects should have a shared API with required code and clearly specified areas where code is replaceable.  It is the Technical Committee (TC) that makes these decisions.  The most meaningful language around this point is to say that OpenStack will have an open reference implementation with allowable alternate implementations.
  • Alternate implementations are useful:  We want to ensure upstream contribution and collaboration on the code base.  Reference implementations ensure that there’s a reason to keep open source OpenStack strong.  Alternate Implementations are important to innovation.
  • Small vs. Large Core: This is an ongoing debate about if OpenStack should have a lot of projects as part of core.  We don’t have an answer but people feel like we’re heading in a direction that resolves this question.
  • Everyone likes tests: We’re heading towards a definition of core that relies heavily on tests.  Everyone expresses concerns that this will place a lot of stress on Tempest (or another framework) and that needs to be addressed as we move forward.

Open Topics:

  • Monolithic vs. Granular Trademark:  We did not discuss if vendors will be able to claim OpenStack trademarks on subcomponents of the whole.  This is related to core but wide considered secondary.
  • API vs. implementation tension:  We accept that OpenStack will lead with implementation.   There’s no official policy that “we are not a standards body” but we may also have to state that tests are not a specification.  There’s a danger that tests will be considered more than they are.  What are they?  “They are an implementation and a source of information.  They are not the definition.”   We expect to have a working model that drives the API not vice versa.
  • Brouhaha about EC2 APIs:  It’s not clear if defining core helps address the OpenStack API discussion.  I hope it will but have not tested it.
  • Usability as core: I had many people insist that usability and ease of use should be as requirements for core because it supports adoption.  Our current positions do not have any statements to support this view.
  • Toxic neighbors: We have not discussed if use of the mark and criteria could be limited by what else you put in your product.  Are there implementation options that we’d consider toxic and automatically violate the mark?  Right now, the positions are worded that if you pass then you play even if you otherwise stink.
  • Which tests are required?  It appears that we’re moving towards using must-pass tests to define the core.  Moving towards tests determining core, we want actual field data to drive which tests are required. That will allow actual user experience to shape which tests are important rather than having it be a theoretical decision.  There’s some interest in asking the User Committee (UC) to recommend which tests are required.  This would be an added responsibility for the UC and needs more discussion.
  • Need visualization:  With 12 positions so far, it’s getting hard to keep it all together.  I’ve taken on an action item to create a diagram that shows which statements apply to which projects against the roles of ownership.

I’ve had some great discussions about core and am looking forward to many more.  I hope these notes help bring you up to speed.   As always, comments and discussion are welcome!

READ POST #6: VISUALIZING CORE

Twelve straw man positions to frame OpenStack “what is core” discussion

THIS POST IS #4 IN A SERIES ABOUT “WHAT IS CORE.”

Train WheelsIt’s time to move to the what from the how and why.  In my experience discussing the framework, it’s important to understand the context at this point because our objective is to define the common ground.  If you mistake the intent then it’s much harder to understand the details, but here they are…

I’m hopeful that the framework we’re defining today will ultimately be taken for granted.  Until then, relax and remember that this straw man is a work in progress.

These positions have been crafted over the course of several weeks and are constantly evolving. Next steps are to expand discussions to the OpenStack Community for input and review at the next Board Meeting on August 6th.   Ultimately, we need to close this issue at the next Summit.

I present them here for reference and encourage the community to bring up dialog on the OpenStack list (positions are sourced here).

Update Aug 13, 2013: These original 12 points have been refined down to 10.

You’re welcome to read the original 12 below, but I recommend skipping over to the refined set.

Continue reading

My insights from OpenStack “what is core” Spider > we need pluggable architectures

THIS POST IS #3 IN A SERIES ABOUT “WHAT IS CORE.”

IdeasSo what did we learn from the spider map exercise?  Above all else, the spider confirmed that to me that OpenStack is a world of paradox.  The perfect definition of core may be elusive, but I believe we can find one that is sufficient.

The goal was understanding not philosophical truth.  In a diverse and vibrant community with many objectives, understanding leads to consensus while being “right” can become very lonely.

Since our goal was not to answer the question, what did we want to accomplish?  Spider success was defined as creating a framework, really a list of agreed positions (post #4), that narrows the scope of the “what is core” dialog.

Too vague a framework leads to uncertainty about what’s included, stable and working while too rigid a baseline could drive away innovation and lead to forking.  Being too aggressively open could discourage commercial investment yet too proprietary an approach contradicts our collaboration and community values.

Having a workable framework that accommodates these diverse positions allows us to move forward.

So what did Alan and I learn from the spider to help the discussion?

  • “Plug-ins” are essential to the definition of core because they create safe places for innovation  [note: there has been much refinement of what "plug-in" means here]
  • It is possible to balance between stability and innovation if we have a way to allow implementations to evolve
  • OpenStack has a significant commercial ecosystem that needs to be accommodated in core
  • We need an approach that allows extension and improvement without having to incubate new projects
  • We need to ensure that we use brand and culture to combat forking
  • Interoperability is a worthy goal
  • Everyone thinks testing is good, but it’s still a sidebar
  • There are multiple distinct audiences with conflicting goals: some want stability and durability while others want innovation and flexibility.

Of these insights, the need to discuss how OpenStack promotes a plug-in architecture seemed address the most points of tension.  [update: in the course of discussion, we've defining plug-in more generally to be something like "a designated section of implementation code that can be altered without negatively changing the base function of the project."]

This is not the only item worth discussing, but it was the one that made the most sense to cover first based on the spider map.  Our idea was that having the community find agreement on how we approach plug-ins would lead us closer to a common ground for the “what is core” discussion.

Finding a common thread shrinks the problem space so the Board, TC and Community can advance in discussion.  So far, that assessment has proven accurate enough to move the dialog forward.

READ POST IS #4: TWELVE POSITIONS

Introducing the OpenStack Spider Graph: untangling our web of interdependencies

This post is #2 in a series about “what is core.”

peaceOne of the OpenStack Foundation’s most fundamental responsibilities is to define “what is core” for the OpenStack project.  A simple yet urgent question with profound implications for the project’s success; consequently, it’s no surprise that finding a consensus answer has been a thorny problem.   In fact, the answer (it’s 42, of course) is so convoluted that we had to step back establish common ground before we could proceed. I’m proud to say that we’ve made substantial progress towards establishing a framework for the discussion.   

This post (#2) is about the process we followed.  My next post (#3) is about what we learned.  Next, we want to engage the community in discussing the baseline positions (post #4) that make up the framework.   The challenge in discussion was that the Board (and Community) lacked common baseline.  When one person talked about commercial extensions to core another would insist that core always be open.  There are compatible statements; yet, we’d often talk past each other because the question was simply too big and fuzzy.

spider boardAlan Clark, Foundation Board Chairman, and I took on the task of breaking the logjam by decomposing the problem.  We used a multilayered mind map that I called the “spider” chart (yes, it’s a misnomer).  In constructing the spider, we first flagged out issue areas (the nodes) and added goals (red text) for each issue area.  Next we linked connected nodes together to form a graph.  We added blue text on connections where goals were opposed or in tension and green text where goals were aligned.

Note: Some of our text is controversial – it was our objective to find the tension.  If you are looking at the image, we also created some grid and flow charts to test the map.

We did not time capture the spider so it’s impossible to sweep back and see how the ideas progressed.  In general, we started at the center from “IncUp Discussion” which was the TC and Board’s collaboration, led by Alan, to define the process by which projects are promoted from incubation to integrated projects.

Some topics were easy like “API vs Implementation” and “Trademark” while “Implementation Led” and “Plug-ins” were more difficult.  Some clarification on those two items is worthwhile:

Implementation Led describes the OpenStack community for starting from working implementations rather than designing an API.  In my experience, this pattern serves the community well as long as we accept that APIs and implementations in incubated projects will take time to mature.   Using this approach gets us out of prolonged design by committee spirals.  It has serious pitfalls also – there’s no milk and honey approach. Plug-ins represented a big pot of politics in the community that was part of the challenge in defining what is core.  The obvious issue is that not all core projects use plug-ins and the ones that do take different approaches.  Even so, we felt that the concept of multiple implementations with a single API was a significant part of the opportunity (and challenge) for OpenStack core.

Ultimately, we decomposed the spider graph into the following table that shows some of our thoughts in raw form.  I’ll discuss the actual insights in my next post (#3) Continue reading