Running with scissors > DefCore “must-pass” Road Show Starts [VIDEOS]

The OpenStack DefCore committee has been very active during this cycle turning the core definition principles into an actual list of “must-pass” capabilities (working page).  This in turn gives the community something tangible enough to review and evaluate.

Capabilities SelectionTL;DR!  We appreciate those in the community who have been patient enough to help define and learn the process we’re using the make selections; however, we also recognize that most people want to jump to the results.

This week, we started a “DefCore roadshow” with the goal of learning how to make this huge body of capabilities, process and impact easier to digest (draft write-up for review & Troy Toman’s notes).  So far we’ve had two great sessions on this topic.  We took notes and recorded at both meetups (San Francisco & Austin).

My takeaways of these initial meetups are:

  • Jump to the Capabilities right away, the process history is not needed up front
  • You need more graphics – specifically, one for the selection criteria (what do you think of my 1st attempt?)
  • Work from some examples of scored capabilities
  • Include some specific use-cases with a user, 2 types of private cloud and a public cloud to help show the impact

Overall, people like what they are hearing.  It makes sense and decisions are justified.

We need more feedback!  Please help us figure out how to explain this for the broader community.

OpenStack Board Elections: What I’ll do in 2014: DefCore, Ops, & Community

Rob HirschfeldOpenStack Community,

The time has come for you to choose who will fill the eight community seats on the Board (ballot links went out Sunday evening CST).  I’ve had the privilege to serve you in that capacity for 16 months and would like to continue.  I have leadership role in Core Definition and want to continue that work.

Here are some of the reasons that I am a strong board member:

  • Proven & Active Leadership on Board - I have been very active and vocal representing the community on the Board.  In addition to my committed leadership in Core Definition, I have played important roles shaping the Gold Member grooming process and trying to adjust our election process.  I am an outspoken yet pragmatic voice for the community in board meetings.
  • Technical Leader but not on the TC – The Board needs members who are technical yet detached from the individual projects enough to represent outside and contrasting views.
  • Strong User Voice – As the senior OpenStack technologist at Dell, I have broad reach in Dell and RedHat partnership with exposure to a truly broad and deep part of the community.  This makes me highly accessible to a lot of people both in and entering the community.
  • Operations Leadership – Dell was an early leader in OpenStack Operations (via OpenCrowbar) and continues to advocate strongly for key readiness activities like upgrade and high availability.  In addition, I’ve led the effort to converge advanced cookbooks from the OpenCrowbar project into the OpenStack StackForge upstreams.  This is not a trivial effort but the right investment to make for our community.
  • And there’s more… you can read about my previous Board history in my 2012 and 2013 “why vote for me” posts or my general OpenStack comments.

And now a plea to vote for other candidates too!

I had hoped that we could change the election process to limit blind corporate affinity voting; however, the board was not able to make this change without a more complex set of bylaws changes.  Based on the diversity and size of OpenStack community, I hope that this issue may no longer be a concern.  Even so, I strongly believe that the best outcome for the OpenStack Board is to have voters look beyond corporate affiliation and consider a range of factors including business vs. technical balance, open source experience, community exposure, and ability to dedicate time to OpenStack.

How are we picking the OpenStack DefCore “must pass” tests?

As part of the DefCore work, we have the challenge of taking all the Tempest tests and figuring out which ones are the “must-pass” tests that will define core (our note pages).  We want to have a very transparent and objective process for picking the tests so we need to have well defined criteria and a selection process.
Figuring out the process will be iterative.  The list below represents a working set of selection criteria that are applied to the tests.  The DefCore committee will determine relative weights for the criteria after the tests have been scored because it was clear in discussion that not all of these criteria should have equal weight.
Once a test passes the minimum criteria score and becomes “must-pass” the criteria score does not matter – the criteria are only used for selecting tests. As per the Core principles, passing all “must-pass” test will be required to be considered core.
So what are these 13 preliminary criteria (source)?
1. Test is required stable for >2 releases (because things leaving Core are bad)
  • the least number/amount of must pass tests as possible (due to above)
  • but noting that the number will increase over time
  • least amount of change from current requirements as possible (nova, swift 2 versions)
  • (Acknowledge that deprecation is punted for now, but can be executed by TC)
2. Where the code being tested has an designed area of alternate implementation (extension framework) as per the Core Principles, there should be parity in capability tested across extension implementations
  • Test is not configuration specific (test cannot meet criteria if it requires a specific configuration)
  • Test does not require an non-open extension to pass (only the OpenStack code)
3. Capability being tested is Service Discoverable (can be found in Keystone and via service introspection) – MONTY TO FIX WORDING around REST/DOCS, etc.
  • Nearly core or “compatible” clouds need to be introspected to see what’s missing
  • Not clear at this point if it’s project or capability level enforced.  Perhaps for Elephant it’s project but it should move to capability for later
4A, 4B & 4C. Candidates are widely used capabilities
  • 4A favor capabilities that are supported by multiple public cloud providers and private cloud products
    • Allow the committee to use expert judgement to promote capabilities that need to resolve the “chicken-and-egg”
    • Goals are both diversity and quantity of users
  • 4B. Should be included if supported by common tools (Ecosystem products includes)
  • 4C. Should be included if part of common libraries (Fog, Apache jclouds, etc)
5. Test capabilities that are required by other must-pass tests and/or depended on by many other capabilities
6. Should reflect future technical direction (from the project technical teams and the TC)
  • Deprecated capabilities would be excluded (or phased out)
  • This could potentially become a “stick” if used incorrectly because we could force capabilities
7. Should be well documented, particularly the expected behavior.
  • includes the technical references for others in the project as well as documentation for the users and or developers accessing the feature or functionality
8. A test that is a must-pass test should stay a must-pass test (makes must-pass tests sticky release per release)
9. A test for a Capability with must-pass tests is more likely to be considered must-pass
10 Capabilities is unique and cannot be build out of other must-pass capabiliies
  • Candidates favor capabilities that users cannot implement if given the presence of other capabilities
  • consider the pain to users if a cloud doesn’t have the capability – not so much pain if they can run it themselves
  • “Unique capabilities that cannot be build out of other must-pass capabilities should not be considered as strongly”
11. Tests do not require administrative right to execute
We expect these criteria to change based on implementation experience and community input; however, we felt that further discussion without implementation was getting diminishing returns.  It’s import to remember that all of the criteria are not equal, they will have relative weights to help drive tune the results.

OpenStack Core Definition (DefCore) Progress in 6 key areas

DefCore Elephant Cycle

I’m excited to report about the OpenStack Board progress on defining OpenStack core.  At the Hong Kong summit, Joshua McKenty and I were asked to chair a new standing committee, now known as DefCore, to define “OpenStack Core” based on the core principles that we determined over the last 6 months (aka “the spider”).

Joshua and I took on the challenge with gusto and I’m proud to say that we’ve already made significant progress against an aggressive timeline to have the pilot must-pass tests for Havana defined before the Juno Summit in April 2014.  It’s important to remember that we’re moving from a project based definition of core to test-driven capabilities because this best addresses our interoperability objectives.

In the 8 weeks since the summit, we’ve had six very productive meetings (etherpads for Prep, DefCore.1, DefCore.2, Criteria 1 and 2) with detailed notes and recorded content. Here’s my summary of our results so far:

  1. An Aggressive Timeline for having pilot Havana must-pass tests approved by the Juno summit in May 2014.  That drives the schedule backward toward a preliminary list in March.  Once we have a pilot list for Havana, we expect to have Ice House done +90 days and Juno done at the Paris summit.

  2. Test Selection Criteria a preliminary set of 14 criteria (needs a stand alone post) that will be used to quantitatively score the current 700+ tests.  We also agreed to use a max 100 point weighting system for the criteria.  The weights and score requirement iteratively once we have done a first scoring pass.  Our objective is to make must-pass test selection as objective and transparent as possible (post with details).

  3. Distinction between Capability & Test is important because we recognize that individual tests may validate multiple capabilities and individual capabilities may have multiple tests.  Our hope is to present the results in terms of capabilities not individual tests.

  4. Holding Off on Bylaws Changes needed to clarify how OpenStack manage core definition.  It was widely expected that the DefCore committee would have to make changes to the OpenStack bylaws; however, we believe we can proceed without rushing changes.  We have an active subcommittee preparing changes in advance of the next DefCore cycle.

  5. Program vs. Project Definition efforts are needed to help take pressure off requests to have “projects promoted to core status” and how the OpenStack trademark is used for projects.  We are trying to clarify OpenStack Programs (e.g.: OpenStack™ Compute) carry to the trademark while OpenStack Projects (e.g.: Nova and Glace) are members of those programs and do not carry the OpenStack trademark directly.  Consequently, we’d expect people to say “OpenStack Compute Project Nova” instead of “OpenStack Nova.”  This approach addresses several issues that impact DefCore Board activities around trademark, core and brand.

  6. RefStack Development and Use Cases provide the framework for community reporting of test results.  We consider this infrastructure critical to getting community input about must-pass tests and also sharing interoperability information.  This effort is just beginning and needs help from the community.

For all this progress, we are only starting!  We’ve cleared the blocks preventing implementation and that will expose a new set issues to discuss.  Look for us to start applying the criteria to tests in the next months.  That will quickly expose the strengths and weaknesses of our criteria set.  We’ve also got to make progress on Program vs. Project and start RefStack coding.

We want community participation!  Please let us know what you think.

Spinning up OpenStack “DefCore” Committee by spotting elephants

ElephantsThis week, Joshua McKenty, me and a handful of interested individuals (board member Eileen Evans included) met to start organizing the DefCore* Committee.  This standing committee was established by an OpenStack Foundation resolution just before the Hong Kong Summit.  Joshua and I were nominated as co-chairs (and about half the board volunteered to be members).    This action was an immediate result of the unanimous passage of the 10 Principles that I was driving in the DefCore “Spider” cycle.

We heard overwhelmingly at the Hong Kong summit that defining core should be a major focus for the Board.

The good news is that we’re doing exactly that in CoreDef.  Our challenge is to go quickly but not get ahead of community consensus.  So far, that means eating the proverbial elephant in small bites and intentionally deferring topics where we cannot find consensus.

This meeting was primarily about Joshua and I figuring out how to drive DefCore quickly (go fast!) without exceeding the communities ability to review and discuss (build consensus!).  While we had future-post-worthy conceptual discussions, we had a substantial agenda of get-it-done in front of us too.

Here’s a summary of key outcomes from the meeting:

1)      We’ve established a tentative schedule for our first two meetings (12/3 and 12/17).

  1. We’ve started building agendas for these two meetings.
  2. We’ve also established rules for governance that include members to do homework!

2)      We’ve agreed it’s important to present a bylaws change to the committee for consideration by the board.

  1. This change is to address confusion around how core is defined and possibly move towards the bylaws defining a core process not a list of core projects.
  2. This is on an accelerated track because we’d like to include it with the Community Board Member elections.

3)      We’ve broken DefCore into clear “cycles” so we can be clearer about concrete objectives and what items are out of scope for a cycle.  We’re using names to designate cycles for clarity.

  1. The first cycle, “Spider,” was about finding the connections between core issues and defining a process to resolve the tension in those connections.
  2. This cycle, “Elephant,” is about breaking the Core definition into
  3. The next cycle(s) will be named when we get there.  For now, they are all “Future”
  4. We agreed there is a lot of benefit from being clear to community about items that we “kick down the road” for future cycles.  And, yes, we will proactively cut off discussion of these items out of respect for time.

4)      We reviewed the timeline proposed at the end of Spider and added it to the agenda.

  1. The timeline assumes a staged introduction starting with Havana and accelerating for each release.
  2. We are working the timeline backwards to ensure time for Board, TC and community input.

5)      We agreed that consensus is going to be a focus for keeping things moving

  1. This will likely drive to a smaller core definition
  2. We will actively defer issues that cannot reach consensus in the Elephant cycle.

6)      We identified some concepts that may help guide the process in this cycle

  1. We likely need to create categories beyond “core” to help bucket tests
  2. Committee discussion is needed but debate will be time limited

7)      We identified the need to start on test criteria immediately

  1. Board member John Zannos (in absentia) offered to help lead this effort
  2. In defining test criteria, we are likely to have lively discussions about “OpenStack’s values”

8)      We identified some out of scope topics that are important but too big to solve.

  1. We are calling these “elephants” (or the elephant in the room).
  2. The list of elephants needs to be agreed by DefCore and clearly communicated
  3. We expect that the Elephant cycle will make discussing these topics more fruitful

9)      We talked about RefStack code features

  1. Allowing users to upload/post test results from their clouds to enable white box test reporting
  2. Allowing users who have uploaded results to +/- vote on tests they think are important
  3. We established a requirement that posting results requires an OpenStack ID
  4. We established a requirement that only a single Corporate designate (provided by the Foundation) can make a result official for their company.
  5. Collecting opt-in data with test results using tags for things like alternate implementations use, host operating system(s), deployment method, size of cloud, and hypervisor.
  6. We discussed (but did not resolve) that it could be possible to have people run RefStack against public cloud end points and post their results
  7. We agreed that RefStack needs to be able to run locally or as a hosted site.

10)   We identified a lot of missing communication channels

  1. We created a DefCore wiki page to be a home for information.
  2. Joshua and I (and others?) will work with the Foundation staff to create “what is core” video to help the community understand the Principles and objectives for the Elephant cycle.
  3. We are in the process of setting up mail lists, IRC, blog tags, etc.

Yikes!  That’s a lot of progress priming the pump for our first DefCore meeting!

* We picked “DefCore” for the core definition committee name.  One overriding reason for the name is that it has very clean search results.  Since the word “core” is so widely used, we wanted to make sure that commentary on this topic is easy to track against the noisy term core.  We also liked 1) the reference to DefCon and 2) that the Urban Dictionary defines it as going deaf from standing too close to the speakers.

OpenStack Core Online Forum, Oct 16 13:30 UTC / Oct 22 0100 UTC

Go Online!OpenStack Community, you are invited on an online discussion about OpenStack Core on October 16th at UTC 13:30 (8:30 am US Central) and October 22nd at UTC 0100 (8:00 pm US Central)

At the next OpenStack Foundation Board meeting, we will be setting a timeline for implementing an OpenStack Core Definition process that promotes a clear and implementation driven metric for deciding which projects should be considered “required.”  This is your chance to review and influence the process!

We’ll review the OpenStack Core Definition process (20 minutes) and then open up the channel for discussion using the IRC (#openstack-meeting) & Google Hangout on Air (link posted in IRC).

The forum will be coordinated through the IRC channel for links and questions.

Can’t make it?  The session was recorded > here!

Thinking about how to Implement OpenStack Core Definition


Tied UpWe’ve had a number of community discussions (OSCON, SFO & SA-TX) around the process for OpenStack Core definition.  These have been animated and engaged discussions (video from SA-TX): my notes for them are below.

While the current thinking of a testing-based definition of Core adds pressure on expanding our test suite, it seems to pass the community’s fairness checks.

Overall, the discussions lead me to believe that we’re on the right track because the discussions jump from process to impacts.  It’s not too late!  We’re continuing to get community feedback.  So what’s next?

First…. Get involved: Upcoming Community Core Discussions

These discussions are expected to have online access via Google Hangout.  Watch Twitter when the event starts for a link.

Want to to discuss this in your meetup? Reach out to me or someone on the Board and we’ll be happy to find a way to connect with your local community!

What’s Next?  Implementation!

So far, the Core discussion has been about defining the process that we’ll use to determine what is core.  Assuming we move forward, the next step is to implement that process by selecting which tests are “must pass.”  That means we have to both figure out how to pick the tests and do the actual work of picking them.  I suspect we’ll also find testing gaps that will have developers scrambling in Ice House.

Here’s the possible (aggressive) timeline for implementation:

  • November: Approval of approach & timeline at next Board Meeting
  • January: Publish Timeline for Roll out (ideally, have usable definition for Havana)
  • March: Identify Havana must pass Tests (process to be determined)
  • April: Integration w/ OpenStack Foundation infrastructure

Obviously, there are a lot of details to work out!  I expect that we’ll have an interim process to select must-pass tests before we can have a full community driven methodology.

Notes from Previous Discussions (earlier notes):

  • There is still confusion around the idea that OpenStack Core requires using some of the project code.  This requirement helps ensure that people claiming to be OpenStack core have a reason to contribute, not just replicate the APIs.
  • It’s easy to overlook that we’re trying to define a process for defining core, not core itself.  We have spent a lot of time testing how individual projects may be effected based on possible outcomes.  In the end, we’ll need actual data.
  • There are some clear anti-goals in the process that we are not ready to discuss but will clearly going to become issues quickly.  They are:
    • Using the OpenStack name for projects that pass the API tests but don’t implement any OpenStack code.  (e.g.: an OpenStack Compatible mark)
    • Having speciality testing sets for flavors of OpenStack that are different than core.  (e.g.: OpenStack for Hosters, OpenStack Private Cloud, etc)
  • We need to be prepared that the list of “must pass” tests identifies a smaller core than is currently defined.  It’s possible that some projects will no longer be “core”
  • The idea that we’re going to use real data to recommend tests as must-pass is positive; however, the time it takes to collect the data may be frustrating.
  • People love to lobby for their favorite projects.  Gaps in testing may create problems.
  • We are about to put a lot of pressure on the testing efforts and that will require more investment and leadership from the Foundation.
  • Some people are not comfortable with self-reporting test compliance.   Overall, market pressure was considered enough to punish cheaters.
  • There is a perceived risk of confusion as we migrate between versions.  OpenStack Core for Havana seems to specific but there is concern that vendors may pass in one release and then skip re-certification.  Once again, market pressure seems to be an adequate answer.
  • It’s not clear if a project with only 1 must-pass test is a core project.  Likely, it would be considered core.  Ultimately, people seem to expect that the tests will define core instead of the project boundary.

What do you think?  I’d like to hear your opinions on this!

Taking OpenStack Core discussions to community


We’ve been building up to a broad discussion about the OpenStack Core and I’d like to invite everyone in the OpenStack community to participate (review latest).

Alan Clark (Board Chairman) officially kicked off this open discussion with his post on the OpenStack blog last week.  And we’re trying to have face-to-face events for dialog like the Core meetup tonight in San Francisco.  Look for more to come!

Of course, this will also be a topic at the summit (Alan and I submitted two sessions about this).  The Board needs to move this forward in the November meeting, so NOW is the time to review and give us input.

OpenStack’s Test Driven Core > it’s where I think “what is core” discussions are heading


core lighthouse

In helping drive OpenStack’s “what is core” dialog, I’ve had the privilege of listening to a lot of viewpoints about what we are and should be.  Throughout the process, I’ve tried to put aside my positions and be an objective listener.  In this post, I’m expressing where I think this effort will lead us.

If OpenStack culture values implementation over API then our core definition should too.

How do we make a core definition that values implementation over API?  I think that our definition should also be based on what’s working in the field over qualitative definitional statements.  The challenge in defining core is to find a way to reinforce this culture in a quantifiable way.

The path forward lies in concrete decomposition (and not because you were talked to death on the sidewalk).

Concrete decomposition means breaking Core into small units for discussion like “is provisioning a single server critical?”  More importantly, we can use tests as the unit for decomposition.  Tests are gold when it comes to defining expected OpenStack behaviors.  In our tests, we have a description of which use-cases have been implemented.  Discussing those use-cases is much more finite than arguing over stable versus innovative development methodologies.

I believe that we are moving toward community tests playing an essential role in OpenStack.  As a believer in the value of BDD and CI, I think that placing high value on tests improves the project in fundamental ways beyond defining Core.  It creates a commercial motivation for contributors to add tests, inches us toward interoperability, and helps drive stability for users.  In these ways, using tests to measure OpenStack drives the right behaviors for the project.

Another consequence I anticipate is a new role for the User Committee (UC).  With a growing body of tests, the OpenStack Foundation needs a way to figure out the subset of tests which are required.  While the Technical Committee (TC) should demand a comprehensive suite of tests for all projects, they lack the perspective to figure out which use-cases are being implemented by our user base.  Gathering that data is already the domain of the UC so asking them to match implemented use-cases to tests seems like a natural extension of their role.

By having data supporting the elevation of tests to must-pass status, I envision a definition of Core that is based on how OpenStack is implemented.  That, in turn, will help drive our broader interoperability objectives.

Visualizing the OpenStack Core discussion points


As we take the OpenStack Core discussion to a larger audience, I was asked to create the summary version the discussion points.  We needed a quick visual way to understand how these consensus statements interconnect and help provide context.  To address this need, I based it on a refined 10 core positions to create the following OpenStack Core flowchart.

core flow

The flow diagram below is grouped into three main areas: core definition (green), technical requirements (blue), and testing impacts (orange).

  1. Core Definition (green) walks through the fundamental scope and premise of the “what is core” discussion.  We are looking for the essential OpenStack: the parts that everyone needs and nothing more.  While OpenStack can be something much larger, core lives at the heart of the use-case venn diagram.  It’s the magical ice cream flavor that everyone loves like Triple Unicorn Rainbow Crunch.
  2. Technical Requirements (blue) covers some of the most contentious parts of the dialog.  This section states the expectation that OpenStack™ implementations must use parts the OpenStack code (you can’t just provide a compatible API).  It goes further to expect that we will maintain an open reference implementation and also identify places where parts of the code can be substituted with alternate implementations.  Examples of alternate implementations are plug-ins, API extensions, different hypervisors, and alternate libraries.
  3. Testing Impacts (orange) reviews some of the important new thinking around Core.  These points focus on the use of OpenStack community tests (e.g.: Tempest) to validate the total code base.  We expect users to be able to self-administer these tests or rely on an external validation.  Either way, we do not expect all tests to pass for all configurations; instead, the Foundation will identify a subset of the tests as required or must-pass.  The current thinking is that these must-pass tests will become the effective definition of OpenStack™ Core.

I hope this helps connect the dots on the core discussions so far.

I’d like to clean-up the positions to match the flow chart and cross reference.  Stay tuned!  This flowchart is a work in process – updates and suggestions are welcome!