DefCore Process 9 Point Graphic balances Community, Vendor, Goverance

I’ve been working on the OpenStack DefCore process for nearly 3 years and our number #1 challenge remains how to explain it simply.

10 days ago, the DefCore committee met face-to-face in Austin to work on documenting the process that we want to follow (see Guidelines).  As we codify DefCore, our top priority is getting community feedback and explaining the process without expecting everyone to read the actual nut-and-bytes of the process.

I think of it as writing the DefCore preamble: “We, the community, in order to form a more perfect cloud….”

defcore 9 pointsI don’t think we’ve reached that level of simplicity; however, we have managed to boil down our thinking into nine key points.  I’m a big Tufte fan and believe that visualizations are essential to understanding complex topics.  In my experience, it takes many many iterations with feedback to create excellent graphics.  This triangle is my first workable pass.

An earlier version of these points were presented to the OpenStack board in December 2014 and we’ve been able to refine these during the latest DefCore community discussions.

We’re interested in hearing your opinions.  Here are the current (2015-Feb-22) points:

  1. COMMUNITY INVOLVEMENT
    1. MAPPING FEATURE AVAILABILITY: We are investing in gathering data driven and community involved feedback tools to engage the largest possible base for core decisions.   This mapping information will be available to the community.
    2. COMMUNITY CHOSEN CAPABILITIES :  Going forward, we want a community process to create, cluster and describe capabilities.  DefCore bootstrapped this process for Havana.  Further, Capabilities are defined by tests in Tempest so test coverage gaps (like Keystone v2) translate into Core gaps that the community will fill by writing tests.
    3. TESTS AS TRUTH: DefCore could expand in the future, but uses Tempest as the source of tests for now.  Gaps in Test will result in DefCore gaps.  We are hosting final documents in the Gerrit, using the OpenStack review process to ensure that we work within the community processes.
  2. VENDOR
    1. CLEAR RESULTS (PASS-FAIL): Vendors must pass all required core tests as defined by this process.  There are no partial results.  Passing additional tests is encouraged but not required.
    2. VENDORS SELF-TEST: Companies are responsible for running tests and submitting to the Foundation for validation against the DefCore criteria.  Approved vendor reports will be available for the community.
    3. APPEAL PROCESS / FLAGGED TESTS: There is a “safety valve” for vendors to deal with test scenarios that are currently difficult to recreate in the field.  We expect flags to be temporary.
  3. GOVERNANCE
    1. SCORING BASED ON TRANSPARENT PROCESS (DEFCORE): The 2015 by-laws change requires the Board and TC to agree to a process by which the Foundation can hold OpenStack Vendors accountable for their use of the trademarks.
    2. BOARD IS FINAL AUTHORITY: The Board is responsible for approving the final artifacts based on the recommendations.  By having a transparent process, community input is expected in advance of that approval.
    3. TIMELY GUIDANCE: The process is time sensitive.  There’s a need for the Board to produce DefCore guidance in a timely way after each release and then feed that result into the next cycle.  The guidance is expected to be drafted for review at each Summit and then approved at the Board meeting three months after the draft is posted.

OpenStack DefCore Accelerates & Simplifies with Clear and Timely Guidelines [Feedback?]

Last week, the OpenStack DefCore committee rolled up our collective sleeves and got to work in a serious way.  We had a in-person meeting with great turn out
with 5 board members, Foundation executives/staff and good community engagement.

defcore timelineTL;DR > We think DefCore deliverables should be dated milestone guidelines instead tightly coupled to release events (see graphic).

DefCore has a single goal expressed from two sides: 1) defining the “what is OpenStack” brand for Vendors and 2) driving interoperability between OpenStack installations.  From that perspective, it is not about releases, but about testable stable capabilities.  Over time, these changes should be incremental and, most importantly, trail behind new features that are added.

For those reasons, it was becoming confusing for DefCore to focus on an “Icehouse” definition when most of the capabilities listed are “Havana” ones.  We also created significant time pressure to get the “Kilo DefCore” out quickly after the release even though there were no “Kilo” specific additions covered.

In the face-to-face, we settled on a more incremental approach.  DefCore would regularly post a set of guidelines for approval by the Board.  These Guidelines would include the required, deprecated (leaving) and advisory (coming) capabilities required for Vendors to use the mark (see footnote*).  As part of defining capabilities, we would update which capabilities were included in each component and  which components were required for the OpenStack Platform.  They would also include the relevant designated sections.  These Guidelines would use the open draft and discussion process that we are in the process of outlining for approval in Vancouver.

Since DefCore Guidelines are simple time based lists of capabilities, the vendors and community can simply reference an approved Guideline using the date of approval (for example DefCore 2015.03) and know exactly what was included.  While each Guideline stands alone, it is easy to compare them for incremental changes.

We’ve been getting positive feedback about this change; however, we are still discussing it and appreciate your input and questions.  It is very important for us to make DefCore simple and easy.  For that, your confused looks and WTF? comments are very helpful.

* footnote: the Foundation manages the OpenStack brand and the process includes multiple facets.  The DefCore Guidelines are just one part of the brand process.

My OpenStack Vancouver Session Promotion Dilemma – please, vote outside your block

We need people to promote their OpenStack Sessions, but how much is too much?

Megaphone!Semi-annually, I choose to be part of the growing dog pile of OpenStack summit submissions.  Looking at the list, I see some truly amazing sessions by committed and smart community members.  There are also a fair share of vendor promotions.

The nature of the crowded OpenStack vendor community is that everyone needs to pick up their social media megaphones (and encourage some internal block voting) to promote their talks.   Consequently, please I need to ask you to consider voting for my list:

  1. DefCore 2015 
  2. The DefCore Show: “is it core or not” feud episode
  3. Mayflies: Improve Cloud Utilization by Forcing Rapid Server Death [Research Analysis] (xref)
  4. It’s all about the Base. If you want stability, start with the underlay [Crowbar] 
  5. State of OpenStack Product Management

Why am I so reluctant to promote these excellent talks?  Because I’m concerned about fanning the “PROMOTE MY TALKS” inferno.

For the community to function, we need for users and operators to be heard.  The challenge is that the twin Conference/Summit venue serves a lot of different audiences.

In my experience, that leads to a lot of contributor navel gazing and vendor-on-vendor celebrations.  That in turn drowns out voices from the critical, but non-block-enabled users and operators.

Yes, please vote those sessions of mine that interest you; however, please take time to vote more broadly too.  The system randomized which talks you see to help distribute voting too.

Thanks.

OpenStack PSA: Individual members we need more help – Please Vote!

1/17 Update: We did it!  We reached quorum and approved all the changes!  Also, I am honored to have been re-elected to the Board.  Thank you for the support.

I saw the latest report and we’ve still got a LONG WAY TO GO to get to the quorum that we need.  Don’t let your co-worker or co-contributor be the one missing vote!

Note: If you thing you should have gotten a ballot email but did not.  Contact the OpenStack Election Secretary for assistance.  OpenStack voting is via YOUR PERSONALIZED EMAIL only – you cannot use someone else’s ballot.

Here’s the official request that we’ve been forwarding in the community

OpenStack Individual Members we need your help – Please Vote!

Untitled drawingIncluded on the upcoming individual elections ballot is set of proposed bylaw changes [note: I am also seeking re-election]. To be enacted, these changes require approval by the individual members. At least 25% of the Individual Members must participate in this election in order for the vote to take effect which is why we are reaching out to you. The election will start Monday January 12, 2015 and run thru Friday January 16, 2015.

The unprecedented growth, community size and active nature of the OpenStack community have precipitated the need for OpenStack Bylaw updates. The updates will enable our community to adapt to our continued rapid growth, change and diversity, while reflecting our success and market leadership. Although the proposed changes only effect a small set of verbiage in the bylaws, the changes eliminate some of the hard coded values and naive initial assumptions that found their way into the bylaws when they were initially created in 2013. Those initial assumptions did not anticipate that by 2015 we would have such a large, active community of over 17,000 individual members, over 430 corporate members, and a large diverse set of OpenStack based products and services.

Through many months of community iterative discussion and debate, the DefCore team and board have unanimously accepted a set of changes that are now placed before you for your approval. The changes replace the original hard coded “core” definition with a process for determining the software elements required for use of the OpenStack commercial trademark. Processes which will also account for future revisions and determinations for Core and Trademark Policy.

Note: Another change sets the quorum level at a more reasonable 10%, so these PSAs should not be required in the future.

Complete details on the proposed changes are located at:
https://wiki.openstack.org/wiki/Governance/Foundation/2014ProposedBylawsAmendment

Complete details on the 2015 Board Election are located at:
http://www.openstack.org/election/2015-individual-director-election/

Voting time for OpenStack! Your help needed to push Bylaws & DefCore forward.

1/17 Update: We did it!  We reached quorum and approved all the changes!  Also, I am honored to have been re-elected to the Board.  Thank you for the support!

What does OpenStack mean to you?

Vote Now!To me, it’s more than infrastructure software: it’s a community of people and companies working together in revolutionary way.  To make that work, we need people who invest time to bring together diverse interests and perspectives.

I have worked hard to provide neutral and inclusive view points on the OpenStack board.  The critical initiatives I focused on [DefCore, Gold Membership, Product Managers] need focused leadership to progress.  I have proven my commitment to those efforts and would be honored to be re-elected [see my platforms for 2012, 2013 & 2014].

But more important than voting for me, is that we reach a 25% quorum!

Why?  Because there are critical changes to the OpenStack bylaws on the ballot.

What type of changes?  Basically, the changes allow OpenStack governance to keep up with the pace of our evolution.    These are pragmatic changes that have been thoroughly discussed in the community.  They enable OpenStack to:

  • set a smarter quorum – past elections only achieved half the required turn out to changes the bylaws
  • enable the DefCore process – current core is defined as simply using code in select projects.
  • improve governance of required committees
  • tweak release language to match current practice,

To hit 25%, we need everyone, not just the typical evangelists to encourage voting.  We need you to help spread the word and get others to do the same.

I’m counting on you, please help. 

10 pounds of OpenStack cloud in a 5 pound bag? Do we need a bigger bag?

Yesterday, I posted about cloud distruptors that are pushing the boundaries of cloud. The same forces pull at OpenStack where we are working to balance between including all aspects of running workloads and focusing on a stable foundation.

Note: I am seeking re-election to the 2015 OpenStack Board.  Voting starts 1/12.

For weeks, I’ve been reading and listening to people inside and outside the community.  There is considerable angst about the direction of OpenStack.  We need to be honest and positive about challenges without simply throwing stones in our hall of mirrors.

Closing 2014, OpenStack has gotten very big, very fast.  We’ve exploded scope, contributions and commercial participants.  Unfortunately, our process infrastructure (especially the governance by-laws) simply have not kept pace.  It’s not a matter of scaling processes we’ve got; many of the challenges created by growth require new approaches and thinking (Thierry’s post).

OpenStack BagIn 2015, we’re trying to put 10 pounds of OpenStack in a 5 pound bag.  That means we have to either a) shed 5 pounds or b) get a bigger bag.  In classic OpenStack style, we’re sort of doing both: identifying a foundational base while expanding to allow more subprojects.

To my ear, most users, operators and business people would like to see the focus being on the getting the integrated release scope solid.  So, in spirit of finding 5 pounds to shave, I’ve got five “shovel ready” items that should help:

  1. Prioritizing stability as our #1 feature.  Accomplishing this will require across the broad alignment of the vendor’s product managers to hold back on their individual priorities in favor of community.  We’ve started this effort but it’s going to take time to create the collaboration needed.
  2. Sending a clear signal about the required baseline for OpenStack.  That’s the purpose of DefCore and should be felt as we work on the Icehouse and Juno definitions.
  3. Alignment of the Board DefCore project with Technical Committee’s Levels/Big Tent initiative.  By design, these efforts interconnect.  We need to make sure the work is coordinated so that we send a clearly aligned message to the technical, operator, vendor and user communities.
  4. Accelerate changes from single node gate to something that’s either a) more services focused or b) multi-node.  OpenStack’s scale of community development  requires automation to validate the new contributions do not harm the existing code base (the gate).  The current single-node gate does not reflect the multi-node environments that users target with the code.  While it’s technically challenging to address this mismatch, it’s also essential so we ensure that we’re able to validate multi-node features.
  5. Continue to reduce drama in the open source processes.   OpenStack is infrastructure software that should enable an exciting and dynamic next generation of IT.  I hear people talk about CloudStack as “it’s not as exciting or active a community but their stuff just works.”  That’s what enterprises and operators want.  Drama is great for grabbing headings but not so great for building solid infrastructure.

What is the downside to OpenStack if we cannot accomplish these changes?  Forks.

I already see a clear pattern where vendors are creating their own distros (which are basically shallow forks) to preserve their own delivery cycle.  OpenStack’s success is tied to its utility for the customers of vendors who fund the contributors.  When the cost of being part of the community outweighs the value, those shallow forks may become true independent products.

In the case of potential forks, they allow vendors to create their own bag and pick how many pounds of cloud they want to carry.  It’s our job as a community in 2015 to make sure that we’ve reduced that temptation.

1/9/15 Note: Here’s the original analogy image used for this post

Nextcast #14 Transcription on OpenStack & Crowbar > “we can’t hand out trophies to everyone”

Last week, I was a guest on the NextCast OpenStack podcast hosted by Niki Acosta (EMC) [Jeff Dickey could not join].   I’ve taken some time to transcribe highlights.

We had a great discussion nextcastabout OpenStack, Ops and Crowbar.  I appreciate Niki’s insightful questions and an opportunity to share my opinions.  I feel that we covered years of material in just 1 hour and I appreciate the opportunity to appear on the podcast.

Video from full post (youtube) and the audio for download.

Plus, a FULL TRANSCRIPT!  Here’s my Next Cast #14 Short Transcripton

The objective of this transcription is to help navigate the recording, not replace it.  I did not provide complete context for remarks.

  • 04:30 Birth of Crowbar (to address Ops battle scars)
  • 08:00 The need for repeatable Ready State baseline to help community work together
  • 10:30 Should hardware matter in OpenStack? It has to, details and topology matters not vendor.
  • 11:20 OpenCompute – people are trying to open source hardware design
  • 11:50 When you are dealing with hardware, it matters. You have to get it right.
  • 12:40 Customers are hardware heterogeneous by design (and for ops tooling). Crowbar is neutral territory
  • 14:50 It’s not worth telling people they are wrong, because they are not. There are a lot of right ways to install OpenStack
  • 16:10 Sometimes people make expensive choices because it’s what they are comfortable with and it’s not helpful for me to them they a wrong – they are not.
  • 16:30 You get into a weird corner if you don’t tell anyone no. And an equally weird corner if you tell everyone yes.
  • 18:00 Aspirations of having an interoperable cloud was much harder than the actual work to build it
  • 18:30 Community want to say yes, “bring your code” but to operators that’s very frustrating because they want to be able to make substitutions
  • 19:30 Thinking that if something is included then it’s required – that’s not clear
  • 19:50 Interlock Dilemma [see my back reference]
  • 20:10 Orwell Animal Farm reference – “all animals equal but pigs are more equal”
  • 22:20 Rob defines DefCore, it’s not big and scary
  • 22:35 DefCore is about commercial use, not running the technical project
  • 23:35 OpenStack had to make money for the companies are paying for the developers who participate… they need to see ROI
  • 24:00 OpenStack is an infrastructure project, stability is the #1 feature
  • 24:40 You have to give a reason why you are saying no and a path to yes
  • 25:00 DefCore is test driven: quantitative results
  • 26:15 Balance between whole project and parts – examples are Swiftstack (wants Object only) and Dreamhost (wants Compute only)
  • 27:00 DefCore created core components vs platform levels
  • 27:30 No vendor has said they can implement DefCore without some effort
  • 28:10 We have outlets for vendors who do not want to implement the process
  • 28:30 The Board is not in a position to make technical call about what’s in, we had to build a process for community input
  • 29:10 We had to define something that could say, “this is it and we have to move on”
  • 29:50 What we want is for people to start with the core and then bring in the other projects. We want to know what people are adding so we can make that core in time
  • 30:10 This is not a recommendation is a base.
  • 30:35 OpenStack is a bubble – does not help if we just get together to pad each other on the back, we want to have a thriving ecosystem
  • 31:15 Question: “have vendors been selfish”
  • 31:35 Rob rephrased as “does OpenStack have a tragedy of the commons” problem
  • 32:30 We need to make sure that everyone is contributing back upstream
  • 32:50 Benefit of a Benevolent Dictator is that they can block features unless community needs are met
  • 33:10 We have NOT made it clear where companies should be contributing to the community. We are not doing a good job directing community efforts
  • 33:45 Hidden Influencers becomes OpenStack Product group
  • 34:55 Hidden Influencers were not connecting at the summit in a public way (like developers were)
  • 35:20 Developers could not really make big commitments of their time without the buy in from their managers (product and line)
  • 35:50 Subtle selfishness – focusing on your own features can disrupt the whole release where things would flow better if they helped others
  • 37:40 Rob was concerned that there was a lot of drift between developers and company’s product descriptions
  • 38:20 BYLAWS CHANGES – vote! here’s why we need to change
  • 38:50 Having whole projects designated as core sucks – code in core should be slower and less changing. Innovation at the core will break interoperability
  • 39:40 Hoping that core will help product managers understand where they are using the standard and adding values
  • 41:10 All babies are ugly > with core, that’s good. We are looking for the grown ups who can do work and deliver value. Babies are things you nurture and help grow because they have potential.
  • 42:00 We undermine our credibility in the community when we talk about projects that are babies as if they were ready.
  • 43:15 DefCore’s job was to help pick projects. If everyone is core then we look like a youth soccer team where everyone is getting a trophy
  • 44:30 Question: “What do you tell to users to instill confidence in OpenStack”
  • 44:50 first thing: focus on operations and automation. Table stakes (for any cloud) is getting your deployments automated. Puppies vs Cattle.
  • 45:25 People who were successful with early OpenStack were using automated deployments against the APIs.
  • 46:00 DevOps is a fundamental part of cloud computing – if you’re hand-built and not automated then you are old school IT.
  • 46:40 Niki references Gartner “Bimodal IT” [excellent reference, go read it!]
  • 47:20 VMWare is a great crutch for OpenStack. We can use VMWare for the puppies.
  • 47:45 OpenStack is not going to run on every servers (perhaps that’s heresy) but it does not make sense in every workload
  • 48:15 One size does not fit all – we need to be good at what we’re good at
  • 48:30 OpenStack needs to focus on doing something really well. That means helping people who want to bring automated workloads into the cloud
  • 49:20 Core was about sending a signal about what’s ready and people can rely on
  • 49:45 Back in 2011, I was saying OpenStack was ready for people who would make the operational investment
  • 50:30 We use Crowbar because it makes it easier to do automated deployments for infrastructure like Hadoop and Ceph where you want access to the physical media
  • 51:00 We should be encouraging people to use OpenStack for its use cases
  • 51:30 Existential question for OpenStack: are we a suite or product. The community is split here
  • 51:30 In comparing with Amazon, does OpenStack have to implement it or build an ecosystem to compete
  • 53:00 As soon as you make something THE OpenStack project (like Heat) you are sending a message that the alternates are not welcome
  • 54:30 OpenStack ends up in a trap if we pick a single project and make it the way that we are going do something. New implementations are going to surface from WITHIN the projects and we need to ready for that.
  • 55:15 new implementations are coming, we have to be ready for that. We can make ourselves vulnerable to splitting if we do not prepare.
  • 56:00 API vs Implementation? This is something that splits the community. Ultimately we to be an API spec but we are not ready for that. We have a lot of work to do first using the same code base.
  • 56:50 DefCore has taken a balanced approach using our diversity as a strength
  • 57:20 Bylaws did not allow for enough flexibility for what is core
  • 59:00 We need voters for the quorum!
  • 59:30 Rob recommended Rocky Grober (Huawei) and Shamail Tahir (EMC) for future shows