Voting time for OpenStack! Your help needed to push Bylaws & DefCore forward.

Posted on January 11, 2015 by Rob H

1/17 Update: We did it! We reached quorum and approved all the changes! Also, I am honored to have been re-elected to the Board. Thank you for the support!

What does OpenStack mean to you?

To me, it’s more than infrastructure software: it’s a community of people and companies working together in revolutionary way. To make that work, we need people who invest time to bring together diverse interests and perspectives.

I have worked hard to provide neutral and inclusive view points on the OpenStack board. The critical initiatives I focused on [DefCore, Gold Membership, Product Managers] need focused leadership to progress. I have proven my commitment to those efforts and would be honored to be re-elected [see my platforms for 2012, 2013 & 2014].

But more important than voting for me, is that we reach a 25% quorum!

Why? Because there are critical changes to the OpenStack bylaws on the ballot.

What type of changes? Basically, the changes allow OpenStack governance to keep up with the pace of our evolution. These are pragmatic changes that have been thoroughly discussed in the community. They enable OpenStack to:

set a smarter quorum – past elections only achieved half the required turn out to changes the bylaws
enable the DefCore process – current core is defined as simply using code in select projects.
improve governance of required committees
tweak release language to match current practice,

To hit 25%, we need everyone, not just the typical evangelists to encourage voting. We need you to help spread the word and get others to do the same.

I’m counting on you, please help.

10 pounds of OpenStack cloud in a 5 pound bag? Do we need a bigger bag?

Posted on December 31, 2014 by Rob H

Yesterday, I posted about cloud distruptors that are pushing the boundaries of cloud. The same forces pull at OpenStack where we are working to balance between including all aspects of running workloads and focusing on a stable foundation.

Note: I am seeking re-election to the 2015 OpenStack Board. Voting starts 1/12.

For weeks, I’ve been reading and listening to people inside and outside the community. There is considerable angst about the direction of OpenStack. We need to be honest and positive about challenges without simply throwing stones in our hall of mirrors.

Closing 2014, OpenStack has gotten very big, very fast. We’ve exploded scope, contributions and commercial participants. Unfortunately, our process infrastructure (especially the governance by-laws) simply have not kept pace. It’s not a matter of scaling processes we’ve got; many of the challenges created by growth require new approaches and thinking (Thierry’s post).

In 2015, we’re trying to put 10 pounds of OpenStack in a 5 pound bag. That means we have to either a) shed 5 pounds or b) get a bigger bag. In classic OpenStack style, we’re sort of doing both: identifying a foundational base while expanding to allow more subprojects.

To my ear, most users, operators and business people would like to see the focus being on the getting the integrated release scope solid. So, in spirit of finding 5 pounds to shave, I’ve got five “shovel ready” items that should help:

Prioritizing stability as our #1 feature. Accomplishing this will require across the broad alignment of the vendor’s product managers to hold back on their individual priorities in favor of community. We’ve started this effort but it’s going to take time to create the collaboration needed.
Sending a clear signal about the required baseline for OpenStack. That’s the purpose of DefCore and should be felt as we work on the Icehouse and Juno definitions.
Alignment of the Board DefCore project with Technical Committee’s Levels/Big Tent initiative. By design, these efforts interconnect. We need to make sure the work is coordinated so that we send a clearly aligned message to the technical, operator, vendor and user communities.
Accelerate changes from single node gate to something that’s either a) more services focused or b) multi-node. OpenStack’s scale of community development requires automation to validate the new contributions do not harm the existing code base (the gate). The current single-node gate does not reflect the multi-node environments that users target with the code. While it’s technically challenging to address this mismatch, it’s also essential so we ensure that we’re able to validate multi-node features.
Continue to reduce drama in the open source processes. OpenStack is infrastructure software that should enable an exciting and dynamic next generation of IT. I hear people talk about CloudStack as “it’s not as exciting or active a community but their stuff just works.” That’s what enterprises and operators want. Drama is great for grabbing headings but not so great for building solid infrastructure.

What is the downside to OpenStack if we cannot accomplish these changes? Forks.

I already see a clear pattern where vendors are creating their own distros (which are basically shallow forks) to preserve their own delivery cycle. OpenStack’s success is tied to its utility for the customers of vendors who fund the contributors. When the cost of being part of the community outweighs the value, those shallow forks may become true independent products.

In the case of potential forks, they allow vendors to create their own bag and pick how many pounds of cloud they want to carry. It’s our job as a community in 2015 to make sure that we’ve reduced that temptation.

1/9/15 Note: Here’s the original analogy image used for this post

Nextcast #14 Transcription on OpenStack & Crowbar > “we can’t hand out trophies to everyone”

Posted on December 26, 2014 by Rob H

Last week, I was a guest on the NextCast OpenStack podcast hosted by Niki Acosta (EMC) [Jeff Dickey could not join]. I’ve taken some time to transcribe highlights.

We had a great discussion about OpenStack, Ops and Crowbar. I appreciate Niki’s insightful questions and an opportunity to share my opinions. I feel that we covered years of material in just 1 hour and I appreciate the opportunity to appear on the podcast.

Video from full post (youtube) and the audio for download.

Plus, a FULL TRANSCRIPT! Here’s my Next Cast #14 Short Transcripton

The objective of this transcription is to help navigate the recording, not replace it. I did not provide complete context for remarks.

04:30 Birth of Crowbar (to address Ops battle scars)
08:00 The need for repeatable Ready State baseline to help community work together
10:30 Should hardware matter in OpenStack? It has to, details and topology matters not vendor.
11:20 OpenCompute – people are trying to open source hardware design
11:50 When you are dealing with hardware, it matters. You have to get it right.
12:40 Customers are hardware heterogeneous by design (and for ops tooling). Crowbar is neutral territory
14:50 It’s not worth telling people they are wrong, because they are not. There are a lot of right ways to install OpenStack
16:10 Sometimes people make expensive choices because it’s what they are comfortable with and it’s not helpful for me to them they a wrong – they are not.
16:30 You get into a weird corner if you don’t tell anyone no. And an equally weird corner if you tell everyone yes.
18:00 Aspirations of having an interoperable cloud was much harder than the actual work to build it
18:30 Community want to say yes, “bring your code” but to operators that’s very frustrating because they want to be able to make substitutions
19:30 Thinking that if something is included then it’s required – that’s not clear
19:50 Interlock Dilemma [see my back reference]
20:10 Orwell Animal Farm reference – “all animals equal but pigs are more equal”
22:20 Rob defines DefCore, it’s not big and scary
22:35 DefCore is about commercial use, not running the technical project
23:35 OpenStack had to make money for the companies are paying for the developers who participate… they need to see ROI
24:00 OpenStack is an infrastructure project, stability is the #1 feature
24:40 You have to give a reason why you are saying no and a path to yes
25:00 DefCore is test driven: quantitative results
26:15 Balance between whole project and parts – examples are Swiftstack (wants Object only) and Dreamhost (wants Compute only)
27:00 DefCore created core components vs platform levels
27:30 No vendor has said they can implement DefCore without some effort
28:10 We have outlets for vendors who do not want to implement the process
28:30 The Board is not in a position to make technical call about what’s in, we had to build a process for community input
29:10 We had to define something that could say, “this is it and we have to move on”
29:50 What we want is for people to start with the core and then bring in the other projects. We want to know what people are adding so we can make that core in time
30:10 This is not a recommendation is a base.
30:35 OpenStack is a bubble – does not help if we just get together to pad each other on the back, we want to have a thriving ecosystem
31:15 Question: “have vendors been selfish”
31:35 Rob rephrased as “does OpenStack have a tragedy of the commons” problem
32:30 We need to make sure that everyone is contributing back upstream
32:50 Benefit of a Benevolent Dictator is that they can block features unless community needs are met
33:10 We have NOT made it clear where companies should be contributing to the community. We are not doing a good job directing community efforts
33:45 Hidden Influencers becomes OpenStack Product group
34:55 Hidden Influencers were not connecting at the summit in a public way (like developers were)
35:20 Developers could not really make big commitments of their time without the buy in from their managers (product and line)
35:50 Subtle selfishness – focusing on your own features can disrupt the whole release where things would flow better if they helped others
37:40 Rob was concerned that there was a lot of drift between developers and company’s product descriptions
38:20 BYLAWS CHANGES – vote! here’s why we need to change
38:50 Having whole projects designated as core sucks – code in core should be slower and less changing. Innovation at the core will break interoperability
39:40 Hoping that core will help product managers understand where they are using the standard and adding values
41:10 All babies are ugly > with core, that’s good. We are looking for the grown ups who can do work and deliver value. Babies are things you nurture and help grow because they have potential.
42:00 We undermine our credibility in the community when we talk about projects that are babies as if they were ready.
43:15 DefCore’s job was to help pick projects. If everyone is core then we look like a youth soccer team where everyone is getting a trophy
44:30 Question: “What do you tell to users to instill confidence in OpenStack”
44:50 first thing: focus on operations and automation. Table stakes (for any cloud) is getting your deployments automated. Puppies vs Cattle.
45:25 People who were successful with early OpenStack were using automated deployments against the APIs.
46:00 DevOps is a fundamental part of cloud computing – if you’re hand-built and not automated then you are old school IT.
46:40 Niki references Gartner “Bimodal IT” [excellent reference, go read it!]
47:20 VMWare is a great crutch for OpenStack. We can use VMWare for the puppies.
47:45 OpenStack is not going to run on every servers (perhaps that’s heresy) but it does not make sense in every workload
48:15 One size does not fit all – we need to be good at what we’re good at
48:30 OpenStack needs to focus on doing something really well. That means helping people who want to bring automated workloads into the cloud
49:20 Core was about sending a signal about what’s ready and people can rely on
49:45 Back in 2011, I was saying OpenStack was ready for people who would make the operational investment
50:30 We use Crowbar because it makes it easier to do automated deployments for infrastructure like Hadoop and Ceph where you want access to the physical media
51:00 We should be encouraging people to use OpenStack for its use cases
51:30 Existential question for OpenStack: are we a suite or product. The community is split here
51:30 In comparing with Amazon, does OpenStack have to implement it or build an ecosystem to compete
53:00 As soon as you make something THE OpenStack project (like Heat) you are sending a message that the alternates are not welcome
54:30 OpenStack ends up in a trap if we pick a single project and make it the way that we are going do something. New implementations are going to surface from WITHIN the projects and we need to ready for that.
55:15 new implementations are coming, we have to be ready for that. We can make ourselves vulnerable to splitting if we do not prepare.
56:00 API vs Implementation? This is something that splits the community. Ultimately we to be an API spec but we are not ready for that. We have a lot of work to do first using the same code base.
56:50 DefCore has taken a balanced approach using our diversity as a strength
57:20 Bylaws did not allow for enough flexibility for what is core
59:00 We need voters for the quorum!
59:30 Rob recommended Rocky Grober (Huawei) and Shamail Tahir (EMC) for future shows

OpenStack DefCore Enters Execution Phase. Help!

Posted on December 11, 2014 by Rob H

OpenStack DefCore Committee has established the principles and first artifacts required for vendors using the OpenStack trademark. Over the next release cycle, we will be applying these to the Ice House and Juno releases.

Learn more? Hear about it LIVE! Rob will be doing two sessions about DefCore next week (will be recorded):

Tues Dec 16 at 9:45 am PST- OpenStack Podcast #14 with Jeff Dickey
Thurs Dec 18 at 9:00 am PST – Online Meetup about DefCore with Rafael Knuth (optional RSVP)

At the December 2014 OpenStack Board meeting, we completed laying the foundations for the DefCore process that we started April 2013 in Portland. These are a set of principles explaining how OpenStack will select capabilities and code required for vendors using the name OpenStack. We also published the application of these governance principles for the Havana release.

The OpenStack Board approved DefCore principles to explain
the landscape of core including test driven capabilities and designated code (approved Nov 2013)
the twelve criteria used to select capabilities (approved April 2014)
the creation of component and framework layers for core (approved Oct 2014)
the ten principles used to select designated sections (approved Dec 2014)

To test these principles, we’ve applied them to Havana and expressed the results in JSON format: Havana Capabilities and Havana Designated Sections. We’ve attempted to keep the process transparent and community focused by keeping these files as text and using the standard OpenStack review process.

DefCore’s work is not done and we need your help! What’s next?

Vote about bylaws changes to fully enable DefCore (change from projects defining core to capabilities)
Work out going forward process for updating capabilities and sections for each release (once authorized by the bylaws, must be approved by Board and TC)
Bring Havana work forward to Ice House and Juno.
Help drive Refstack process to collect data from the field

To thrive, OpenStack must better balance dev, ops and business needs.

Posted on November 21, 2014 by Rob H

OpenStack has grown dramatically in many ways but we have failed to integrate development, operations and business communities in a balanced way.

My most urgent observation from Paris is that these three critical parts of the community are having vastly different dialogs about OpenStack.

At the Conference, business people were talking were about core, stability and utility while the developers were talking about features, reorganizing and expanding projects. The operators, unfortunately segregated in a different location, were trying to figure out how to share best practices and tools.

Much of this structural divergence was intentional and should be (re)evaluated as we grow.

OpenStack events are split into distinct focus areas: the conference for business people, the summit for developers and specialized days for operators. While this design serves a purpose, the community needs to be taking extra steps to ensure communication. Without that communication, corporate sponsors and users may find it easier to solve problems inside their walls than outside in the community.

The risk is clear: vendors may find it easier to work on a fork where they have business and operational control than work within the community.

Inside the community, we are working to help resolve this challenge with several parallel efforts. As a community member, I challenge you to get involved in these efforts to ensure the project balances dev, biz and ops priorities. As a board member, I feel it’s a leadership challenge to make sure these efforts converge and that’s one of the reasons I’ve been working on several of these efforts:

OpenStack Project Managers (was Hidden Influencers) across companies in the ecosystem are getting organized into their own team. Since these managers effectively direct the majority of OpenStack developers, this group will allow
DefCore Committee works to define a smaller subset of the overall OpenStack Project that will be required for vendors using the OpenStack trademark and logo. This helps the business community focus on interoperability and stability.
Technical leadership (TC) lead “Big Tent” concept aligns with DefCore work and attempts to create a stable base platform while making it easier for new projects to enter the ecosystem. I’ve got a lot to say about this, but frankly, without safeguards, this scares people in the ops and business communities.
An operations “ready state” baseline keeps the community from being able to share best practices – this has become a pressing need. I’d like to suggest as OpenCrowbar an outside of OpenStack a way to help provide an ops neutral common starting point. Having the OpenStack developer community attempting to create an installer using OpenStack has proven a significant distraction and only further distances operators from the community.

We need to get past seeing the project primarily as a technology platform. Infrastructure software has to deliver value as an operational tool for enterprises. For OpenStack to thrive, we must make sure the needs of all constituents (Dev, Biz, Ops) are being addressed.

Leveling OpenStack’s Big Tent: is OpenStack a product, platform or suite?

Posted on November 21, 2014 by Rob H

Question of the day: What should OpenStack do with all those eager contributors? Does that mean expanding features or focusing on a core?

In the last few months, the OpenStack technical leadership (Sean Dague, Monty Taylor) has been introducing two interconnected concepts: big tent and levels.

Big tent means expanding the number of projects to accommodate more diversity (both in breath and depth) in the official OpenStack universe. This change accommodates the growth of the community.
Levels is a structured approach to limiting integration dependencies between projects. Some OpenStack components are highly interdependent and foundational (Keystone, Nova, Glance, Cinder) while others are primarily consumers (Horizon, Saraha) of lower level projects.

These two concepts are connected because we must address integration challenges that make it increasingly difficult to make changes within the code base. If we substantially expand the code base with big tent then we need to make matching changes to streamline integration efforts. The levels proposal reflects a narrower scope at the base than we currently use.

By combining big tent and levels, we are simultaneously growing and shrinking: we grow the community and scope while we shrink the integration points. This balance may be essential to accommodate OpenStack’s growth.

UNIVERSALLY, the business OpenStack community who wants OpenStack to be a product. Yet, what’s included in that product is unclear.

Expanding OpenStack projects tends to turn us into a suite of loosely connected functions rather than a single integrated platform with an ecosystem. Either approach is viable, but it’s not possible to be both simultaneously.

On a cautionary note, there’s an anti-Big Tent position I heard expressed at the Paris Conference. It’s goes like this: until vendors start generating revenue from the foundation components to pay for developer salaries; expanding the scope of OpenStack is uninteresting.

Recent DefCore changes also reflect the Big Tent thinking by adding component and platform levels. This change was an important and critical compromise to match real-world use patterns by companies like Swiftstack (Object), DreamHost (Compute+Ceph), Piston (Compute+Ceph) and others; however, it creates the need to explain “which OpenStack” these companies are using.

I believe we have addressed interoperability in this change. It remains to be seen if OpenStack vendors will choose to offer the broader platform or limit to themselves to individual components. If vendors chase the components over platform then OpenStack becomes a suite of loosely connect products. It’s ultimately a customer and market decision.

It’s not too late to influence these discussions! I’m very interested in hearing from people in the community which direction they think the project should go.

OpenStack DefCore Process Flow: Community Feedback Cycles for Core [6 points + chart]

Posted on September 2, 2014 by Rob H

If you’ve been following my DefCore posts, then you already know that DefCore is an OpenStack Foundation Board managed process “that sets base requirements by defining 1) capabilities, 2) code and 3) must-pass tests for all OpenStack™ products. This definition uses community resources and involvement to drive interoperability by creating the minimum standards for products labeled OpenStack™.”

In this post, I’m going to be very specific about what we think “community resources and involvement” entails.

The draft process flow chart was provided to the Board at our OSCON meeting without additional review. It below boils down to a few key points:

We are using the documents in the Gerrit review process to ensure that we work within the community processes.
Going forward, we want to rely on the technical leadership to create, cluster and describe capabilities. DefCore bootstrapped this process for Havana. Further, Capabilities are defined by tests in Tempest so test coverage gaps (like Keystone v2) translate into Core gaps.
We are investing in data driven and community involved feedback (via Refstack) to engage the largest possible base for core decisions.
There is a “safety valve” for vendors to deal with test scenarios that are difficult to recreate in the field.
The Board is responsible for approving the final artifacts based on the recommendations. By having a transparent process, community input is expected in advance of that approval.
The process is time sensitive. There’s a need for the Board to produce Core definition in a timely way after each release and then feed that into the next one. Ideally, the definitions will be approved at the Board meeting immediately following the release.

Process shows how the key components: designated sections and capabilities start from the previous release’s version and the DefCore committee manages the update process. Community input is a vital part of the cycle. This is especially true for identifying actual use of the capabilities through the Refstack data collection site.

Blue is for Board activities
Yellow is or user/vendor community activities
Green is for technical community activities
White is for process artifacts

This process is very much in draft form and any input or discussion is welcome! I expect DefCore to take up formal review of the process in October.

Your baby is ugly! Picking which code is required for Commercial Core.

Posted on August 18, 2014 by Rob H

There’s no point in sugar-coating this: selecting API and code sections for core requires making hard choices and saying no. DefCore makes this fair by 1) defining principles for selection, 2) going slooooowly to limit surprises and 3) being transparent in operation. When you’re telling someone who their baby is not handsome enough you’d better be able to explain why.

The truth is that from DefCore’s perspective, all babies are ugly. If we are seeking stability and interoperability, then we’re looking for adults not babies or adolescents.

Explaining why is exactly what DefCore does by defining criteria and principles for our decisions. When we do it right, it also drives a positive feedback loop in the community because the purpose of designated sections is to give clear guidance to commercial contributors where we expect them to be contributing upstream. By making this code required for Core, we are incenting OpenStack vendors to collaborate on the features and quality of these sections.

This does not lessen the undesignated sections! Contributions in those areas are vital to innovation; however, they are, by design, more dynamic, specialized or single vendor than the designated areas.

The seven principles of designated sections (see my post with TC member Michael Still) as defined by the Technical Committee are:

Should be DESIGNATED:

code provides the project external REST API, or
code is shared and provides common functionality for all options, or
code implements logic that is critical for cross-platform operation

Should NOT be DESIGNATED:

code interfaces to vendor-specific functions, or
project design explicitly intended this section to be replaceable, or
code extends the project external REST API in a new or different way, or
code is being deprecated

While the seven principles inform our choices, DefCore needs some clarifications to ensure we can complete the work in a timely, fair and practical way. Here are our additions:

8. UNdesignated by Default

Unless code is designated, it is assumed to be undesignated.
This aligns with the Apache license.
We have a preference for smaller core.

9. Designated by Consensus

If the community cannot reach a consensus about designation then it is considered undesignated.
Time to reach consensus will be short: days, not months
Except obvious trolling, this prevents endless wrangling.
If there’s a difference of opinion then the safe choice is undesignated.

10. Designated is Guidance

Loose descriptions of designated sections are acceptable.
The goal is guidance on where we want upstream contributions not a code inspection police state.
Guidance will be revised per release as part of the DefCore process.

In my next DefCore post, I’ll review how these 10 principles are applied to the Havana release that is going through community review before Board approval.

Patchwork Onion delivers stability & innovation: the graphics that explains how we determine OpenStack Core

Posted on August 12, 2014 by Rob H

This post was coauthored by the DefCore chairs, Rob Hirschfeld & Joshua McKenty.

The OpenStack board, through the DefCore committee, has been working to define “core” for commercial users using a combination of minimum required capabilities (APIs) and code (Designated Sections). These minimums are decided on a per project basis so it can be difficult to visualize the impact on the overall effect on the Integrated Release.

We’ve created the patchwork onion graphic to help illustrate how core relates to the integrated release. While this graphic is pretty complex, it was important to find a visual way to show how different DefCore identifies distinct subsets of APIs and code from each project. This graphic tries to show how that some projects have no core APIs and/or code.

For OpenStack to grow, we need to have BOTH stability and innovation. We need to give clear guidance to the community what is stable foundation and what is exciting sandbox. Without that guidance, OpenStack is perceived as risky and unstable by users and vendors. The purpose of defining “Core” is to be specific in addressing that need so we can move towards interoperability.

Interoperability enables an ecosystem with multiple commercial vendors which is one of the primary goals of the OpenStack Foundation.

Originally, we thought OpenStack would have “core” and “non-core” projects and we baked that expectation into the bylaws. As we’ve progressed, it’s clear that we need a less binary definition. Projects themselves have a maturity cycle (ecosystem -> incubated -> integrated) and within the project some APIs are robust and stable while others are innovative and fluctuating.

Encouraging this mix of stabilization and innovation has been an important factor in our discussions about DefCore. Growing the user base requires encouraging stability and growing the developer base requires enabling innovation within the same projects.

The consequence is that we are required to clearly define subsets of capabilities (APIs) and implementation (code) that are required within each project. Designating 100% of the API or code as Core stifles innovation because stability dictates limiting changes while designating 0% of the code (being API only) lessens the need to upstream. Core reflects the stability and foundational nature of the code; unfortunately, many people incorrectly equate “being core” with the importance of the code, and politics ensues.

To combat the politics, DefCore has taken a transparent, principles-based approach to selecting core. You can read about in Rob’s upcoming “Ugly Babies” post (check back on 8/14) .

Share the love & vote for OpenStack Paris Summit Sessions (closes Wed 8/6)

Posted on August 5, 2014 by Rob H

This is a friendly PSA that OpenStack Paris Summit session community voting ends on Wednesday 8/6. There are HUNDREDS (I heard >1k) submissions so please set aside some time to review a handful.

MY PLEA TO YOU > There is a tendency for companies to “vote-up” sessions from their own employees. I understand the need for the practice BUT encourage you to make time to review other sessions too. Affiliation voting is fine, robot voting is not.

If you are interested topics that I discuss on this blog, here’s a list of sessions I’m involved in:

Rob Hirschfeld

On Computing, Containers, Cloud & Tech Culture

Category Archives: DefCore