OpenStack has grown dramatically in many ways but we have failed to integrate development, operations and business communities in a balanced way.
My most urgent observation from Paris is that these three critical parts of the community are having vastly different dialogs about OpenStack.
At the Conference, business people were talking were about core, stability and utility while the developers were talking about features, reorganizing and expanding projects. The operators, unfortunately segregated in a different location, were trying to figure out how to share best practices and tools.
Much of this structural divergence was intentional and should be (re)evaluated as we grow.
OpenStack events are split into distinct focus areas: the conference for business people, the summit for developers and specialized days for operators. While this design serves a purpose, the community needs to be taking extra steps to ensure communication. Without that communication, corporate sponsors and users may find it easier to solve problems inside their walls than outside in the community.
The risk is clear: vendors may find it easier to work on a fork where they have business and operational control than work within the community.
Inside the community, we are working to help resolve this challenge with several parallel efforts. As a community member, I challenge you to get involved in these efforts to ensure the project balances dev, biz and ops priorities. As a board member, I feel it’s a leadership challenge to make sure these efforts converge and that’s one of the reasons I’ve been working on several of these efforts:
OpenStack Project Managers (was Hidden Influencers) across companies in the ecosystem are getting organized into their own team. Since these managers effectively direct the majority of OpenStack developers, this group will allow
DefCore Committee works to define a smaller subset of the overall OpenStack Project that will be required for vendors using the OpenStack trademark and logo. This helps the business community focus on interoperability and stability.
Technical leadership (TC) lead “Big Tent” concept aligns with DefCore work and attempts to create a stable base platform while making it easier for new projects to enter the ecosystem. I’ve got a lot to say about this, but frankly, without safeguards, this scares people in the ops and business communities.
An operations “ready state” baseline keeps the community from being able to share best practices – this has become a pressing need. I’d like to suggest as OpenCrowbar an outside of OpenStack a way to help provide an ops neutral common starting point. Having the OpenStack developer community attempting to create an installer using OpenStack has proven a significant distraction and only further distances operators from the community.
We need to get past seeing the project primarily as a technology platform. Infrastructure software has to deliver value as an operational tool for enterprises. For OpenStack to thrive, we must make sure the needs of all constituents (Dev, Biz, Ops) are being addressed.
PMs starting this group already direct the work for a super majority (>66%) of active contributors.
The primary mission for the group is to collaborate and communicate around development priorities so that we can ensure that project commitments get met.
It was recognized that the project technical leads are already strapped coordinating release and technical objectives. Further, the product managers are already but independently engaged in setting strategic direction, we cannot rely on existing OpenStack technical leadership to have the bandwidth.
This effort will succeed to the extent that we can help the broader community tied in and focus development effort back to dollars for the people paying for those developers. In my book, that’s what product managers are supposed to do. Hopefully, getting this group organized will help surface that discussion.
This is a big challenge considering that these product managers have to balance corporate, shared project and individual developers’ requirements. Overall, I think Allison Randall summarized our objectives best: “we’re herding cats in the same direction.”
When Dell pulled out from OpenCrowbar last April, I made a commitment to our community to find a way to keep it going. Since my exit from Dell early in October 2014, that commitment has taken the form of RackN.
Today, we’re ready to help people run and expand OpenCrowbar (days away from v2.1!). We’re also seeking investment to make the project more “enterprise-ready” and build integrations that extend ready state.
RackN focuses on maintenance and support of OpenCrowbar for ready state physical provisioning. We will build the community around Crowbar as an open operations core and extend it with a larger set of hardware support and extensions. We are building partnerships to build application integration (using Chef, Puppet, Salt, etc) and platform workloads (like OpenStack, Hadoop, Ceph, CloudFoundry and Mesos) above ready state.
I’ve talked with hundreds of people about the state of physical data center operations at scale. Frankly, it’s a scary state of affairs: complexity is increasing for physical infrastructure and we’re blurring the lines by adding commodity networking with local agents into the mix.
Making this jumble of stuff work together is not sexy cloud work – I describe it as internet plumbing to non-technical friends. It’s unforgiving, complex and full of sharp edge conditions; however, people are excited to hear about our hardware abstraction mission because it solves a real pain for operators.
I hope you’ll stay tuned, or even play along, as we continue the Open Ops journey.
For nearly two years, the OpenStack Board has been moving towards creating a common platform definition that can help drive interoperability. At the last meeting, the Board paused to further review one of the core tenants of the DefCore process (Item #3: Core definition can be applied equally to all usage models).
Outside of my role as DefCore chair, I see the OpenStack community asking itself an existential question: “are we one platform or a suite of projects?” I’m having trouble believing “we are both” is an acceptable answer.
During the post-meeting review, Mark Collier drafted a Foundation supported recommendation that basically creates an additional core tier without changing the fundamental capabilities & designated code concepts. This proposal has been reviewed by the DefCore committee (but not formally approved in a meeting).
The original DefCore proposed capabilities set becomes the “platform” level while capability subsets are called “components.” We are considering two initial components, Compute & Object, and both are included in the platform (see illustration below). The approach leaves the door open for new core component to exist both under and outside of the platform umbrella.
In the proposal, OpenStack vendors who meet either component or platform requirements can qualify for the “OpenStack Powered” logo; however, vendors using the only a component (instead of the full platform) will have more restrictive marks and limitations about how they can use the term OpenStack.
This approach addresses the “is Swift required?” question. For platform, Swift capabilities will be required; however, vendors will be able to implement the Compute component without Swift and implement the Object component without Nova/Glance/Cinder.
It’s important to note that there is only one yard stick for components or the platform: the capabilities groups and designed code defined by the DefCore process. From that perspective, OpenStack is one consistent thing. This change allows vendors to choose sub-components if that serves their business objectives.
It’s up to the community to prove the platform value of all those sub-components working together.
Baby Bear: Is OpenStack a single product or a “big tent” of integrated components?
After last summit, a few of us started a dialog around Hidden Influencers that helps to frame these questions in an actionable way. Now, it’s time for us to come together and talk in Paris in the hallways and specifically on Monday, November 3rd, in the afternoon (schedule link). From there, we’ll figure out about next steps using these three questions as a baseline.
If you’ve got opinions about these questions, don’t wait for Paris! I’d love to start the discussion here in the comments, on twitter (@zehicle), by phone, with email or via carrier pidgins.
I’ve been watching to see if other open “bare metal” projects would morph to match the system-level capabilities that we proved in Crowbar v1 and honed in the re-architecture of OpenCrowbar. The answer appears to be that Crowbar simply takes a broader approach to solving the physical ops repeatably problem.
Crowbar Architect Victor Lowther says “What makes Crowbar a better tool than Cobbler, Razor, or Foreman is that Crowbar has an orchestration engine that can be used to safely and repeatably deploy complex workloads across large numbers of machines. This is different from (and better than, IMO) just being able to hand responsibility off to Chef/Puppet/Salt, because we can manage the entire lifecycle of a machine where Cobbler, Razor and Chef cannot, we can describe how we want workloads configured at a more abstract level than Foreman can, and we do it all using the same API and UI.”
Since we started with a vision of an integrated system to address the “apply-rinse-repeat” cycle; it’s no surprise that Crowbar remains the only open platform that’s managed to crack the complete physical deployment life-cycle.
The Crowbar team realized that it’s not just about automation setting values: physical ops requires orchestration to make sure the values are set in the correct sequence on the appropriate control surface including DNS, DHCP, PXE, Monitoring, et cetera. Unlike architectures for aaS platforms, the heterogeneous nature of the physical control planes requires a different approach.
We’ve seen that making more and more complex kickstart scripts or golden images is not a sustainable solution. There is simply too much hardware variation and dependency thrash for operators to collaborate with those tools. Instead, we’ve found that decomposing the provisioning operations into functional layers with orchestration is much more multi-site repeatable.
Accepting that physical ops (discovered infrastructure) is fundamentally different from cloud ops (created infrastructure) has been critical to architecting platforms that were resilient enough for the heterogeneous infrastructure of data centers.
If we want to start cleaning up physical ops, we need to stop looking at operating system provisioning in isolation and start looking at the full server bring up as just a part of a broader system operation that includes networking, management and operational integration.
I’ve been trying to explain the pain Tao of physical ops in a way that’s accessible to people without scale ops experience. It comes down to a yin-yang of two elements: exploding complexity and iterative learning.
Exploding complexity is pretty easy to grasp when we stack up the number of control elements inside a single server (OS RAID, 2 SSD cache levels, 20 disk JBOD, and UEFI oh dear), the networks that server is connected to, the multi-layer applications installed on the servers, and the change rate of those applications. Multiply that times 100s of servers and we’ve got a problem of unbounded scope even before I throw in SDN overlays.
But that’s not the real challenge! The bigger problem is that it’s impossible to design for all those parameters in advance.
When my team started doing scale installs 5 years ago, we assumed we could ship a preconfigured system. After a year of trying, we accepted the reality that it’s impossible to plan out a scale deployment; instead, we had to embrace a change tolerant approach that I’ve started calling “Apply, Rinse, Repeat.”
Using Crowbar to embrace the in-field nature of design, we discovered a recurring pattern of installs: we always performed at least three full cycle installs to get to ready state during every deployment.
The first cycle was completely generic to provide a working baseline and validate the physical environment.
The second cycle attempted to integrate to the operational environment and helped identify gaps and needed changes.
The third cycle could usually interconnect with the environment and generally exposed new requirements in the external environment
The subsequent cycles represented additional tuning, patches or redesigns that could only be realized after load was applied to the system in situ.
Every time we tried to shortcut the Apply-Rinse-Repeat cycle, it actually made the total installation longer! Ultimately, we accepted that the only defense was to focus on reducing A-R-R cycle time so that we could spend more time learning before the next cycle started.