Operators, they don’t want to swim Upstream

Operators Dinner 11/10

Nov 10, Palo Alto Operators Dinner

Last Tuesday, I had the honor of joining an OpenStack scale operators dinner. Foundation executives, Jonathan Bryce and Lauren Sell, were also on the guest list so talk naturally turned to “how can OpenStack better support operators.” Notably, the session was distinctly not OpenStack bashing.

The conversation was positive, enthusiastic and productive, but one thing was clear: the OpenStack default “we’ll fix it in the upstream” answer does not work for this group of operators.

What is upstreaming?  A sans nuance answer is that OpenStack drives fixes and changes in the next community release (longer description).  The project and community have a tremendous upstream imperative that pervades the culture so deeply that we take it for granted.  Have an issue with OpenStack?  Submit a patch!  Is there any other alternative?

Upstreaming [to trunk] makes perfect sense considering the project vendor structure and governance; however, it is a very frustrating experience for operators.   OpenStack does have robust processes to backport fixes and sustain past releases and documentation; yet, the feeling at the table was that they are not sufficiently operator focused.

Operators want fast, incremental and pragmatic corrections to the code and docs they are deploying (which is often two releases back).  They want it within the community, not from individual vendors.

There are great reasons for focusing on upstream trunk.  It encourages vendors to collaborate and makes it much easier to add and expand the capabilities of the project.  Allowing independent activity on past releases creates a forward integration mess and could make upgrades even harder.  It will create divergence on APIs and implementation choices.

The risk of having a stable, independently sustained release is that operators have less reason to adopt the latest shiny release.  And that is EXACTLY what they are asking for.

Upstreaming is a core value to OpenStack and essential to our collaborative success; however, we need to consider that it is not the right answer to all questions.  Discussions at that dinner reinforced that pushing everything to latest trunk creates a significant barrier for OpenStack operators and users.

What are your experiences?  Is there a way to balance upstreaming with forking?  How can we better serve operators?

To avoid echo chamber, OpenStack must embrace competitive cloud ecosystem

Japanese Bullet Train View

I was in Japan before the Tokyo summit on a bullet train to Kyoto watching the mix of heavy industry and bucolic mountains pass by. That scene reflects an OpenStack duality: we want to be both a dominant platform delivering core cloud services and an open source values driven collective.

First, I fundamentally believe in the success of OpenStack as the open virtual infrastructure management platform.

I believe that we have solved the virtual compute/storage/network problem sufficiently to become the de facto open IaaS platform. While not perfect, the technologies are sufficient assuming we continue to improve ease of use and operational hardening. Pursing that base capability is my primary motivation for DefCore work.

I don’t believe that the OpenStack community is, or should try to become, the authority on “all things cloud.”

In the presence of Amazon, VMware, Microsoft and Google, we cannot make that claim with any degree of self-respect. Even newcomers like DigitalOcean have an undeniable footprint and influence. Those vendor platforms drive cloud ecosystems and technologies which foster fast innovation because there is no friction to joining their ecosystems and they are sufficiently large and stable enough to represent a target market. We’ve seen clear signs from Rackspace, HP and others that platform diversity improves cloud strength.

I continue to think we (OpenStack) spend too much time evaluating what is “in” or “out” of the project and too little time talking about what’s “on,” “under” and “with” the project like Kubernetes, Mesos, Docker, SDN, Hadoop and Ceph. That type of thinking creates distance between OpenStack efforts and the majority of the market.

What motivates the drive to an all open captive community? It’s the reasonable concern that critical parts of the infrastructure will become pay-to-play. For example, what if a non-OpenStack alternative to Heat Orchestration gained popularity for OpenStack implementers. Perhaps something that ran on Amazon also. That would create external pressure that would drive internal priorities. These “non-OpenStack” products would then have influence without having to contribute back to upstream.

Can we afford to have external entities driving internal priorities? Hell yes, that’s what customer adoption looks like.

OpenStack does not own the market sufficiently to create cloud echo chamber. The next wave of cloud innovation (my money is on container platforms) will follow the path of least resistance and widest adoption. We need to embrace that these innovations will not all be inside our community so that we can welcome them as part of our ecosystem. The community needs to find peace with that.

As Docker rises above (and disrupts) clouds, I’m thinking about their community landscape

Watching the lovefest of DockerConf last week had me digging up my April 2014 “Can’t Contain(erize) the Hype” post.  There’s no doubt that Docker (and containers more broadly) is delivering on it’s promise.  I was impressed with the container community navigating towards an open platform in RunC and vendor adoption of the trusted container platforms.

I’m a fan of containers and their potential; yet, remotely watching the scope and exuberance of Docker partnerships seems out of proportion with the current capabilities of the technology.

The latest update to the Docker technology, v1.7, introduces a lot of important network, security and storage features.  The price of all that progress is disruption to ongoing work and integration to the ecosystem.

There’s always two sides to the rapid innovation coin: “Sweet, new features!  Meh, breaking changes to absorb.”

Docker Ecosystem Explained

Docker Ecosystem Explained

There remains a confusion between Docker the company and Docker the technology.  I like how the chart (right) maps out potential areas in the Docker ecosystem.  There’s clearly a lot of places for companies to monetize the technology; however, it’s not as clear if the company will be able to secede lucrative regions, like orchestration, to become a competitive landscape.

While Docker has clearly delivered a lot of value in just a year, they have a fair share of challenges ahead.  

If OpenStack is a leading indicator, we can expect to see vendor battlegrounds forming around networking and storage.  Docker (the company) has a chance to show leadership and build community here yet could cause harm by giving up the arbitrator role be a contender instead.

One thing that would help control the inevitable border skirmishes will be clear definitions of core, ecosystem and adjacencies.  I see Docker blurring these lines with some of their tools around orchestration, networking and storage.  I believe that was part of their now-suspended kerfuffle with CoreOS.

Thinking a step further, parts of the Docker technology (RunC) have moved over to Linux Foundation governance.  I wonder if the community will drive additional shared components into open governance.  Looking at Node.js, there’s clear precedent and I wonder if Joyent’s big Docker plans have them thinking along these lines.

Hidden costs of Cloud? No surprises, it’s still about complexity = people cost

Last week, Forbes and ZDnet posted articles discussing the cost of various cloud (451 source material behind wall) full of dollar per hour costs analysis.  Their analysis talks about private infrastructure being an order of magnitude cheaper (yes, cheaper) to own than public cloud; however, the open source price advantages offered by OpenStack are swallowed by added cost of finding skilled operators and its lack of maturity.

At the end of the day, operational concerns are the differential factor.

The Magic 8 Cube

The Magic 8 Cube

These articles get tied down into trying to normalize clouds to $/vm/hour analysis and buried the lead that the operational decisions about what contributes to cloud operational costs.   I explored this a while back in my “magic 8 cube” series about six added management variations between public and private clouds.

In most cases, operations decisions is not just about cost – they factor in flexibility, stability and organizational readiness.  From that perspective, the additional costs of public clouds and well-known stacks (VMware) are easily justified for smaller operations.  Using alternatives means paying higher salaries and finding talent that requires larger scale to justify.

Operational complexity is a material cost that strongly detracts from new platforms (yes, OpenStack – we need to address this!)

Unfortunately, it’s hard for people building platforms to perceive the complexity experienced by people outside their community.  We need to make sure that stability and operability are top line features because complexity adds a very real cost because it comes directly back to cost of operation.

In my thinking, the winners will be solutions that reduce BOTH cost and complexity.  I’ve talked about that in the past and see the trend accelerating as more and more companies invest in ops automation.

How Nebula shows why the OpenStack community (and OSS in general) should care about profits

Fancy DogsIn dog piling onto the news about early OpenStack provider Nebula’s demise, I’ll use their news to reinforce my belief that open communities need to ensure that funds are flowing to leaders and contributors.  This is one of the reasons that I invest time in DefCore: having a clear base product definition helps create healthy vendors.

Open source software is not simply free VC funded projects.  The industry needs a stable transparent supply chain of software to build on.

I am not asserting anything about Nebula’s business or model.  To me, it is a single downward data point in my ongoing review of vendor health in the OpenStack community.  OpenStack features and functions will not matter if vendors in the community cannot afford to sustain them.

For enterprises to rely on open source software supply chains, they need to make sure they are paying into the community.  The simplest route to “paying in” is through vendors.

My OpenStack Super User Interview [cross-post]

This post of my interview for the OpenStack Super User site originally appeared there on 3/23 under the title “OpenStack at 10: different code, same collaboration?”

With over 15 years of cloud experience, Rob Hirschfeld also goes way back with OpenStack. His involvement dates to before it was officially founded and he was also one of the initial Board Members. In addition to his role as Individual Director, Hirschfeld currently chairs the DefCore committee. He’ll be speaking about DefCore at the upcoming Vancouver Summit with Alan Clark, Egle Sigler and Sean Roberts.

He talks to Superuser about the importance of patches, priorities for 2015 and why you should care about OpenStack vendors making money.

Superuser: You’ve been with the project since before it started, where do you hope it will be in five years?

In five years, I expect that nearly every line of code will have been replaced. The thing that will endure is the community governance and interaction models that we’re working out to ensure commercial collaboration.

[3/24 Added Clarification Note: I find humbled watching traditionally open-unfriendly corporations using OpenStack to learn how to become open source collaborations.  Our governance choices will have long lasting ramifications in the industry.] 

What is something that a lot of people don’t know about OpenStack?

It was essentially a “rewrite fork” of Eucalyptus created because they would not accept patches.  That’s a cautionary tale about why accepting patches is essential that should not get lost from the history books.

Any thoughts on your first steps to the priorities you laid out in your candidacy profile?

I’ve already started to get DefCore into an execution phase with additional Board and Foundation leadership joining into the effort.  We’ve set a very active schedule of meetings with two sub-committees running in parallel…It’s going to be a busy spring.

You say that the company you founded, RackN, is not creating an OpenStack product. How are you connected to the community?

RackN supports OpenCrowbar which provides a physical ready state infrastructure for scale platforms like OpenStack. We are very engaged in the community from below by helping make other distributions, vendors and operators successful.

What are the next steps to creating the “commercially successful ecosystem” you mentioned in your candidacy profile? What are the biggest obstacles to this?

We have to make stability and scale a critical feature. This will mean slowing features and new projects; however, I hear a lot of frustration that OpenStack is not focused on delivering a solid base.

Without a base, the vendors cannot build profitable products.  Without profits, they cannot keep funding the project. This may be radical for an open project, but I think everyone needs to care more if vendors are making money.

What are some more persistent myths about the cloud?

That the word cloud really means anything.  Everyone has their own definition.  Mine is “infrastructure with an API” but I’d happily tell you it’s also about process and ops.

Who are your real-life heroes?

FIRST (For Inspiration and Recognition of Science and Technology) founders Dean Kamen and Woodie Flowers. They executed a real vision about how to train for both competition and collaboration in the next generation of engineers.  Their efforts in building the next generation of leaders really impact how we will should open source collaboration. That’s real innovation.

What do you hope to get out of the next summit?

First, I want to see vendors passing DefCore requirements.  After that, I’d like to see the operators get more equal treatment and I’m hoping to spend more time working with them so they can create places to share knowledge.

What’s your favorite/most important OpenStack debate?

There are two.  First, I think the API vs. implementation is a critical growth curve for OpenStack.  We need to mature past being so implementation driven so we can have stand alone APIs.

Second, I think the “benevolent dictator” discussion is useful. Since we are never going to have one, we need a real discussion about how to define and defend project wide priorities in a meaningful way.  Resolving both items is essential to our long-term viability.

Start-ups are Time Machines for Big Companies [and open source is a worm hole]

My time at Dell (ended 10/2014) forced me to correct one of the most common misconceptions I hear about big companies – that they cannot innovate. My surprise at Dell was not the lack of innovation, but it’s overabundance. Having talked to many colleagues at big companies, I find the same pattern is everywhere. It’s not that these companies lack amazing and creative ideas, it’s that they have so many that it’s impossible for them to filter and promote them.

Innovation at a big company is like a nest of baby chicks all fighting for a worm but the parent bird can’t decide which chicks to feed.

In order for an idea to win at a big company, it generally has to shout so loudly and promise so extravagantly that it’s setup to fail right out of incubation. Consequently, great ideas are either never launched or killed in adolescence. Of course YMMV, but I’ve seen this pattern repeated throughout the tech industry.

TeamTimeCar.com-BTTF DeLorean Time Machine-OtoGodfrey.com-JMortonPhoto.com-07.jpg

“TeamTimeCar.com-BTTF DeLorean Time Machine-OtoGodfrey.com-JMortonPhoto.com-07” by Terabass. Licensed under CC BY-SA 4.0 via Wikimedia Commons

What big companies really need is a time machine. That way, they can retroactively pick the right innovation and nurture it into a product that immediately benefits from their customer base, support infrastructure and market presence.

Money is a time machine.

With enough money, they can go backwards in time and unwind the decision to not invest in that innovative idea or team. It’s called purchasing a company. Sure, there’s a significant cash premium but that’s easier than stealing more plutonium for your DeLorean. In my experience, it’s behaviorally consistent for companies to act quickly on large outlays for retroactively correct decisions while being unwilling to deal with the political and long-term planning aspects of incubation.

I’ve come to embrace this cycle of innovation with an interesting twist: the growth of open source business models enables a new degree of cross innovation between start-ups and big companies. With open source, corporate locked innovators can exercise their ideas with start-ups and start-ups can leverage the talent and financial depth of big companies.

That’s like creating temporal worm holes in the venture-time continuum. Now that sounds like a topic for a future post… thoughts?