Operators, they don’t want to swim Upstream

Posted on November 12, 2015 by Rob H

Nov 10, Palo Alto Operators Dinner

Last Tuesday, I had the honor of joining an OpenStack scale operators dinner. Foundation executives, Jonathan Bryce and Lauren Sell, were also on the guest list so talk naturally turned to “how can OpenStack better support operators.” Notably, the session was distinctly not OpenStack bashing.

The conversation was positive, enthusiastic and productive, but one thing was clear: the OpenStack default “we’ll fix it in the upstream” answer does not work for this group of operators.

What is upstreaming? A sans nuance answer is that OpenStack drives fixes and changes in the next community release (longer description). The project and community have a tremendous upstream imperative that pervades the culture so deeply that we take it for granted. Have an issue with OpenStack? Submit a patch! Is there any other alternative?

Upstreaming [to trunk] makes perfect sense considering the project vendor structure and governance; however, it is a very frustrating experience for operators. OpenStack does have robust processes to backport fixes and sustain past releases and documentation; yet, the feeling at the table was that they are not sufficiently operator focused.

Operators want fast, incremental and pragmatic corrections to the code and docs they are deploying (which is often two releases back). They want it within the community, not from individual vendors.

There are great reasons for focusing on upstream trunk. It encourages vendors to collaborate and makes it much easier to add and expand the capabilities of the project. Allowing independent activity on past releases creates a forward integration mess and could make upgrades even harder. It will create divergence on APIs and implementation choices.

The risk of having a stable, independently sustained release is that operators have less reason to adopt the latest shiny release. And that is EXACTLY what they are asking for.

Upstreaming is a core value to OpenStack and essential to our collaborative success; however, we need to consider that it is not the right answer to all questions. Discussions at that dinner reinforced that pushing everything to latest trunk creates a significant barrier for OpenStack operators and users.

What are your experiences? Is there a way to balance upstreaming with forking? How can we better serve operators?

The Upstream Imperative: paving the way for content creators is required for platform success

Posted on August 6, 2014 by Rob H

Since content is king, platform companies (like Google, Microsoft, Twitter, Facebook and Amazon) win by attracting developers to build on their services. Open source tooling and frameworks are the critical interfaces for these adopters; consequently, they must invest in building communities around those platforms even if it means open sourcing previously internal only tools.

This post expands on one of my OSCON observations: companies who write lots of code have discovered an imperative to upstream their internal projects. For background, review my thoughts about open source and supply chain management.

Huh? What is an “upstream imperative?” It sounds like what salmon do during spawning then read the post-script!

Historically, companies with a lot of internal development tools had no inventive to open those projects. In fact, the “collaboration tax” of open source discouraged companies from sharing code for essential operations. Historically, open source was considered less featured and slower than commercial or internal projects; however, this perception has been totally shattered. So companies are faced with a balance between the overhead of supporting external needs (aka collaboration) and the innovation those users bring into the effort.

Until recently, this balance usually tipped towards opening a project but under-investing in the community to keep the collaboration costs low. The change I saw at OSCON is that companies understand that making open projects successful bring communities closer to their products and services.

That’s a huge boon to the overall technology community.

Being able to leverage and extend tools that have been proven by these internal teams strengthens and accelerates everyone. These communities act as free laboratories that breed new platforms and build deep relationships with critical influencers. The upstream savvy companies see returns from both innovation around their tools and more content that’s well matched to their platforms.

Oh, and companies that fail to upstream will find it increasingly hard to attract critical mind share. Thinking the alternatives gives us a Windows into how open source impacts past incumbents.

That leads to a future post about how XaaS dog fooding and “pure-play” aaS projects like OpenStack and CloudFoundry.

Post Script about Upstreaming:

Continue reading →

4 item OSCON report: no buzz winner, OpenStack is DownStack?, Free vs Open & the upstream imperative

Posted on August 4, 2014 by Rob H

Now that my PDX Trimet pass expired, it’s time to reflect on OSCON 2014. Unfortunately, I did not ride my unicorn home on a rainbow; this year’s event seemed to be about raising red flags.

My four key observations:

No superstar. Past OSCONs had at least one buzzy community super star. 2013 was Docker and 2011 was OpenStack. This was not just my hallway track perception, I asked around about this specifically. There was no buzz winner in 2014.
People were down on OpenStack (“DownStack”). Yes, we did have a dedicated “Open Cloud Day” event but there was something missing. OpenSack did not sponsor and there were no major parties or releases (compared to previous years) and little OpenStack buzz. Many people I talked to were worried about the direction of the community, fragmentation of the project and operational readiness. I would be more concerned about “DownStack” except that no open infrastructure was a superstar either (e.g.: Mesos, Kubernetes and CoreOS). Perhaps, OSCON is simply not a good venue open infrastructure projects compared to GlueCon or Velocity? Considering the rapid raise of container-friendly OpenStack alternatives; I think the answer may be that the battle lines for open infrastructure are being redrawn.
Free vs. Open. Perhaps my perspective is more nuanced now (many in open source communities don’t distinguish between Free and Open source) but there’s a real tension between Free (do what you want) and Open (shared but governed) source. Now that open source is a commercial darling, there is a lot of grumbling in the Free community about corporate influence and heavy handedness. I suspect this will get louder as companies try to find ways to maintain control of their projects.
Corporate upstreaming becomes Imperative. There’s an accelerating upstreaming trend for companies that write lots of code to support their primary business (Google is a primary example) to ensure that code becomes used outside their company. They open their code and start efforts to ensure its adoption. This requires a dedicated post to really explain.

There’s a clear theme here: Open source is going mainstream corporate.

We’ve been building amazing software in the open that create real value for companies. Much of that value has been created organically by well-intentioned individuals; unfortunately, that model will not scale with the arrival for corporate interests.

Open source is thriving not dying: these companies value the transparency, collaboration and innovation of open development. Instead, open source is transforming to fit within corporate investment and governance needs. It’s our job to help with that metamorphosis.

What does “enable upstream recipes” mean? Not just fishing for community goodness!

Posted on July 5, 2012 by Rob H

One of the major Crowbar 2.0 design targets is to allow you to “upstream” operations scripts more easily. “Upstream code” means that parts of Crowbar’s source code could be maintained in other open source repositories. This is beyond a simple dependency (like Rails, Curl, Java or Apache): Upstreaming allows Crowbar can use code managed in the other open source repositories for more general application. This is important because Crowbar users can leverage DevOps logic that is more broadly targeted than just Crowbar. Even more importantly, upstreaming means that we can contribute and take advantage of community efforts to improve the upstream source.

Specifically, Crowbar maintains a set of OpenStack cookbooks that make up the core of our OpenStack deployment. These scripts have been widely cloned (not forked) and deCrowbarized for other deployments. Unfortunately, that means that we do not benefit from downstream improvements and the cloners cannot easily track our updates. This happened because Crowbar was not considered a valid upstream OpenStack repository because our deployment scripts required Crowbar. The consequence of this cloning is that incompatible OpenStack recipes have propagated like cracks in a windshield.

While there are concrete benefits to upstreaming, there are risks too. We have to evaluate if the upstream code has been adequately tested, operates effectively, implements best practices and leverages Crowbar capabilities. I believe strongly that untested deployment code is worse than useless; consequently, the Dell Crowbar team provides significant value by validating that our deployments work as an integrated system. Even more importantly, we will not upstream from unmoderated sources where changes are accepted without regard for downstream impacts. There is a significant amount of trust required for upstreaming to work.

If upstreaming is so good, why did we not start out with upstream code? It was simply not an option at the time – Crowbar was the first (and is still!) most complete set of DevOps deployment scripts for OpenStack in a public repository.

By design, Crowbar 1.0 was tightly coupled to Opscode Chef and required users to inject Crowbar dependencies into their Chef Recipes. This approach allowed us to more quickly integrate capabilities between recipes and with nascent Crowbar features. Our top design requirement was that our deployment was tightly integrated between hardware, networking, operating system, operations infrastructure and the application. Figuring out the correct place to separate concerns was impractical; consequently, we injected dependencies into our Chef code.

We have reached a point with Crowbar development that we can correctly decouple Crowbar and Chef.

The benefits to upstreaming go far beyond enabling more collaboration on OpenStack deployments. These same changes make it easier for Crowbar to leverage community deployment scripts without one-way modifications. If you have a working Chef Recipe then making it work with Crowbar will no longer require changes that break it outside of Crowbar; therefore, you can leverage Crowbar capabilities without losing community input and without being locked into Crowbar.

Rob Hirschfeld

On Computing, Containers, Cloud & Tech Culture

Tag Archives: Upstream