Help OpenStack build more Open Infrastructure communities

Posted on January 7, 2018 by Rob H

Note: OpenStack voting is limited to community members – if you registered by the deadline, you will receive your unique ballot by email. You have 8 votes to distribute as you see fit.

I believe open infrastructure software is essential for our IT future.

Open source has been a critical platform for innovation and creating commercial value for our entire industry; however, we have to deliberately foster communities for open source activities that connect creators, users and sponsors. OpenStack has built exactly that for people interested in infrastructure and that is why I am excited to run for the Foundation Board again.

OpenStack is at a critical juncture in transitioning from a code focus to a community focus.

We must allow the OpenStack code to consolidate around a simple mission while the community explores adjacent spaces. It will be a confusing and challenging transition because we’ll have to create new spaces that leave part of the code behind – what we’d call the Innovator’s Dilemma inside of a single company. And, I don’t think OpenStack has a lot of time to figure this out.

That change requires both strong and collaborative leadership by people who know the community but are not too immersed in the code.

I am seeking community support for my return to the OpenStack Foundation Board. In the two years since I was on the board, I’ve worked in the Kubernetes community to support operators. While on the board, I fought hard to deliver testable interoperability (DefCore) and against expanding the project focus (Big Tent). As a start-up and open source founder, I bring a critical commercial balance to a community that is too easily dominated by large vendor interests.

Re-elected or not, I’m a committed member of the OpenStack community who is enthusiastically supporting the new initiatives by the Foundation. I believe strongly that our industry needs to sponsor and support open infrastructure. I also believe that dominate place for OpenStack IaaS code has changed and we also need to focus those efforts to be highly collaborative.

OpenStack cannot keep starting with “use our code” – we have to start with “let’s understand the challenges.” That’s how we’ll keep building an strong open infrastructure community.

If these ideas resonate with you, then please consider supporting me for the OpenStack board. If they don’t, please vote anyway! There are great candidates on the ballot again and voting supports the community.

Deep Thinking & Tech + Great Guests – L8ist Sh9y podcast relaunched

Posted on October 17, 2017 by Rob H

I love great conversations about technology – especially ones where the answer is not very neatly settled into winners and losers (which is ALL of them in IT). I’m excited that RackN has (re)launched the L8ist Sh9y (aka Latest Shiny) podcast around this exact theme.

Please check out the deep and thoughtful discussion I just had with Mark Thiele (notes) of Aperca where we covered Mark’s thought on why public cloud will be under 20% of IT and culture issues head on.

Spoiler: we have David Linthicum coming next, SO SUBSCRIBE.

I’ve been a guest on some great podcasts (Cloudcast, gcOnDemand, Datanauts, IBM Dojo, HPE, Foodfight) and have deep respect for critical work they do in industry.

We feel there’s still room for deep discussions specifically around automated IT Operations in cloud, data center and edge; consequently, we’re branching out to start including deep interviews in addition to our initial stable of IT Ops deep technical topics like Terraform, Edge Computing, GartnerSYM review, Kubernetes and, of course, our own Digital Rebar.

Soundcloud Subscription Information

(re)Finding an Open Infrastructure Plan: Bridging OpenStack & Kubernetes

Posted on June 1, 2017 by Rob H

TL;DR: infrastructure operations is hard and we need to do a lot more to make these systems widely accessible, easy to sustain and lower risk. We’re discussing these topics on twitter…please join in. Themes include “do we really have consensus and will to act” and “already a solved problem” and “this hurts OpenStack in the end.”

pexels-photo-229949 I am always looking for ways to explain (and solve!) the challenges that we face in IT operations and open infrastructure. I’ve been writing a lot about my concern that data center automation is not keeping pace and causing technical debt. That concern led to my recent SRE blogging for RackN.

It’s essential to solve these problems in an open way so that we can work together as a community of operators.

It feels like developers are quick to rally around open platforms and tools while operators tend to be tightly coupled to vendor solutions because operational work is tightly coupled to infrastructure. From that perspective, I’m been very involved in OpenStack and Kubernetes open source infrastructure platforms because I believe the create communities where we can work together.

This week, I posted connected items on VMblog and RackN that layout a position where we bring together these communities.

Of course, I do have a vested interest here. Our open underlay automation platform, Digital Rebar, was designed to address a missing layer of physical and hybrid automation under both of these projects. We want to help accelerate these technologies by helping deliver shared best practices via software. The stack is additive – let’s build it together.

I’m very interested in hearing from you about these ideas here or in the context of the individual posts. Thanks!

How scared do we need to be for Ops collaboration & investment?

Posted on March 8, 2017 by Rob H

Note: Yesterday RackN posted Are you impatient enough to be an SRE? and then the CIA wikileaks news hit… perhaps the right question is “Are you scared enough to automate deeply yet?”

Cia leak (1) As an industry, the CIA hacking release yesterday should be driving discussions about how to make our IT infrastructure more robust and fluid. It is not simply enough to harden because both the attack and the platforms are evolving to quickly.

We must be delivering solutions with continuous delivery and immutability assumptions baked in.

A more fluid IT that assumes constant updates and rebuilding from sources (immutable) is not just a security posture but a proven business benefit. For me, that means actually building from the hardware up where we patch and scrub systems regularly to shorten the half-life of all attach surfaces. It also means enabling existing security built into our systems that are generally ignored because of configuration complexity. These are hard but solvable automation challenges.

The problem is too big to fix individually: we need to collaborate in the open.

I’ve been really thinking deeply about how we accelerate SRE and DevOps collaboration across organizations and in open communities. The lack of common infrastructure foundations costs companies significant overhead and speed as teams across the globe reimplement automation in divergent ways. It also drags down software platforms that must adapt to each data center as a unique snowflake.

That’s why hybrid automation within AND between companies is an imperative. It enables collaboration.

Making automation portable able to handle the differences between infrastructure and environments is harder; however, it also enables sharing and reuse that creates allows us to improve collectively instead of individually.

That’s been a vision driving us at RackN with the open hybrid Digital Rebar project. Curious? Here’s RackN post that inspired this one:

From RackN’s Are you impatient enough to be an SRE?

“Like the hardware that runs it, the foundation automation layer must be commoditized. That means that Operators should be able to buy infrastructure (physical and cloud) from any vendor and run it in a consistent way. Instead of days or weeks to get infrastructure running, it should take hours and be fully automated from power-on. We should be able to rehearse on cloud and transfer that automation directly to (and from) physical without modification. That practice and pace should be the norm instead of the exception.”

Evolution or Rebellion? The rise of Site Reliability Engineers (SRE)

Posted on December 29, 2016 by Rob H

What is a Google SRE? Charity Majors gave a great overview on Datanauts #65, Susan Fowler from Uber talks about “no ops” tensions and Patrick Hill from Atlassian wrote up a good review too. This is not new: Ben Treynor defined it back in 2014.

DevOps is under attack.

Well, not DevOps exactly but the common misconception that DevOps is about Developers doing Ops (it’s really about lean process, system thinking, and positive culture). It turns out the Ops is hard and, as I recently discussed with John Furrier, developers really really don’t want be that focused on infrastructure.

In fact, I see containers and serverless as a “developers won’t waste time on ops revolt.” (I discuss this more in my 2016 retrospective).

The tension between Ops and Dev goes way back and has been a source of confusion for me and my RackN co-founders. We believe we are developers, except that we spend our whole time focused on writing code for operations. With the rise of Site Reliability Engineers (SRE) as a job classification, our type of black swan engineer is being embraced as a critical skill. It’s recognized as the only way to stay ahead of our ravenous appetite for computing infrastructure.

I’ve been writing about Site Reliability Engineering (SRE) tasks for nearly 5 years under a lot of different names such as DevOps, Ready State, Open Operations and Underlay Operations. SRE is a term popularized by Google (there’s a book!) for the operators who build and automate their infrastructure. Their role is not administration, it is redefining how infrastructure is used and managed within Google.

Using infrastructure effectively is a competitive advantage for Google and their SREs carry tremendous authority and respect for executing on that mission.

Managers Meanwhile, we’re in the midst of an Enterprise revolt against running infrastructure. Companies, for very good reasons, are shutting down internal IT efforts in favor of using outsourced infrastructure. Operations has simply not been able to complete with the capability, flexibility and breadth of infrastructure services offered by Amazon.

SRE is about operational excellence and we keep up with the increasingly rapid pace of IT. It’s a recognition that we cannot scale people quickly as we add infrastructure. And, critically, it is not infrastructure specific.

Over the next year, I’ll continue to dig deeply into the skills, tools and processes around operations. I think that SRE may be the right banner for these thoughts and I’d like to hear your thoughts about that.

MORE? Here’s the next post in the series about Spiraling Ops Debt. Or Skip to Podcasts with Eric Wright and Stephen Spector.

Can we control Hype & Over-Vendoring?

Posted on December 12, 2016 by Rob H

Q: Is over-vendoring when you’ve had to much to drink?
A: Yes, too much Kool Aid.

There’s a lot of information here – skip to the bottom if you want to see my recommendation.

Last week on TheNewStack, I offered eight ways to keep Kubernetes on the right track (abridged list here) and felt that item #6 needed more explanation and some concrete solutions.

DO: Focus on a Tight Core
DO: Build a Diverse Community
DO: Multi-cloud and Hybrid
DO: Be Humble and Honest
AVOID: “The One Ring” Universal Solution Hubris
AVOID: Over-Vendoring (discussed here)
AVOID: Coupling Installers, Brokers and Providers to the core
AVOID: Fast Release Cycles without LTS Releases

What is Over-Vendoring? It’s when vendors’ drive their companies’ brands ahead of the health of the project. Generally by driving an aggressive hype cycle where vendors are trying to jump on the hype bandwagon.

Hype can be very dangerous for projects (David Cassel’s TNS article) because it is easy to bypass the user needs and boring scale/stabilization processes to focus on vendor differentiation. Unfortunately, common use-cases do not drive differentiation and are invisible when it comes to company marketing budgets. That boring common core has the effect creating tragedy of the commons which undermines collaboration on shared code bases.

The solution is to aggressively keep the project core small so that vendors have specific and limited areas of coopetition.

A small core means we do not compel collaboration in many areas of project. This drives competition and diversity that can be confusing. The temptation to endorse or nominate companion projects is risky due to the hype cycle. Endorsements can create a bias that actually hurts innovation because early or loud vendors do not generally create the best long term approaches. I’ve heard this described as “people doing the real work don’t necessarily have time to brag about it.”

Keeping a small core mantra drives a healthy plug-in model where vendors can differentiate. It also ensures that projects can succeed with a bounded set of core contributors and support infrastructure. That means that we should not measure success by commits, committers or lines of code because these will drop as projects successfully modularize. My recommendation for a key success metric is to the ratio of committers to ecosystem members and users.

Tracking improving ratio of core to ecosystem shows that improving efficiency of investment. That’s a better sign of health than project growth.

It’s important to note that there is also a serious risk of under-vendoring too!

We must recognize and support vendors in open source communities because they sustain the project via direct contributions and bringing users. For a healthy ecosystem, we need to ensure that vendors can fairly profit. That means they must be able to use their brand in combination with the project’s brand. Apache Project is the anti-pattern because they have very strict “no vendor” trademark marketing guidelines that can strand projects without good corporate support.

I’ve come to believe that it’s important to allow vendors to market open source projects brands; however, they also need to have some limits on how they position the project.

How should this co-branding work? My thinking is that vendor claims about a project should be managed in a consistent and common way. Since we’re keeping the project core small, that should help limit the scope of the claims. Vendors that want to make ecosystem claims should be given clear spaces for marketing their own brand in participation with the project brand.

I don’t pretend that this is easy! Vendor marketing is planned quarters ahead of when open source projects are ready for them: that’s part of what feeds the hype cycle. That means that projects will be saying no to some free marketing from their ecosystem. Ideally, we’re saying yes to the right parts at the same time.

Ultimately, hype control means saying no to free marketing. For an open source project, that’s a hard but essential decision.

Infrastructure Masons is building a community around data center practice

Posted on November 15, 2016 by Rob H

IT is subject to seismic shifts right now. Here’s how we cope together.

For a long time, I’ve advocated for open operations (“OpenOps”) as a way to share best practices about running data centers. I’ve worked hard in OpenStack and, recently, Kubernetes communities to have operators collaborate around common architectures and automation tools. I believe the first step in these efforts starts with forming a community forum.

I’m very excited to have the RackN team and technology be part of the newly formed Infrastructure Masons effort because we are taking this exact community first approach.

Here’s how Dean Nelson, IM organizer and head of Uber Compute, describes the initiative:

An Infrastructure Mason Partner is a professional who develop products, build or support infrastructure projects, or operate infrastructure on behalf of end users. Like their end users peers, they are dedicated to the advancement of the Industry, development of their fellow masons, and empowering business and personal use of the infrastructure to better the economy, the environment, and society.

We’re in the midst of tremendous movement in IT infrastructure. The change to highly automated and scale-out design was enabled by cloud but is not cloud specific. This requirement is reshaping how IT is practiced at the most fundamental levels.

We (IT Ops) are feeling amazing pressure on operations and operators to accelerate workflow processes and innovate around very complex challenges.

Open operations loses if we respond by creating thousands of isolated silos or moving everything to a vendor specific island like AWS. The right answer is to fund ways to share practices and tooling that is tolerant of real operational complexity and the legitimate needs for heterogeneity.

Interested in more? Get involved with the group! I’ll be sharing more details here too.

Why Fork Docker? Complexity Wack-a-Mole and Commercial Open Source

Posted on August 31, 2016 by Rob H

Update 12/14/16: Docker announced that they would create a container engine only project, ContinainerD, to decouple the engine from management layers above. Hopefully this addresses this issues outlined in the post below.

Monday, The New Stack broke news about a possible fork of the Docker Engine and prominently quoted me saying “Docker consistently breaks backend compatibility.” The technical instability alone is not what’s prompting industry leaders like Google, Red Hat and Huawei to take drastic and potentially risky community action in a central project.

So what’s driving a fork? It’s the intersection of Cash, Complexity and Community.

hamster In fact, I’d warned about this risk over a year ago: Docker is both a core infrastucture technology (the docker container runner, aka Docker Engine) and a commercial company that manages the Docker brand. The community formed a standard, runC, to try and standardize; however, Docker continues to deviate from (or innovate faster) that base.

It’s important for me to note that we use Docker tools and technologies heavily. So far, I’ve been a long-time advocate and user of Docker’s innovative technology. As such, we’ve also had to ride the rapid release roller coaster.

Let’s look at what’s going on here in three key areas:

1. Cash

The expected monetization of containers is the multi-system orchestration and support infrastructure. Since many companies look to containers as leading the disruptive next innovation wave, the idea that Docker is holding part of their plans hostage is simply unacceptable.

So far, the open source Docker Engine has been simply included without payment into these products. That changed in version 1.12 when Docker co-mingled their competitive Swarm product into the Docker Engine. That effectively forces these other parties to advocate and distribute their competitors product.

2. Complexity

When Docker added cool Swarm Orchestration features into the v1.12 runtime, it added a lot of complexity too. That may be simple from a “how many things do I have to download and type” perspective; however, that single unit is now dragging around a lot more code.

In one of the recent comments about this issue, Bob Wise bemoaned the need for infrastructure to be boring. Even as we look to complex orchestration like Swarm, Kubernetes, Mesos, Rancher and others to perform application automation magic, we also need to reduce complexity in our infrastructure layers.

Along those lines, operators want key abstractions like containers to be as simple and focused as possible. We’ve seen similar paths for virtualization runtimes like KVM, Xen and VMware that focus on delivering a very narrow band of functionality very well. There is a lot of pressure from people building with containers to have a similar experience from the container runtime.

This approach both helps operators manage infrastructure and creates a healthy ecosystem of companies that leverage the runtimes.

Note: My company, RackN, believes strongly in this need and it’s a core part of our composable approach to automation with Digital Rebar.

3. Community

Multi-vendor open source is a very challenging and specialized type of community. In these communities, most of the contributors are paid by companies with a vested (not necessarily transparent) interest in the project components. If the participants of the community feel that they are not being supported by the leadership then they are likely to revolt.

Ultimately, the primary difference between Docker and a fork of Docker is the brand and the community. If there companies paying the contributors have the will then it’s possible to move a whole community. It’s not cheap, but it’s possible.

Developers vs Operators

One overlooked aspect of this discussion is the apparent lock that Docker enjoys on the container developer community. The three Cs above really focus on the people with budgets (the operators) over the developers. For a fork to succeed, there needs to be a non-Docker set of tooling that feeds the platform pipeline with portable application packages.

In Conclusion…

The world continues to get more and more heterogeneous. We already had multiple container runtimes before Docker and the idea of a new one really is not that crazy right now. We’ve already got an explosion of container orchestration and this is a reflection of that.

My advice? Worry less about the container format for now and focus on automation and abstractions.

OpenStack Interop, Container Security, Install & Open Source Posts

Posted on June 10, 2016 by Rob H

In case you missed it, I posted A LOT of content this week on other sites covering topics for OpenStack Interop, Container Security, Anti-Universal Installers and Monetizing Open Source. Here are link-bait titles & blurbs from each post so you can decide which topics pique your interest.

Thirteen Ways Containers are More Secure than Virtual Machines on TheNewStack.com

Last year, conventional wisdom had it that containers were much less secure than virtual machines (VMs)! Since containers have such thin separating walls; it was easy to paint these back door risks with a broad brush. Here’s a reality check: Front door attacks and unpatched vulnerabilities are much more likely than these backdoor hacks.

It’s Time to Slay the Universal Installer Unicorn on DevOps.com

While many people want a universal “easy button installer,” they also want it to work on their unique snowflake of infrastructures, tools, networks and operating systems. Because there is so much needful variation and change, it is better to give up on open source projects trying to own an installer and instead focus on making their required components more resilient and portable.

King of the hill? Discussing practical OpenStack interoperability on OpenStack SuperUser

Can OpenStack take the crown as cloud king? In our increasingly hybrid infrastructure environment, the path to the top means making it easier to user to defect from the current leaders (Amazon AWS; VMware) instead of asking them to blaze new trails. Here are my notes from a recent discussion about that exact topic…

Have OpenSource, Will Profit?! 5 thoughts from Battery Ventures OSS event on RobHirschfeld.com

As “open source eats software” the profit imperative becomes ever more important to figure out. We have to find ways to fund this development or acknowledge that software will simply become waste IP and largess from mega brands. The later outcome is not particularly appealing or innovative.

Have OpenSource, Will Profit?! 5 thoughts from Battery Ventures OSS event

Posted on June 7, 2016 by Rob H

Last week, Battery Ventures hosted a “Venture and Open Source Software (OSS)” event that crystallized several key points around OSS business models. The speakers (really, check out the list!) were deeply experienced with thoughtful points that reflected a balanced perspective. In those post, I’m trying to synthesize rather than give attribution. Please review the vibrant #BVOSS twitter stream for specific quotes and pictures.

**There is no valuation difference between for OSS and Proprietary. OSS is a business model, you are running a software company.**

It’s hard to monetize a company with OSS. While there are limited IPO benchmarks; the model is clearly being adopted deeply. It’s also not clear if it’s better to be the primary driver for a project or to ride a larger effort.

Should companies avoid OSS? It’s hard to monetize any idea – OSS is not a deciding factor. At the end of the day, it seemed pretty clear that open source strategies are simply a new components of building software companies.

Big companies are willing to fund OSS projects (but late in the cycle).

Companies are recognized that they have a role to play in open source by funding projects. They do this to ensure the project is sustained, maintain influence and ensure road map direction. It’s often via support contracts.

This influence appears to be late in the OSS cycle. It’s not clear how companies invest in early phases of development that are critical to start-up success. In my daily business, I’d love to see companies set aside low-barrier $20-100k exploration funds to seed PoCs so that early stage projects could afford to hand-hold enterprise adopters.

I can think of many examples where the project is effectively spun out of other corporate or consulting efforts (including RackN core OSS IP, Digital Rebar). IMHO, it’s less common to have an organic OSS company.

OSS companies need to hold something back to monetize.

This point was made emphatically multiple times. Customers do not fund OSS projects out of good will, some hook is needed to entice payment. It does not take much (Marten Mickos called it a pinch of salt) but having something was considered important. Generally these are extras that are needed for advanced or scale users.

While this approach draws negative comments, the “open core” approach seemed to be the expectation from the room. An all-things-open approach would result in trying to sell consulting services as the primary revenue model – this is generally not considered an attractive venture strategy for start-ups.

OSS value to the business increases with ubiquity/popularity of the project.

This may be obvious; however, an effective OSS model needs to have community validation. The concept is that there is some conversion ratio, so having users makes up for a poor ratio. Overall, the room assumed that community was a good thing.

I can think of cases where having a HUGE user base did not immediately translate into monetization. In some cases (OpenStack, Docker), it translated into a large ecosystem and brisk competition.

As a Service (Hosted OSS) may be critical monetization path.

It was recognized that service providers (Amazon was the goat here) are doing a very robust job monetizing OSS. For example, users pay significant premiums for MySQL hosted by RDS while they are much less willing to pay Oracle when they run it themselves. It’s very clear that managed service models rely on cheap software.

It’s equally clear that users are willing to pay for services when they are reluctant to buy licenses. The room felt that OSS companies should seriously evaluate this path to revenue and adoption.

Overall, it was a fantastic summit that left me, as a OSS start-up entrepreneur, thinking carefully about how we are shaping businesses to create and exploit OSS. Getting these models right is essential to maintaining our pace of innovation.

Rob Hirschfeld

On Computing, Containers, Cloud & Tech Culture

Category Archives: Open Infrastructure