By Dan Choquette

Is DevOps at scale like a major city’s subway system? Both require strict processes and operational excellence to move a lot of different parts at once. How else?

If you had the pleasure of riding the Tokyo Metro, you might agree that it’s an interesting – and confusing experience (especially if you need to change lines!) All totaled, there are 9 lines, roughly 180+ stations with a daily ridership of almost 7 million people!

A few days ago, I had a conversation with a potential user deploying Kubernetes with Contrail networking on Google Cloud repeatedly in a build/test/dev scenario. The other conversation was around the need to provision thousands of x86 bare metal servers once to twice a week with different configurations and networking with the need to ultimately control their metal as they would a cloud instance in AWS. Cool stuff!

Since we here at RackN believe Hybrid DevOps is a MUST for Hybrid IT (after all, we are a start-up and have bet our very lives on such a thing so we REALLY believe it!) I thought about how Hybrid DevOps compares to the Tokyo Metro (earlier that day I read about Tokyo in the news and the mind wandered). In my attempt to draw the parallel, below is an SDLC DevOps framework that you have seen 233 times before in blogs like this one.

In terms of process, I’m sure you can notice how similar it is to the Metro, right?

<crickets>

<more crickets>

When both operate as they should, they are the epitome of automation, control, repeat-ability and reliability. In disciplined, automated at-scale DevOps environments it does have some similarity to the Ginza or Tozai line. You have different people (think apps) of all walks of life boarding a train needing to get somewhere and need to follow steps in a process (maybe the “Pusher” is the scrum or DevOps governance tool but we’ll leave that determination for the end). However, as I compare it to Hybrid DevOps, the Tokyo Metro is not hybrid-tolerant. With subways, if a new subway car is added, tracks are changed, or a new station is added instantaneously to better handle congestion everything stops or turns into a logistical disaster. In addition, there is no way of testing how it will all flow before hand. There will be operational glitches and millions of angry customers will not reach their destination in a timely fashion- or at all.

The same is metaphorically true for Hybrid DevOps in Hybrid IT. In theory, the Hybrid DevOps pipeline includes build/test/dev and continuous integration/deployment for all platforms, business models, governance models, security and software stacks in which are dependent with the physical/IaaS/container underlay. Developers and operators need to test against multiple platforms (cloud, VM and metal) and in order to realize value, assimilate into production rapidly while at the same time frequently adjusting to changes of all kinds. They also require the ability to layer multiple technologies and security policies into an operational pipeline which in turn has hundreds of moving parts which require precise configuration settings to be made in a sequenced, orchestrated manner.

At RackN, in order to continuously test, integrate, deploy and manage complex technologies in a Hybrid IT scenario is critical to a successful adoption in production. The most optimal way to accomplish that is to have in place a central platform than can govern Hybrid DevOps at scale that can automate, orchestrate and compose allthe necessary configurations and components in a sequenced fashion. Without one, hap-hazard assembly and lack of governance erodes the overall process and leads to failure. Just like the “Pusher” on the platform, without governance both the Tokyo Metro and a Hybrid DevOps model at scale being used for a Hybrid IT use case leads to massive delays, dissatisfied customers and chaos.

By Rob Hirschfeld

Lately, I’ve been talking about the general concept of hybrid DevOps adding composability, orchestration and services to traditional configuration. It’s time add a concrete example because the RackN team is deliving it with Digital Rebar and Kubernetes.

So far, we enabled a single open platform to install over 18 different configurations of Kubernetes simply by changing command line flags [videos below].

By taking advantage of the Digital Rebar underlay abstractions and orchestration, we are able to use open community installation playbooks for a wide range of configurations.

So far, we’re testing against:

• Three different clouds (AWS, Google and Packet) not including the option of using bare metal.
• Two different operating systems (Ubuntu and Centos)
• Three different software defined networking systems (Flannel, Calico and OpenContrail)

Those 18 are just the tip of the iceberg that we are actively testing. The actual matrix is much deeper.

BUT THAT’S AN EXPLODING TEST MATRIX!?! No. It’s not.

The composable architecture of Digital Rebar means that all of these variations are isolated. We are not creating 18 distinct variations; instead, the system chains options together and abstracts the differences between steps.

That means that we could add different logging options, test sequences or configuration choices into the deployment with minimal coupling of previous steps. This enables operator choice and vendor injection in a way to allows collaboration around common components. By design, we’ve eliminated fragile installation monoliths.

All it takes is a Packet, AWS or Google account to try this out for yourself!

Lately, I’ve been talking about the general concept of hybrid DevOps adding composability, orchestration and services to traditional configuration.  It’s time add a concrete example: the RackN team using Hybrid DevOps to deliver Kubernetes via Digital Rebar using community tools and scripts.

So far, we provision over 18 different configurations of Kubernetes simply by changing command line flags [videos below].  That does not include optional post install steps like tests and applications.

By taking advantage of the Digital Rebar underlay abstractions and orchestration, we are able to use open community installation playbooks for a wide range of configurations.

So far, we’re testing against:

• Three different clouds (AWS, Google and Packet.net) not including the option of using bare metal.
• Two different operating systems (Ubuntu and Centos)
• Three different software defined networking systems (Flannel, Calico and OpenContrail)

Those 18 are just the tip of the iceberg that we are actively testing.  The actual matrix is much deeper.

BUT THAT’S AN EXPLODING TEST MATRIX!?!  No.  It’s not.

The composable architecture of Digital Rebar means that all of these variations are isolated.  We are not creating 18 distinct variations; instead, the system chains options together and abstracts the differences between steps.

That means that we could add different logging options, test sequences or configuration choices into the deployment with minimal coupling of previous steps.  This enables operator choice and vendor injection in a way to allows collaboration around common components.  By design, we’ve eliminated fragile installation monoliths.

All it takes is a Packet, AWS or Google account to try this out for yourself!

Using the CLI script to install Kubernetes:

Deep Dive into adding OpenContrail SDN:

At RackN, we’re huge fans of Docker.  We’ve been using the engine for years (since v0.8!) and you can read about our lessions from when we rearchitected around Docker Compose.  Now we’ve built  “one-click” hybrid cluster automation for Kubernetes, Docker Swarm and others.

However, I’m concerned that Docker installs reveal a lack of cluster operation focus.  These platforms are evolving very rapidly exposing users to both breaking upgrades and security risks.  This drives a requirement for cluster automation.

What are cluster operations?  It is the system level activity of creating an integrated platform that is repeatable, secure, networked and sustainable.  As use of Docker transitions from single node activity into multi-node and hybrid clusters, we need to approach the install and configuration as a system activity.

Cluster configuration requires system activity because there are so many moving pieces and necessary pre-configurations of networking, security, storage and roles.  These choices need to be implemented before the actual cluster software is installed because they drive how the cluster is configured and managed.  They continue to be a major factor as we grow, shrink and ugprade the cluster.

Why don’t people do this already?  Because cluster configuration requires additional setup and planning.  Operators are struggling just to keep up with API changes between quarterly updates.

Our mission is to eliminate the overhead of cluster operations so you can focus on using the Cluster not building it.

The RackN team has been working on deployment of Docker Swarm (and container orchestration more generally) to make sure Cluster Operations and underlay are robust and automated on every platform from cloud to metal.

The video below (and others in my channel) show how we’ve made it “one click” easy to create container clusters in nearly any environment.  While this is an evolving process, we believe that it is critical to start with cluster automation.

Let us show you how we’ve made that both fast and painless.

Last week, I made my second apperance on the Brian Gracely & Aaron Delp’s excellent podcast, The Cloudcast.   A lot has changed since my first appearance in 2011 but we’re still struggling to create consistent operations around these new platforms.  Then it was OpenStack, today it is container orchestration.

I loved this closing comment from Brian, “[the Cloudcast] loves people who are down in the dirt… you are living it and it’s going into the product.”

Total time, 38 minutes

• 02:15: Interview Starts
• 03:15: RackN path to Digital Rebar.  History of team going back to Crowbar
• 06:05: Why we moved to containers for Digital Rebar (blog details here)
• 07:20: The process to transform from monolith to services
• 07:50: As backgrouind, what is Digital Rebar?  Configuration & Services in Sequence.
• 09:30: How/Why to use Consul for services
• 10:30: Why Immutability is Hard  (technial use of the word “cheese”)
• 12:20: Challenge of restarting & state in Microservices
• 14:30: Need for Iterative design process to improve as you learn the pattern.
• 15:10: “If you are not using containers for at least packaging, your are crazy”
• 15:40: We choose not to talk about OpenStack!
• 16:15: Fidelity Gap and cloud portability
• 17:10: Rob does funny voice about idea that with containers “devs don’t have to do ops”
• 18:00: Why adding some overhead for deveopers is a good investment.
• 18:40: Rob throws OpenStack under the bus for Devstack and “it worked in Devstack” mentality
• 21:20: Containers do not solve all problems, in some ways they make things harder (especially on networking)/
• 21:55: “we are about to put a serious hurt on networking management”
• 21:50: Networking configuration is hard to build in a consisent way.   You have to automated it – there is no other choice.
• 24:20: Hybrid Cloud priorities with RackN
• 25:00: We “declared default” on trying to create a mono-cloud and accepted that infrastructure is hybrid.
• 25:40: Openness comes from having multiple providers.  Composable ops allows you to cope with heterogeneous APIs
• 28:55: Businesses want choice and control about infrastructure.  Do not want to deployments to hardcode to platforms or tooling.
• 29:30: “I have not met anyone who is just using one cloud, tool or platform”
• 30:30: Brian asks Rob to pick winners and trends.  “we like to let people pick and choose.”
• 31:00: Container orchestration with networking and storage are going to be huge.
• 31:30: Rob compares Kubernetes, Docker, Mesos, Rancher and Cloudsoft.
• 32:20: The importance of adjacencies.  Things you need to make the core stuff work.
• 34:20: “Watch out for the adjacencies because they will slow you down.”
• 36:10: “We love guests who live in the dirt” and “built the technology that they wanted to get their jobs”

Source: From Start to Scale: learn faster with heterogenous deployments

Why did your mom care about underwear? She wanted you to have good hygiene. What is good Ops hygiene? It’s not as simple as keeping up with the laundry, but the idea is similar. It means that we’re not going to get surprised by something in our environment that we’d taken for granted. It means that we have a fundamental level of control to keep clean. Let’s explore this in context.

I’ve struggled with the term “underlay” for infrastructure of a long time. At RackN, we generally prefer the term “ready state” to describe getting systems prepared for install; however, underlay fits very well when we consider it as the foundation for a more building up a platform like Kubernetes, Docker Swarm, Ceph and OpenStack. Even more than single operator applications, these community built platforms require carefully tuned and configured environments. In my experience, getting the underlay right dramatically reduces installation challenges of the platform.

What goes into a clean underlay? All your infrastructure and most of your configuration.

Just buying servers (or cloud instances) does not make a platform. Cloud underlay is nearly as complex, but let’s assume metal here. To turn nodes into a cluster, you need setup their RAID and BIOS. Generally, you’ll also need to configure out-of-band management IPs and security. Those RAID and BIOS settings specific to the function of each node, so you’d better get that right. Then install the operating system. That will need access keys, IP addresses, names, NTP, DNS and proxy configuration just as a start. Before you connect to the wide, make sure to update to your a local mirror and site specific requirements. Installing Docker or a SDN layer? You may have to patch your kernel. It’s already overwhelming and we have not even gotten to the platform specific details!

Buried in this long sequence of configurations are critical details about your network, storage and environment.

Any mistake here and your install goes off the rails. Imagine that your building a house: it’s very expensive to change the plumbing lines once the foundation is poured. Thankfully, software configuration is not concrete but the costs of dealing with bad setup is just as frustrating.

The underlay is the foundation of your install. It needs to be automated and robust.

The challenge compounds once an installation is already in progress because adding the application changes the underlay. When (not if) you make a deploy mistake, you’ll have to either reset the environment or make your deployment idempotent (meaning, able to run the same script multiple times safely). Really, you need to do both.

Why do you need both fast resets and component idempotency? They each help you troubleshoot issues but in different ways. Fast resets ensure that you understand the environment your application requires. Post install tweaks can mask systemic problems that will only be exposed under load. Idempotent action allows you to quickly iterate over individual steps to optimize and isolate components. Together they create resilient automation and good hygiene.

In my experience, the best deployments involved a non-recoverable/destructive performance test followed by a completely fresh install to reset the environment. The Ops equivalent of a full dress rehearsal to flush out issues. I’ve seen similar concepts promoted around the Netflix Chaos Monkey pattern.

If your deployment is too fragile to risk breaking in development and test then you’re signing up for an on-going life of fire fighting. In that case, you’ll definitely need all the “clean underwear” you can find.

The RackN team has been working on making DevOps more portable for over five years.  Portable between vendors, sites, tools and operating systems means that our automation needs be to hybrid in multiple dimensions by design.

Why drive for hybrid?  It’s about giving users control.

I believe that application should drive the infrastructure, not the reverse.  I’ve heard may times that the “infrastructure should be invisible to the user.”  Unfortunately, lack of abstraction and composibility make it difficult to code across platforms.  I like the term “fidelity gap” to describe the cost of these differences.

What keeps DevOps from going hybrid?  Shortcuts related to platform entangled configuration management.

Everyone wants to get stuff done quickly; however, we make the same hard-coded ops choices over and over again.  Big bang configuration automation that embeds sequence assumptions into the script is not just technical debt, it’s fragile and difficult to upgrade or maintain.  The problem is not configuration management (that’s a critical component!), it’s the lack of system level tooling that forces us to overload the configuration tools.

What is system level tooling?  It’s integrating automation that expands beyond configuration into managing sequence (aka orchestration), service orientation, script modularity (aka composibility) and multi-platform abstraction (aka hybrid).

My ops automation experience says that these four factors must be solved together because they are interconnected.

What would a platform that embraced all these ideas look like?  Here is what we’ve been working towards with Digital Rebar at RackN:

Should we call this “Hybrid Devops?”  That sounds so buzz-wordy!

I’ve come to believe that Hybrid DevOps is the right name.  More technical descriptions like “composable ops” or “service oriented devops” or “cross-platform orchestration” just don’t capture the real value.  All these names fail to capture the portability and multi-system flavor that drives the need for user control of hybrid in multiple dimensions.

Simply put, we need devops without borders!

What do you think?  Do you have a better term?

The RackN team did not plan to replace Cobbler, we just needed something that responded to our need for full-cycle cross-platform DevOps automation.

Provisioning an O/S is never enough!  You need to coordinate a lot of operational activity to deploy a multi-node system, like OpenStack, Kubernetes, Docker Swarm or Ceph.  Since we believe an automated upgrade path is also required, there is a huge gap in provisioning.

So what was needed?  Here’s our (rather long!) list of gaps to fill for full Metal DevOps provisioning:

We built Digital Rebar to close these gaps and many others (like transparent in operation, working in containers, and failing fast).  We think it’s time to bring cloud operational practices into metal.  With this type of automation, we can make it happen!

What are your biggest challenges with Metal Ops?   Does it match this list?  I’ve love to hear your opinion.