Hybrid & Container Disruption [Notes from CTP Mike Kavis’ Interview]

Posted on February 23, 2016 by Rob H

Last week, Cloud Technology Partner VP Mike Kavis (aka MadGreek65) and I talked for 30 minutes about current trends in Hybrid Infrastructure and Containers.

Mike Kavis

Three of the top questions that we discussed were:

Why Composability is required for deployment? [5:45]
Is Configuration Management dead? [10:15]
How can containers be more secure than VMs? [23:30]

Here’s the audio matching the time stamps in my notes:

00:44: What is RackN? – scale data center operations automation
01:45: Digital Rebar is… 3^rd generation provisioning to manage data center ops & bring up
02:30: Customers were struggling on Ops more than code or hardware
04:00: Rethinking “open” to include user choice of infrastructure, not just if the code is open source.
05:00: Use platforms where it’s right for users.
05:45: Composability – it’s how do we deal with complexity. Hybrid DevOps
06:40: How do we may Ops more portable
07:00: Five components of Hybrid DevOps
07:27: Rob has “Rick Perry” Moment…
08:30: 80/20 Rule for DevOps where 20% is mixed.
10:15: “Is configuration management dead” > Docker does hurt Configuration Management
11:00: How Service Registry can replace Configuration.
11:40: Reference to John Willis on the importance of sequence.
12:30: Importance of Sequence, Services & Configuration working together
12:50: Digital Rebar intermixes all three
13:30: The race to have orchestration – “it’s always been there”
14:30: Rightscale Report > Enterprises average SIX platforms in use
15:30: Fidelity Gap – Why everyone will hybrid but need to avoid monoliths
16:50: Avoid hybrid trap and keep a level of abstraction
17:41: You have to pay some “abstraction tax” if you want to hybrid BUT you can get some additional benefits: hybrid + ops management.
18:00: Rob gives a shout out to Rightscale
19:20: Rushing to solutions does not create secure and sustained delivery
20:40: If you work in a silo, you loose the ability to collaborate and reuse other works
21:05: Rob is sad about “OpenStack explosion of installers”
21:45: Container benefit from services containers – how they can be MORE SECURE
23:00: Automation required for security
23:30: How containers will be more secure than containers
24:30: Rob bring up “cheese” again…
26:15: If you have more situational awareness, you can be more secure WITHOUT putting more work for developers.
27:00: Containers can help developers worry about as many aspects of Ops
27:45: Wrap up

What do you think? I’d love to hear your opinion on these topics!

Composability is Critical in DevOps: let’s break the monoliths

Posted on February 22, 2016 by Rob H

This post was inspired by my DevOps.com Git for DevOps post and is an evolution of my “Functional Ops (the cake is a lie)” talks.

2016 is the year we break down the monoliths. We’ve spent a lot of time talking about monolithic applications and microservices; however, there’s an equally deep challenge in ops automation.

Anti-monolith composability means making our automation into function blocks that can be chained together by orchestration.

What is going wrong? We’re building fragile tightly coupled automation.

Most of the automation scripts that I’ve worked with become very long interconnected sequences well beyond the actual application that they are trying to install. For example, Kubernetes needs etcd as a datastore. The current model is to include the etcd install in the install script. The same is true for SDN install/configuation and post-install test and dashboard UIs. The simple “install Kubernetes” quickly explodes into a kitchen sink of related adjacent components.

Those installs quickly become fragile and bloated. Even worse, they have hidden dependencies. What happens when etcd changes. Now, we’ve got to track down all the references to it burried in etcd based applications. Further, we don’t get the benefits of etcd deployment improvements like secure or scale configuration.

What can we do about it? Resist the urge to create vertical silos.

It’s temping and fast to create automation that works in a very prescriptive way for a single platform, operating system and tool chain. The work of creating abstractions between configuration steps seems like a lot of overhead. Even if you create those boundaries or reuse upstream automation, you’re likely to be vulnerable to changes within that component. All these concerns drive operators to walk away from working collaboratively with each other and with developers.

Giving up on collaborative Ops hurts us all and makes it impossible to engineer excellent operational tools.

Don’t give up! Like git for development, we can do this together.

DevOps workers, you mother was right: always bring a clean Underlay.

Posted on February 10, 2016 by Rob H

Why did your mom care about underwear? She wanted you to have good hygiene. What is good Ops hygiene? It’s not as simple as keeping up with the laundry, but the idea is similar. It means that we’re not going to get surprised by something in our environment that we’d taken for granted. It means that we have a fundamental level of control to keep clean. Let’s explore this in context.

I’ve struggled with the term “underlay” for infrastructure of a long time. At RackN, we generally prefer the term “ready state” to describe getting systems prepared for install; however, underlay fits very well when we consider it as the foundation for a more building up a platform like Kubernetes, Docker Swarm, Ceph and OpenStack. Even more than single operator applications, these community built platforms require carefully tuned and configured environments. In my experience, getting the underlay right dramatically reduces installation challenges of the platform.

What goes into a clean underlay? All your infrastructure and most of your configuration.

Just buying servers (or cloud instances) does not make a platform. Cloud underlay is nearly as complex, but let’s assume metal here. To turn nodes into a cluster, you need setup their RAID and BIOS. Generally, you’ll also need to configure out-of-band management IPs and security. Those RAID and BIOS settings specific to the function of each node, so you’d better get that right. Then install the operating system. That will need access keys, IP addresses, names, NTP, DNS and proxy configuration just as a start. Before you connect to the wide, make sure to update to your a local mirror and site specific requirements. Installing Docker or a SDN layer? You may have to patch your kernel. It’s already overwhelming and we have not even gotten to the platform specific details!

Buried in this long sequence of configurations are critical details about your network, storage and environment.

Any mistake here and your install goes off the rails. Imagine that your building a house: it’s very expensive to change the plumbing lines once the foundation is poured. Thankfully, software configuration is not concrete but the costs of dealing with bad setup is just as frustrating.

The underlay is the foundation of your install. It needs to be automated and robust.

The challenge compounds once an installation is already in progress because adding the application changes the underlay. When (not if) you make a deploy mistake, you’ll have to either reset the environment or make your deployment idempotent (meaning, able to run the same script multiple times safely). Really, you need to do both.

Why do you need both fast resets and component idempotency? They each help you troubleshoot issues but in different ways. Fast resets ensure that you understand the environment your application requires. Post install tweaks can mask systemic problems that will only be exposed under load. Idempotent action allows you to quickly iterate over individual steps to optimize and isolate components. Together they create resilient automation and good hygiene.

In my experience, the best deployments involved a non-recoverable/destructive performance test followed by a completely fresh install to reset the environment. The Ops equivalent of a full dress rehearsal to flush out issues. I’ve seen similar concepts promoted around the Netflix Chaos Monkey pattern.

If your deployment is too fragile to risk breaking in development and test then you’re signing up for an on-going life of fire fighting. In that case, you’ll definitely need all the “clean underware” you can find.

We need DevOps without Borders! Is that “Hybrid DevOps?”

Posted on February 9, 2016 by Rob H

The RackN team has been working on making DevOps more portable for over five years. Portable between vendors, sites, tools and operating systems means that our automation needs be to hybrid in multiple dimensions by design.

Why drive for hybrid? It’s about giving users control.

launch! I believe that application should drive the infrastructure, not the reverse. I’ve heard may times that the “infrastructure should be invisible to the user.” Unfortunately, lack of abstraction and composibility make it difficult to code across platforms. I like the term “fidelity gap” to describe the cost of these differences.

What keeps DevOps from going hybrid? Shortcuts related to platform entangled configuration management.

Everyone wants to get stuff done quickly; however, we make the same hard-coded ops choices over and over again. Big bang configuration automation that embeds sequence assumptions into the script is not just technical debt, it’s fragile and difficult to upgrade or maintain. The problem is not configuration management (that’s a critical component!), it’s the lack of system level tooling that forces us to overload the configuration tools.

What is system level tooling? It’s integrating automation that expands beyond configuration into managing sequence (aka orchestration), service orientation, script modularity (aka composibility) and multi-platform abstraction (aka hybrid).

My ops automation experience says that these four factors must be solved together because they are interconnected.

What would a platform that embraced all these ideas look like? Here is what we’ve been working towards with Digital Rebar at RackN:

Mono-Infrastructure IT	“Hybrid DevOps”
Locked into a single platform	Portable between sites and infrastructures with layered ops abstractions.
Limited interop between tools	Adaptive to mix and match best-for-job tools. Use the right scripting for the job at hand and never force migrate working automation.
Ad hoc security based on site specifics	Secure using repeatable automated processes. We fail at security when things get too complex change and adapt.
Difficult to reuse ops tools	Composable Modules enable Ops Pipelines. We have to be able to interchange parts of our deployments for collaboration and upgrades.
Fragile Configuration Management	Service Oriented simplifies API integration. The number of APIs and services is increasing. Configuration management is not sufficient.
Big bang: configure then deploy scripting	Orchestrated action is critical because sequence matters. Building a cluster requires sequential (often iterative) operations between nodes in the system. We cannot build robust deployments without ongoing control over order of operations.

Should we call this “Hybrid Devops?” That sounds so buzz-wordy!

I’ve come to believe that Hybrid DevOps is the right name. More technical descriptions like “composable ops” or “service oriented devops” or “cross-platform orchestration” just don’t capture the real value. All these names fail to capture the portability and multi-system flavor that drives the need for user control of hybrid in multiple dimensions.

Simply put, we need devops without borders!

What do you think? Do you have a better term?

Full Metal DevOps: 12 things we needed beyond Cobbler

Posted on February 3, 2016 by Rob H

Almost a manifesto!

Rob Hirschfeld

The RackN team did not plan to replace Cobbler, we just needed something that responded to our need for full-cycle cross-platform DevOps automation.

Provisioning an O/S is never enough! You need to coordinate a lot of operational activity to deploy a multi-node system, like OpenStack, Kubernetes, Docker Swarm or Ceph. Since we believe an automated upgrade path is also required, there is a huge gap in provisioning.

So what was needed? Here’s our (rather long!) list of gaps to fill for full Metal DevOps provisioning:

Gap	Commentary
1	Needs to work with Cobbler!	Improve? Yes. Disrupt? Hell No! It has to be OK to leave Cobbler in place while we do something better. I’d be OK to tweak my Cobber to point it to the new stuff.
2	REST API & JSON CLI	Beyond the obvious API, we really want a way to write scripts that drive deployment proactively.
3	Modular Components	If…

View original post 358 more words

Post-OpenStack DefCore, I’m Chasing “open infrastructure” via cross-platform Interop

Posted on January 26, 2016 by Rob H

Like my previous DefCore interop windmill tilting, this is not something that can be done alone. Open infrastructure is a collaborative effort and I’m looking for your help and support. I believe solving this problem benefits us as an industry and individually as IT professionals.

So, what is open infrastructure? It’s not about running on open source software. It’s about creating platform choice and control. In my experience, that’s what defines open for users (and developers are not users).

I’ve spent several years helping lead OpenStack interoperability (aka DefCore) efforts to ensure that OpenStack cloud APIs are consistent between vendors. I strongly believe that effort is essential to build an ecosystem around the project; however, in talking to enterprise users, I’ve learned that that their real interoperability gap is between that many platforms, AWS, Google, VMware, OpenStack and Metal, that they use everyday.

Instead of focusing inward to one platform, I believe the bigger enterprise need is to address automation across platforms. It is something I’m starting to call hybrid DevOps because it allows users to mix platforms, service APIs and tools.

Open infrastructure in that context is being able to work across platforms without being tied into one platform choice even when that platform is based on open source software. API duplication is not sufficient: the operational characteristics of each platform are different enough that we need a different abstraction approach.

We have to be able to compose automation in a way that tolerates substitution based on infrastructure characteristics. This is required for metal because of variation between hardware vendors and data center networking and services. It is equally essential for cloud because of variation between IaaS capabilities and service delivery models. Basically, those minor differences between clouds create significant challenges in interoperability at the operational level.

Rationalizing APIs does little to address these more structural differences.

The problem is compounded because the differences are not nicely segmented behind abstraction layers. If you work to build and sustain a fully integrated application, you must account for site specific needs throughout your application stack including networking, storage, access and security. I’ve described this as all deployments have 80% of the work common but the remaining 20% is mixed in with the 80% instead of being nicely layers. So, ops is cookie dough not vinaigrette.

Getting past this problem for initial provisioning on a single platform is a false victory. The real need is portable and upgrade-ready automation that can be reused and shared. Critically, we also need to build upon the existing foundations instead of requiring a blank slate. There is openness value in heterogeneous infrastructure so we need to embrace variation and design accordingly.

This is the vision the RackN team has been working towards with open source Digital Rebar project. We now able to showcase workload deployments (Docker, Kubernetes, Ceph, etc) on multiple cloud platforms that also translate to full bare metal deployments. Unlike previous generations of this tooling (some will remember Crowbar), we’ve been careful to avoid injecting external dependencies into the DevOps scripts.

While we’re able to demonstrate a high degree of portability (or fidelity) across multiple platforms, this is just the beginning. We are looking for users and collaborators who want to want to build open infrastructure from an operational perspective.

You are invited to join us in making open cross-platform operations a reality.

Deployment Fidelity – reducing tooling transistions for fun and profit

Posted on January 11, 2016 by Rob H

At the OpenStack Tokyo summit, I gave a short interview on Deployment Fidelity. I’ve come to see the fidelity problem more broadly as the hybrid DevOps challenge that I described in my 2016 Predictions post as the end of mono-clouds. Thanks Ken Hui from OpenStack Superuser TV for resurfacing this link!

From Start to Scale: learn faster with heterogenous deployments

Posted on November 3, 2015 by Rob H

Why mix VMs and Physical? Having a consistent deploy approach can dramatically speed learning cycles that result in better scale ops. I would never deploy production OpenStack on VMs but I strongly recommend rehearsing that deployment on VMs hundreds of times before I touch metal.

Over the last two months, the RackN team redefined “heterogeneous” infrastructure in Digital Rebar from being “just” multi-vendor hardware to include any server resource from containers and Vagrant/Virtualbox to clouds like AWS or Packet. To support this truly diverse range, there were both technical and operational challenges to overcome.

The technical challenge rises from the fundamental control differences between cloud and physical infrastructure. In cloud, infrastructure is much more prescribed – you cannot change most aspects of your system and especially not your network interfaces or IPs. To provision hardware efficiently, we had to establish control over the very things that Cloud systems manage for you.

That management diversity exercised the full extent of the Digital Rebar “functional ops” architecture.

Over the last year, we’ve been unwinding baked-in control assumptions from earlier versions of Digital Rebar. That added flexibility allows Digital Rebar to mix control APIs for infrastructure ranging from using Cobbler to Docker, Vagrant and AWS. Since we could already cope with heterogeneous control APIs using Digital Rebar’s unique functional ops design, we retained the ability to mix and match container, virtual and physical infrastructure.

The operational challenge was more subtle. We were motivated to make this change by first hand observations of the fidelity gap. I am a strong believer that container platforms will directly target metal in the next two years. The challenge is how do we get there from our current virtualization-focused infrastructure.

It’s easy to look at the completed work as an obvious step forward. Looking over my shoulder, I know that it took years of learning and perseverance to create a platform that was flexible enough to handle both extremes of control. Even more important was understanding why it was so important for a physical scale deployment platform to provide ops fidelity for developers too.

With the infrastructure work behind us, we’re seeing Digital Rebar deliver real operational transformation. We want to help IT embrace containers and immutable infrastructure without having to discard the hard won battles installing cloud and traditional infrastructure. Most critically, we hope that you’ll join our open community and share your operational journey with us.

A year of RackN – 9 lessons from the front lines of evangalizing open physical ops

Posted on October 1, 2015 by Rob H

Let’s avoid this > “We’re heading right at the ground, sir! Excellent, all engines full power!

another scale? oars & motors. WWF managing small scale fisheries

RackN is refining our from “start to scale” message and it’s also our 1 year anniversary so it’s natural time for reflection. While it’s been a year since our founders made RackN a full time obsession, the team has been working together for over 5 years now with the same vision: improve scale datacenter operations.

As a backdrop, IT-Ops is under tremendous pressure to increase agility and reduce spending. Even worse, there’s a building pipeline of container driven change that we are still learning how to operate.

Over the year, we learned that:

no one has time to improve ops
everyone thinks their uniqueness is unique
most sites have much more in common than is different
the differences between sites are small
small differences really do break automation
once it breaks, it’s much harder to fix
everyone plans to simplify once they stop changing everything
the pace of change is accelerating
apply, rinse, repeat with lesson #1

Where does that leave us besides stressed out? Ops is not keeping up. The solution is not to going faster: we have to improve first and then accelerate.

What makes general purpose datacenter automation so difficult? The obvious answer, variation, does not sufficiently explain the problem. What we have been learning is that the real challenge is ordering of interdependencies. This is especially true on physical systems where you have to really grok* networking.

The problem would be smaller if we were trying to build something for a bespoke site; however, I see ops snowflaking as one of the most significant barriers for new technologies. At RackN, we are determined to make physical ops repeatable and portable across sites.

What does that heterogeneous-first automation look like? First, we’ve learned that to adapt to customer datacenters. That means using the DNS, DHCP and other services that you already have in place. And dealing with heterogeneous hardware types and a mix of devops tools. It also means coping with arbitrary layer 2 and layer 3 networking topologies.

This was hard and tested both our patience and architecture pattern. It would be much easier to enforce a strict hardware guideline, but we knew that was not practical at scale. Instead, we “declared defeat” about forcing uniformity and built software that accepts variation.

So what did we do with a year? We had to spend a lot of time listening and learning what “real operations” need. Then we had to create software that accommodated variation without breaking downstream automation. Now we’ve made it small enough to run on a desktop or cloud for sandboxing and a new learning cycle begins.

We’d love to have you try it out: rebar.digital.

* Grok is the correct work here. Thinking that you “understand networking” is often more dangerous when it comes to automation.

How do platforms die? One step at a time [the Fidelity Gap]

Posted on September 26, 2015 by Rob H

The RackN team is working on the “Start to Scale” position for Digital Rebar that targets the IT industry-wide “fidelity gap” problem. When we started on the Digital Rebar journey back in 2011 with Crowbar, we focused on “last mile” problems in metal and operations. Only in the last few months did we recognize the importance of automating smaller “first mile” desktop and lab environments.

A fidelity gap is created when work done on one platform, a developer laptop, does not translate faithfully to the next platform, a QA lab. Since there are gaps at each stage of deployment, we end up with the ops staircase of despair.

These gaps hide defects until they are expensive to fix and make it hard to share improvements. Even worse, they keep teams from collaborating.

With everyone trying out Container Orchestration platforms like Kubernetes, Docker Swarm, Mesosphere or Cloud Foundry (all of which we deploy, btw), it’s important that we can gracefully scale operational best practices.

For companies implementing containers, it’s not just about turning their apps into microservice-enabled immutable-rock stars: they also need to figure out how to implement the underlying platforms at scale.

My example of fidelity gap harm is OpenStack’s “all in one, single node” DevStack. There is no useful single system OpenStack deployment; however, that is the primary system for developers and automated testing. This design hides production defects and usability issues from developers. These are issues that would be exposed quickly if the community required multi-instance development. Even worse, it keeps developers from dealing with operational consequences of their decisions.

What are we doing about fidelity gaps? We’ve made it possible to run and faithfully provision multi-node systems in Digital Rebar on a relatively light system (16 Gb RAM, 4 cores) using VMs or containers. That system can then be fully automated with Ansible, Chef, Puppet and Salt. Because of our abstractions, if deployment works in Digital Rebar then it can scale up to 100s of physical nodes.

My take away? If you want to get to scale, start with the end in mind.