Evolution or Rebellion? The rise of Site Reliability Engineers (SRE)

Posted on December 29, 2016 by Rob H

What is a Google SRE? Charity Majors gave a great overview on Datanauts #65, Susan Fowler from Uber talks about “no ops” tensions and Patrick Hill from Atlassian wrote up a good review too. This is not new: Ben Treynor defined it back in 2014.

DevOps is under attack.

Well, not DevOps exactly but the common misconception that DevOps is about Developers doing Ops (it’s really about lean process, system thinking, and positive culture). It turns out the Ops is hard and, as I recently discussed with John Furrier, developers really really don’t want be that focused on infrastructure.

In fact, I see containers and serverless as a “developers won’t waste time on ops revolt.” (I discuss this more in my 2016 retrospective).

The tension between Ops and Dev goes way back and has been a source of confusion for me and my RackN co-founders. We believe we are developers, except that we spend our whole time focused on writing code for operations. With the rise of Site Reliability Engineers (SRE) as a job classification, our type of black swan engineer is being embraced as a critical skill. It’s recognized as the only way to stay ahead of our ravenous appetite for computing infrastructure.

I’ve been writing about Site Reliability Engineering (SRE) tasks for nearly 5 years under a lot of different names such as DevOps, Ready State, Open Operations and Underlay Operations. SRE is a term popularized by Google (there’s a book!) for the operators who build and automate their infrastructure. Their role is not administration, it is redefining how infrastructure is used and managed within Google.

Using infrastructure effectively is a competitive advantage for Google and their SREs carry tremendous authority and respect for executing on that mission.

Managers Meanwhile, we’re in the midst of an Enterprise revolt against running infrastructure. Companies, for very good reasons, are shutting down internal IT efforts in favor of using outsourced infrastructure. Operations has simply not been able to complete with the capability, flexibility and breadth of infrastructure services offered by Amazon.

SRE is about operational excellence and we keep up with the increasingly rapid pace of IT. It’s a recognition that we cannot scale people quickly as we add infrastructure. And, critically, it is not infrastructure specific.

Over the next year, I’ll continue to dig deeply into the skills, tools and processes around operations. I think that SRE may be the right banner for these thoughts and I’d like to hear your thoughts about that.

MORE? Here’s the next post in the series about Spiraling Ops Debt. Or Skip to Podcasts with Eric Wright and Stephen Spector.

Please stop the turtles! Underlay is it’s own thing.

Posted on December 14, 2016 by Rob H

DISCLAIMER: “Abstractions are helpful till they are not” rant about using the right tools for the job follows…

Turtle Stacking From Dr Seuss’ Yertle The Turtle

I’ve been hearing the Hindu phrase “turtles all the way down” very often lately to describe the practice of using products to try and install themselves (my original posting attributed this to Dr Seuss) . This seems especially true of the container platforms that use containers to install containers that manage the containers. Yes, really – I don’t make this stuff up.

While I’m a HUGE fan of containers (RackN uses them like crazy with Digital Rebar), they do not magically solve operational issues like security, upgrade or networking. In fact, they actually complicate operational concerns by creating additional segmentation.

Solving these issues requires building a robust, repeatable, and automated underlay. That is a fundamentally different problem than managing containers or virtual machines. Asking container or VM abstraction APIs to do underlay work breaks the purpose of the abstraction which is to hide complexity.

The lure of a universal abstraction, the proverbial “single pane of glass,” is the ultimate siren song that breeds turtle recursion.

I’ve written about that on DevOps.com in a pair of articles: It’s Time to Slay the Universal Installer Unicorn and How the Lure of an ‘Easy Button’ Installer Traps Projects. It seems obvious to me that universal abstraction is a oxymoron.

Another form of this pattern emerges from the square peg / round hole syndrome when we take a great tool and apply it to every job. For example, I was in a meeting when I heard “If you don’t think Kubernetes is greatest way to deploy software then go away [because we’re using it to install Kubernetes].” It may be the greatest way to deploy software for applications that fit its model, but it’s certainly not the only way.

What’s the solution? We should accept that there are multiple right ways to manage platforms depending on the level of abstraction that we want to expose.

Using an abstraction in the wrong place, hides information that we need to make good decisions. That makes it harder to automate, monitor and manage. It’s always faster, easier and safer when you’ve got the right tool for the job.

DevOps workers, you mother was right: always bring a clean Underlay.

Posted on February 10, 2016 by Rob H

Why did your mom care about underwear? She wanted you to have good hygiene. What is good Ops hygiene? It’s not as simple as keeping up with the laundry, but the idea is similar. It means that we’re not going to get surprised by something in our environment that we’d taken for granted. It means that we have a fundamental level of control to keep clean. Let’s explore this in context.

I’ve struggled with the term “underlay” for infrastructure of a long time. At RackN, we generally prefer the term “ready state” to describe getting systems prepared for install; however, underlay fits very well when we consider it as the foundation for a more building up a platform like Kubernetes, Docker Swarm, Ceph and OpenStack. Even more than single operator applications, these community built platforms require carefully tuned and configured environments. In my experience, getting the underlay right dramatically reduces installation challenges of the platform.

What goes into a clean underlay? All your infrastructure and most of your configuration.

Just buying servers (or cloud instances) does not make a platform. Cloud underlay is nearly as complex, but let’s assume metal here. To turn nodes into a cluster, you need setup their RAID and BIOS. Generally, you’ll also need to configure out-of-band management IPs and security. Those RAID and BIOS settings specific to the function of each node, so you’d better get that right. Then install the operating system. That will need access keys, IP addresses, names, NTP, DNS and proxy configuration just as a start. Before you connect to the wide, make sure to update to your a local mirror and site specific requirements. Installing Docker or a SDN layer? You may have to patch your kernel. It’s already overwhelming and we have not even gotten to the platform specific details!

Buried in this long sequence of configurations are critical details about your network, storage and environment.

Any mistake here and your install goes off the rails. Imagine that your building a house: it’s very expensive to change the plumbing lines once the foundation is poured. Thankfully, software configuration is not concrete but the costs of dealing with bad setup is just as frustrating.

The underlay is the foundation of your install. It needs to be automated and robust.

The challenge compounds once an installation is already in progress because adding the application changes the underlay. When (not if) you make a deploy mistake, you’ll have to either reset the environment or make your deployment idempotent (meaning, able to run the same script multiple times safely). Really, you need to do both.

Why do you need both fast resets and component idempotency? They each help you troubleshoot issues but in different ways. Fast resets ensure that you understand the environment your application requires. Post install tweaks can mask systemic problems that will only be exposed under load. Idempotent action allows you to quickly iterate over individual steps to optimize and isolate components. Together they create resilient automation and good hygiene.

In my experience, the best deployments involved a non-recoverable/destructive performance test followed by a completely fresh install to reset the environment. The Ops equivalent of a full dress rehearsal to flush out issues. I’ve seen similar concepts promoted around the Netflix Chaos Monkey pattern.

If your deployment is too fragile to risk breaking in development and test then you’re signing up for an on-going life of fire fighting. In that case, you’ll definitely need all the “clean underware” you can find.

Introducing Digital Rebar. Building strong foundations for New Stack infrastructure

Posted on September 23, 2015 by Rob H

This week, I have the privilege to showcase the emergence of RackN’s updated approach to data center infrastructure automation that is container-ready and drives “cloud-style” DevOps on physical metal. While it works at scale, we’ve also ensured it’s light enough to run a production-fidelity deployment on a laptop.

You grow to cloud scale with a ready-state foundation that scales up at every step. That’s exactly what we’re providing with Digital Rebar.

Over the past two years, the RackN team has been working on microservices operations orchestration in the OpenCrowbar code base. By embracing these new tools and architecture, Digital Rebar takes that base into a new directions. Yet, we also get to leverage a scalable heterogeneous provisioner and integrations for all major devops tools. We began with critical data center automation already working.

Why Digital Rebar? Traditional data center ops is being disrupted by container and service architectures and legacy data centers are challenged with gracefully integrating this new way of managing containers at scale: we felt it was time to start a dialog the new foundational layer of scale ops.

Both our code and vision has substantially diverged from the groundbreaking “OpenStack Installer” MVP the RackN team members launched in 2011 from inside Dell and is still winning prizes for SUSE.

We have not regressed our leading vendor-neutral hardware discovery and configuration features; however, today, our discussions are about service wrappers, heterogeneous tooling, immutable container deployments and next generation platforms.

Over the next few days, I’ll be posting more about how Digital Rebar works (plus video demos).

@NextCast chat about DefCore, Metal Ops and OpenStack evolution

Posted on June 3, 2015 by Rob H

In Vancouver, I sat down with Scott Sanchez (EMC) and Jeff Dickey (Redapt) for a NextCast discussion. We covered a lot of my favorite subjects including DefCore and Ready State bare metal operations.

One of the things I liked about this discussion was that we were able to pull together the seemly disparate threads that I’m work on around OpenStack.

Manage Hardware like a BOSS – latest OpenCrowbar brings API to Physical Gear

Posted on April 9, 2015 by Rob H

A few weeks ago, I posted about VMs being squeezed between containers and metal. That observation comes from our experience fielding the latest metal provisioning feature sets for OpenCrowbar; consequently, so it’s exciting to see the team has cut the next quarterly release: OpenCrowbar v2.2 (aka Camshaft). Even better, you can top it off with official software support.

Dual overhead camshaft housing by Neodarkshadow from Wikimedia Commons

The Camshaft release had two primary objectives: Integrations and Services. Both build on the unique functional operations and ready state approach in Crowbar v2.

1) For Integrations, we’ve been busy leveraging our ready state API to make physical servers work like a cloud. It gets especially interesting with the RackN burn-in/tear-down workflows added in. Our prototype Chef Provisioning driver showed how you can use the Crowbar API to spin servers up and down. We’re now expanding this cloud-like capability for Saltstack, Docker Machine and Pivotal BOSH.

2) For Services, we’ve taken ops decomposition to a new level. The “secret sauce” for Crowbar is our ability to interweave ops activity between components in the system. For example, building a cluster requires setting up pieces on different systems in a very specific sequence. In Camshaft, we’ve added externally registered services (using Consul) into the orchestration. That means that Crowbar will either use existing DNS, Database, or NTP services or set it’s own. Basically, Crowbar can now work FIT YOUR EXISTING OPS ENVIRONMENT without forcing a dedicated Crowbar only services like DHCP or DNS.

In addition to all these features, you can now purchase support for OpenCrowbar from RackN (my company). The Enterprise version includes additional server life-cycle workflow elements and features like HA and Upgrade as they are available.

There are AMAZING features coming in the next release (“Drill”) including a message bus to broadcast events from the system, more operating systems (ESXi, Xenserver, Debian and Mirantis’ Fuel) and increased integration/flexibility with existing operational environments. Several of these have already been added to the develop branch.

It’s easy to setup and test OpenCrowbar using containers, VMs or metal. Want to learn more? Join our community in Gitter, email list or weekly interactive community meetings (Wednesdays @ 9am PT).

Showing to how others explain Ready State & OpenCrowbar

Posted on April 2, 2015 by Rob H

I’m working on a series for DevOps.com to explain Functional Ops (expect it to start early next week!) and it’s very hard to convey it’s east-west API nature. So I’m always excited to see how other people explain how OpenCrowbar does ops and ready state.

This week I was blown away by the drawing that I’ve recreated for this blog post. It’s very clear graphic showing the operational complexity of heterogeneous infrastructure AND how OpenCrowbar normalizes it into a ready state.

It’s critical to realize that the height of each component tower varies by vendor and also by location with in the data center topology. Ready state is not just about normalizing different vendors gear; it’s really about dealing with the complexity that’s inherent in building a functional data center. It’s “little” things liking knowing how to to enumerate the networking interfaces and uplinks to build the correct teams.

If you think this graphic helps, please let me know.

Talking Functional Ops & Bare Metal DevOps with vBrownBag [video]

Posted on March 16, 2015 by Rob H

Last Wednesday (3/11/15), I had the privilege of talking with the vBrownBag crowd about Functional Ops and bare metal deployment. In this hour, I talk about how functional operations (FuncOps) works as an extension of ready state. FuncOps is a critical concept for providing abstractions to scale heterogeneous physical operations.

Timing for this was fantastic since we’d just worked out ESXi install capability for OpenCrowbar (it will exposed for work starting on Drill, the next Crowbar release cycle).

Here’s the brown bag:

If you’d like to see a demo, I’ve got hours of them posted:

Crowbar v2.1 demo: Visual Table of Contents [click for playlist]

Online Meetup Today (1/13): Build a rock-solid foundation under your OpenStack cloud

Posted on January 13, 2015 by Rob H

Reminder: Online meetup w/ Crowbar + OpenStack DEMO TODAY

Here’s the notice from the site (with my added Picture)

Tuesday, January 13, 2015
9:00 AM to 10:00 AM Pacific Time
Registration Link

Building cloud infrastructure requires a rock-solid foundation.

In this hour, Rob Hirschfeld will demo automated tooling, specifically OpenCrowbar, to prepare and integrate physical infrastructure to ready state and then use PackStack to install OpenStack.

The OpenCrowbar project started in 2011 as an OpenStack installer and had grown into a general purpose provisioning and infrastructure orchestration framework that works in parallel with multiple hardware vendors, operating systems and devops tools. These tools create a fast, durable and repeatable environment to install OpenStack, Ceph, Kubernetes, Hadoop or other scale platforms.

Rob will show off the latest features and discuss key concepts from the Crowbar operational model including Ready State, Functional Operations and Late Binding. These concepts, built into Crowbar, can be applied generally to make your operations more robust and scalable.

OpenCrowbar v2.1 Video Tour from Metal to OpenStack and beyond

Posted on January 8, 2015 by Rob H

With the OpenCrowbar v2.1 out, I’ve been asked to update the video library of Crowbar demos. Since a complete tour is about 3 hours, I decided to cut it down into focused demos that would allow you to start at an area of interest and work backwards.

I’ve linked all the videos below by title. Here’s a visual table on contents: