why is hardware hard? Ready State Physical Ops Meetup on Tuesday 12/2 9am PT

Posted on December 1, 2014 by Rob H

meh. Compared to cloud, Ops on physical infrastructure sinks.

Unfortunately, the cloud and scale platforms need to run someone so someone’s got to deal with it. In fact, we’ve got to deal with crates of cranky servers and flocks of finicky platforms. It’s enough to keep a good operator down.

If that’s you, or someone you care about, join us for a physical ops ~~support group~~ meetup on Tuesday 9am PT (11 central).

There is a light at the end of the tunnel! We can make it repeatable to provision OpenStack, Hadoop and other platforms.

As a community, we’re steadily bringing best practices and proven automation from cloud ops down into the physical space. On the OpenCrowbar project, we’re accelerating this effort using the ready state concept as a hand off point for “physical-cloud equivalency” and exploring the concept of “functional operations” to make DevOps scripts more portable.

Join me and Rafael Knuth for a discussion about how operators can work together to break the cycle.

Starting RackN – Delivering open ops by pulling an OpenCrowbar Bunny out of our hat

Posted on October 30, 2014 by Rob H

When Dell pulled out from OpenCrowbar last April, I made a commitment to our community to find a way to keep it going. Since my exit from Dell early in October 2014, that commitment has taken the form of RackN.

Today, we’re ready to help people run and expand OpenCrowbar (days away from v2.1!). We’re also seeking investment to make the project more “enterprise-ready” and build integrations that extend ready state.

RackN focuses on maintenance and support of OpenCrowbar for ready state physical provisioning. We will build the community around Crowbar as an open operations core and extend it with a larger set of hardware support and extensions. We are building partnerships to build application integration (using Chef, Puppet, Salt, etc) and platform workloads (like OpenStack, Hadoop, Ceph, CloudFoundry and Mesos) above ready state.

I’ve talked with hundreds of people about the state of physical data center operations at scale. Frankly, it’s a scary state of affairs: complexity is increasing for physical infrastructure and we’re blurring the lines by adding commodity networking with local agents into the mix.

Making this jumble of stuff work together is not sexy cloud work – I describe it as internet plumbing to non-technical friends. It’s unforgiving, complex and full of sharp edge conditions; however, people are excited to hear about our hardware abstraction mission because it solves a real pain for operators.

I hope you’ll stay tuned, or even play along, as we continue the Open Ops journey.

Unicorn captured! Unpacking multi-node OpenStack Juno from ready state.

Posted on October 23, 2014 by Rob H

OpenCrowbar Packstack install demonstrates that abstracting hardware to ready state smooths install process. It’s a working balance: Crowbar gets the hardware, O/S & networking right while Packstack takes care of OpenStack.

The Crowbar team produced the first open OpenStack installer back in 2011 and it’s been frustrating to watch the community fragment around building a consistent operational model. This is not an OpenStack specific problem, but I think it’s exaggerated in a crowded ecosystem.

When I step back from that experience, I see an industry wide pattern of struggle to create scale deployments patterns that can be reused. Trying to make hardware uniform is unicorn hunting, so we need to create software abstractions. That’s exactly why IaaS is powerful and the critical realization behind the OpenCrowbar approach to physical ready state.

So what has our team created? It’s not another OpenStack installer – we just made the existing one easier to use.

We build up a ready state infrastructure that makes it fast and repeatable to use Packstack, one of the leading open OpenStack installers. OpenCrowbar can do the same for the OpenStack Chef cookbooks or Salt Formula. It can even use Saltstack, Chef and Puppet together (which we do for the Packstack work)! Plus we can do it on multiple vendors hardware and with different operating systems. Plus we build the correct networks!

For now, the integration is available as a private beta (inquiries welcome!) because our team is not in the OpenStack support business – we are in the “get scale systems to ready state and integrate” business. We are very excited to work with people who want to take this type of functionality to the next level and build truly repeatable, robust and upgradable application deployments.

OpenCrowbar bootstrap positions SSH Keys for hand-offs

Posted on October 21, 2014 by Rob H

I was reading a ComputerWorld article about how Google and Amazon achieve scale. The theme: you must do better than linear cost scale and the only way to achieve that is to automate and commoditize hardware. I find interesting parallels in the Crowbar physical devops effort.

As the OpenCrowbar team continues to explore the concepts around “ready state,” I discover more and more small ops nuisances that need to be included in the build up before installing software. These small items quickly add up at scale breaking the rule above.

I’ve already posted about the performance benefit of building a Squid Proxy fabric as part of the underlying ops environment. As we work on Chef Metal, SaltStack and Packstack integrations (private beta), we’ve rediscovered the importance of management/population of SSH public keys.

In cloud infrastructure, key injection is taken for granted; however, it’s not an automatic behavior in the physical ops. Since OpenCrowbar handles keys by default but other tools (like Cobbler or Razor) expect that you will use kickstart to inject your SSH keys when you install the Operating System..

Including keys in kickstart (which I’m using generically instead of preseed, auto-yast, jumpstart, etc) hand generated scripts is a potentially dangerous security practice since it makes it difficult to propagate and manage your keys. It also means that every time a new operating system update is released that you may have to update and retest your kickstarts. OpenCrowbar has the same challenge but our approach allows everyone can share in the work because our bootstrapping files are scripted and generic.

OpenCrowbar takes care of these ready state configurations in our integrations with these DevOps platforms. Our experience has been that little items like SSH keys and proxy configurations can make a disproportionate advantage in running scale ops or during iterative development.

OpenCrowbar 2.B to deliver multiple hardware vendor support and advanced integrations

Posted on October 9, 2014 by Rob H

I’ve stayed quiet on the subject of Crowbar for a few months, but that does not mean that Crowbar has been. Activity has been picking up, after Dell pulled resources off, to complete hardware configuration.

[Disclosure: As of 10/3/2014, I am no longer a Dell employee]

With the re-addition of hardware configuration, OpenCrowbar delivers the essential requirements for Ready State and we’ve piloted integration that shows how to drive Crowbar via the API.

From BuildersKnowledge

There has been substantial burn-down on the Broom release theme of hardware workload deliverable which mainly focus on the IPMI/BMC, RAID and BIOS functions working in the framework. It has required us to add additional out-of-band abstractions (“hammers”) and node abstractions (“quirks”).

We’ve also had a chance to work ahead on the Camshaft release theme of tools integration components like:

SaltStack Integration – Crowbar sets up a Salt master and minions on discovered metal (pull request)
Chef Metal Integration – a Chef Metal driver talks to the Crowbar API to claim discovered servers from an allocation pool (Judd’s repo).
Puppet Integration – Crowbar is able to use the stand-alone mode to execute Puppet manifests on the nodes (as a replacement for Foreman) (puppet sa client).
Chef Integration – not new, but worth including in the list so it’s not overlooked! (chef-client install)
We also added some essential operational configurations including Squid proxy setup and auto configuration and preparing a Consul foundation for future integration with HashiCorp tools

These initial integration are key to being able to bring in OpenStack via Packstack, Chef Cookbooks, or Salt formulas. Since Crowbar is agnostic about OS, Hardware and Configuration Management tools (Chef, Puppet, Salt), I am seeing interest from several fronts in parallel. There seems to be substantial interest in RDO + Centos 7 using Packstack or Chef. Happily, OpenCrowbar.Broom is ready to sweep in those workloads.

There is significant need for Crowbar to deliver ready state under these deployers. For example, preparing the os, disk, monitoring, cache, networking and SDN infrastructure (OVS, Contrails) are outside the scripts but essential to a sustainable deployment.

These ready state configurations are places where Crowbar creates repeatable cross-platform base that spans the operational choices.

Boot me up! out-of-band IPMI rocks then shuts up and waits

Posted on July 16, 2014 by Rob H

It’s hard to get excited about re-implementing functionality from v1 unless the v2 happens to also be freaking awesome. It’s awesome because the OpenCrowbar architecture allows us to it “the right way” with real out-of-band controls against the open WSMAN APIs.

With out-of-band control, we can easily turn systems on and off using OpenCrowbar orchestration. This means that it’s now standard practice to power off nodes after discovery & inventory until they are ready for OS installation. This is especially interesting because many servers RAID and BIOS can be configured out-of-band without powering on at all.

Frankly, Crowbar 1 (cutting edge in 2011) was a bit hacky. All of the WSMAN control was done in-band but looped through a gateway on the admin server so we could access the out-of-band API. We also used the vendor (Dell) tools instead of open API sets.

That means that OpenCrowbar hardware configuration is truly multi-vendor. I’ve got Dell & SuperMicro servers booting and out-of-band managed. Want more vendors? I’ll give you my shipping address.

OpenCrowbar does this out of the box and in the open so that everyone can participate. That’s how we solve this problem as an industry and start to cope with hardware snowflaking.

And this out-of-band management gets even more interesting…

Since we’re talking to servers out-of-band (without the server being “on”) we can configure systems before they are even booted for provisioning. Since OpenCrowbar does not require a discovery boot, you could pre-populate all your configurations via the API and have the Disk and BIOS settings ready before they are even booted (for models like the Dell iDRAC where the BMCs start immediately on power connect).

Those are my favorite features, but there’s more to love:

the new design does not require network gateway (v1 did) between admin and bmc networks (which was a security issue)
the configuration will detect and preserves existing assigned IPs. This is a big deal in lab configurations where you are reusing the same machines and have scripted remote consoles.
OpenCrowbar offers an API to turn machines on/off using the out-of-band BMC network.
The system detects if nodes have IPMI (VMs & containers do not) and skip configuration BUT still manage to have power control using SSH (and could use VM APIs in the future)
Of course, we automatically setup BMC network based on your desired configuration

Ops Bridges > Building a Sharable Ops Infrastructure with Composable Tool Chain Orchestration

Posted on July 2, 2014 by Rob H

This posted started from a discussion with Judd Maltin that he documented in a post about “wanting a composable run deck.”

I’ve had several conversations comparing OpenCrowbar with other “bare metal provisioning” tools that do thing like serve golden images to PXE or IPXE server to help bootstrap deployments. It’s those are handy tools, they do nothing to really help operators drive system-wide operations; consequently, they have a limited system impact/utility.

In building the new architecture of OpenCrowbar (aka Crowbar v2), we heard very clearly to have “less magic” in the system. We took that advice very seriously to make sure that Crowbar was a system layer with, not a replacement to, standard operations tools.

Specifically, node boot & kickstart alone is just not that exciting. It’s a combination of DHCP, PXE, HTTP and TFTP or DHCP and an IPXE HTTP Server. It’s a pain to set this up, but I don’t really get excited about it anymore. In fact, you can pretty much use open ops scripts (Chef) to setup these services because it’s cut and dry operational work.

Note: Setting up the networking to make it all work is perhaps a different question and one that few platforms bother talking about.

So, if doing node provisioning is not a big deal then why is OpenCrowbar important? Because sustaining operations is about ongoing system orchestration (we’d say an “operations model“) that starts with provisioning.

It’s not the individual services that’s critical; it’s doing them in a system wide sequence that’s vital.

Crowbar does NOT REPLACE the services. In fact, we go out of our way to keep your proven operations tool chain. We don’t want operators to troubleshoot our IPXE code! We’d much rather use the standard stuff and orchestrate the configuration in a predicable way.

In that way, OpenCrowbar embraces and composes the existing operations tool chain into an integrated system of tools. We always avoid replacing tools. That’s why we use Chef for our DSL instead of adding something new.

What does that leave for Crowbar? Crowbar is providing a physical infratsucture targeted orchestration (we call it “the Annealer”) that coordinates this tool chain to work as a system. It’s the system perspective that’s critical because it allows all of the operational services to work together.

For example, when a node is added then we have to create v4 and v6 IP address entries for it. This is required because secure infrastructure requires reverse DNS. If you change the name of that node or add an alias, Crowbar again needs to update the DNS. This had to happen in the right sequence. If you create a new virtual interface for that node then, again, you need to update DNS. This type of operational housekeeping is essential and must be performed in the correct sequence at the right time.

The critical insight is that Crowbar works transparently alongside your existing operational services with proven configuration management tools. Crowbar connects links in your tool chain but keeps you in the driver’s seat.

OpenCrowbar stands up 100 node community challenge

Posted on July 2, 2014 by Rob H

OpenCrowbar community contributors are offering a “100 Node Challenge” by volunteering to setup a 100+ node Crowbar system to prove out the v2 architecture at scale. We picked 100* nodes since we wanted to firmly break the Crowbar v1 upper ceiling.

The goal of the challenge is to prove scale of the core provisioning cycle. It’s intended to be a short action (less than a week) so we’ll need advanced information about the hardware configuration. The expectation is to do a full RAID/Disk hardware configuration beyond the base IPMI config before laying down the operating system.

The challenge logistics starts with an off-site prep discussion of the particulars of the deployment, then installing OpenCrowbar at the site and deploying the node century. We will also work with you about using OpenCrowbar to manage the environment going forward.

Sound too good to be true? Well, as community members are doing this on their own time, we are only planning one challenge candidate and want to find the right target.

We will not be planning custom code changes to support the deployment, however, we would be happy to work with you in the community to support your needs. If you want help to sustain the environment or have longer term plans, I have also been approached by community members who willing to take on full or part-time Crowbar consulting engagements.

Let’s get rack’n!

* we’ll consider smaller clusters but you have to buy the drinks and pizza.

You need a Squid Proxy fabric! Getting Ready State Best Practices

Posted on July 1, 2014 by Rob H

Sometimes a solving a small problem well makes a huge impact for operators. Talking to operators, it appears that automated configuration of Squid does exactly that.

If you were installing OpenStack or Hadoop, you would not find “setup a squid proxy fabric to optimize your package downloads” in the install guide. That’s simply out of scope for those guides; however, it’s essential operational guidance. That’s what I mean by open operations and creating a platform for sharing best practice.

Deploying a base operating system (e.g.: Centos) on a lot of nodes creates bit-tons of identical internet traffic. By default, each node will attempt to reach internet mirrors for packages. If you multiply that by even 10 nodes, that’s a lot of traffic and a significant performance impact if you’re connection is limited.

For OpenCrowbar developers, the external package resolution means that each dev/test cycle with a node boot (which is up to 10+ times a day) is bottle necked. For qa and install, the problem is even worse!

Our solution was 1) to embed Squid proxies into the configured environments and the 2) automatically configure nodes to use the proxies. By making this behavior default, we improve the overall performance of a deployment. This further improves the overall network topology of the operating environment while adding improved control of traffic.

This is a great example of how Crowbar uses existing operational tool chains (Chef configures Squid) in best practice ways to solve operations problems. The magic is not in the tool or the configuration, it’s that we’ve included it in our out-of-the-box default orchestrations.

It’s time to stop fumbling around in the operational dark. We need to compose our tool chains in an automated way! This is how we advance operational best practice for ready state infrastructure.

OpenCrowbar Design Principles: Simulated Annealing [Series 4 of 6]

Posted on May 29, 2014 by Rob H

This is part 4 of 6 in a series discussing the principles behind the “ready state” and other concepts implemented in OpenCrowbar. The content is reposted from the OpenCrowbar docs repo.

Simulated Annealing

Simulated Annealing is a modeling strategy from Computer Science for seeking optimum or stable outcomes through iterative analysis. The physical analogy is the process of strengthening steel by repeatedly heating, quenching and hammering. In both computer science and metallurgy, the process involves evaluating state, taking action, factoring in new data and then repeating. Each annealing cycle improves the system even though we may not know the final target state.

Annealing is well suited for problems where there is no mathematical solution, there’s an irregular feedback loop or the datasets change over time. We have all three challenges in continuous operations environments. While it’s clear that a deployment can modeled as directed graph (a mathematical solution) at a specific point in time, the reality is that there are too many unknowns to have a reliable graph. The problem is compounded because of unpredictable variance in hardware (NIC enumeration, drive sizes, BIOS revisions, network topology, etc) that’s even more challenging if we factor in adapting to failures. An operating infrastructure is a moving target that is hard to model predictively.

Crowbar implements the simulated annealing algorithm by decomposing the operations infrastructure into atomic units, node-roles, that perform the smallest until of work definable. Some or all of these node-roles are changed whenever the infrastructure changes. Crowbar anneals the environment by exercising the node-roles in a consistent way until system re-stabilizes.

One way to visualize Crowbar annealing is to imagine children who have to cross a field but don’t have a teacher to coordinate. Once student takes a step forward and looks around then another sees the first and takes two steps. Each child advances based on what their peers are doing. None wants to get too far ahead or be left behind. The line progresses irregularly but consistently based on the natural relationships within the group.

To understand the Crowbar Annealer, we have to break it into three distinct components: deployment timeline, annealing and node-role state. The deployment timeline represents externally (user, hardware, etc) initiated changes that propose a new target state. Once that new target is committed, Crowbar anneals by iterating through all the node-roles in a reasonable order. As the Annealer runs the node-roles they update their own state. The aggregate state of all the node-roles determines the state of the deployment.

A deployment is a combination of user and system defined state. Crowbar’s job is to get deployments stable and then maintain over time.

Rob Hirschfeld

On Computing, Containers, Cloud & Tech Culture

Tag Archives: OpenCrowbar