Why DNS? Maintaining DNS is essential to scale ops. It’s not as simple as naming servers because each server will have multiple addresses (IPv4, IPv6, teams, bridges, etc) on multiple NICs depending on the systems function and applications. Plus, Errors in DNS are hard to diagnose.
I love talking about the small Ops things that make a huge impact in quality of automation. Things like automatically building a squid proxy cache infrastructure.
Today, I get to rave about the DNS integration that just surfaced in the OpenCrowbar code base. RackN CTO, Greg Althaus, just completed work that incrementally updates DNS entries as new IPs are added into the system.
Why is that a big deal? There are a lot of names & IPs to manage.
In physical ops, every time you bring up a physical or virtual network interface, you are assigning at least one IP to that interface. For OpenCrowbar, we are assigning two addresses: IPv4 and IPv6. Servers generally have 3 or more active interfaces (e.g.: BMC, admin, internal, public and storage) so that’s a lot of references. It gets even more complex when you factor in DNS round robin or other common practices.
Plus mistakes are expensive. Name resolution is an essential service for operations.
I know we all love memorizing IPv4 addresses (just wait for IPv6!) so accurate naming is essential. OpenCrowbar already aligns the address 4th octet (Admin .106 goes to the same server as BMC .106) but that’s not always practical or useful. This is not just a Day 1 problem – DNS drift or staleness becomes an increasing challenging problem when you have to reallocate IP addresses. The simple fact is that registering IPs is not the hard part of this integration – it’s the flexible and dynamic updates.
What DNS automation did we enable in OpenCrowbar? Here’s a partial list:
recovery of names and IPs when interfaces and systems are decommissioned
use of flexible naming patterns so that you can control how the systems are registered
ability to register names in multiple DNS infrastructures
ability to understand sub-domains so that you can map DNS by region
ability to register the same system under multiple names
wild card support for C-Names
ability to create a DNS round-robin group and keep it updated
But there’s more! The integration includes both BIND and PowerDNS integrations. Since BIND does not have an API that allows incremental additions, Greg added a Golang service to wrap BIND and provide incremental updates and deletes.
When we talk about infrastructure ops automation and ready state, this is the type of deep integration that makes a difference and is the hallmark of the RackN team’s ops focus with RackN Enterprise and OpenCrowbar.
I’ve just completed a basic Docker Machine driver for OpenCrowbar. This enables you to quickly spin-up (and down) remote Docker hosts on bare metal servers from their command line tool. There are significant cost, simplicity and performance advantages for this approach if you were already planning to dedicate servers to container workloads.
The basics are pretty simple: using Docker Machine CLI you can “create” and “rm” new Docker hosts on bare metal using the crowbar driver. Since we’re talking about metal, “create” is really “assign a machine from an available pool.”
Behind the scenes Crowbar is doing a full provision cycle of the system including installing the operating system and injecting the user keys. Crowbar’s design would allow operators to automatically inject additional steps, add monitoring agents and security, to the provisioning process without changing the driver operation.
Beyond Create, the driver supports the other Machine verbs like remove, stop, start, ssh and inspect. In the case of remove, the Machine is cleaned up and put back in the pool for the next user [note: work remains on the full remove>recreate process].
Overall, this driver allows Docker Machine to work transparently against metal infrastructure along side whatever cloud services you also choose.
Want to try it out?
You need to setup OpenCrowbar – if you follow the defaults (192.168.124.10 ip, user, password) then the Docker Machine driver defaults will also work. Also, make sure you have the Ubuntu 14.04 ISO available for the Crowbar provisioner
Discover some nodes in Crowbar – you do NOT need metal servers to try this, the tests work fine with virtual machines (tools/kvm-slave &)
Clone my Machine repo (Wde’re looking for feedback before a pull to Docker/Machine)
Compile the code using script/build.
Allocate a Docker Node using ./docker-machine create –driver crowbar testme
Go to the Crowbar UI to watch the node be provisioned and configured into the Docker-Machines pool
Release the node using ./docker-machine rm testme
Go to the Crowbar UI to watch the node be redeployed back to the System pool
A few weeks ago, I posted about VMs being squeezed between containers and metal. That observation comes from our experience fielding the latest metal provisioning feature sets for OpenCrowbar; consequently, so it’s exciting to see the team has cut the next quarterly release: OpenCrowbar v2.2 (aka Camshaft). Even better, you can top it off with official software support.
Dual overhead camshaft housing by Neodarkshadow from Wikimedia Commons
The Camshaft release had two primary objectives: Integrations and Services. Both build on the unique functional operations and ready state approach in Crowbar v2.
1) For Integrations, we’ve been busy leveraging our ready state API to make physical servers work like a cloud. It gets especially interesting with the RackN burn-in/tear-down workflows added in. Our prototype Chef Provisioning driver showed how you can use the Crowbar API to spin servers up and down. We’re now expanding this cloud-like capability for Saltstack, Docker Machine and Pivotal BOSH.
2) For Services, we’ve taken ops decomposition to a new level. The “secret sauce” for Crowbar is our ability to interweave ops activity between components in the system. For example, building a cluster requires setting up pieces on different systems in a very specific sequence. In Camshaft, we’ve added externally registered services (using Consul) into the orchestration. That means that Crowbar will either use existing DNS, Database, or NTP services or set it’s own. Basically, Crowbar can now work FIT YOUR EXISTING OPS ENVIRONMENT without forcing a dedicated Crowbar only services like DHCP or DNS.
In addition to all these features, you can now purchase support for OpenCrowbar from RackN (my company). The Enterprise version includes additional server life-cycle workflow elements and features like HA and Upgrade as they are available.
There are AMAZING features coming in the next release (“Drill”) including a message bus to broadcast events from the system, more operating systems (ESXi, Xenserver, Debian and Mirantis’ Fuel) and increased integration/flexibility with existing operational environments. Several of these have already been added to the develop branch.
Last Wednesday (3/11/15), I had the privilege of talking with the vBrownBag crowd about Functional Ops and bare metal deployment. In this hour, I talk about how functional operations (FuncOps) works as an extension of ready state. FuncOps is a critical concept for providing abstractions to scale heterogeneous physical operations.
Timing for this was fantastic since we’d just worked out ESXi install capability for OpenCrowbar (it will exposed for work starting on Drill, the next Crowbar release cycle).
Add hosts range “192.168.124.[81:83] ansible_ssh_user=root” to the
If you are really lazy, add “[Default] // host_key_checking = False” to your “~/.ansible.cfg” file
now ping the hosts, “ansible all -m ping”
pat yourself on the back, you’re done.
to show off:
touch all machines “ansible all -a “/bin/echo hello”
look at types of Linux “ansible all -a “uname -a”
Further integration work can make this even more powerful.
I’d like to see OpenCrowbar generate the Ansible inventory file from the discovery data and to map Ansible groups from deployments. Crowbar could also call Ansible directly to use playbooks or even do a direct hand-off to Tower to complete an install without user intervention.
Building cloud infrastructure requires a rock-solid foundation.
In this hour, Rob Hirschfeld will demo automated tooling, specifically OpenCrowbar, to prepare and integrate physical infrastructure to ready state and then use PackStack to install OpenStack.
The OpenCrowbar project started in 2011 as an OpenStack installer and had grown into a general purpose provisioning and infrastructure orchestration framework that works in parallel with multiple hardware vendors, operating systems and devops tools. These tools create a fast, durable and repeatable environment to install OpenStack, Ceph, Kubernetes, Hadoop or other scale platforms.
Rob will show off the latest features and discuss key concepts from the Crowbar operational model including Ready State, Functional Operations and Late Binding. These concepts, built into Crowbar, can be applied generally to make your operations more robust and scalable.
With the OpenCrowbar v2.1 out, I’ve been asked to update the video library of Crowbar demos. Since a complete tour is about 3 hours, I decided to cut it down into focused demos that would allow you to start at an area of interest and work backwards.
I’ve linked all the videos below by title. Here’s a visual table on contents:
Crowbar v2.1 demo: Visual Table of Contents [click for playlist]
The heart of the demo series is the Annealer and Ready State (video #3).