How OpenStack installer (crowbar + chefops) works (video from 3/14 demo) March 14, 2011
Posted by Rob H in Crowbar, Greg Althaus, OpenStack, SXSW.Tags: Chef, Crowbar, DHCP, Open Source, OpenStack, OpsCode, People/Dell, PXE, rackspace
trackback
August 5th 2011 Update:
While still relevant and accurate, the information on this page does not reflect the latest information about the now Apache 2 released Crowbar code. In the 4+ months following this post, we substantially refactored the code make make it more modular (see Barclamps), better looking, and multi-vendor/multi-application (Hadoop & RHEL). If you want more information, I recommend that you try Crowbar for yourself.
Original March 14th 2011 Text:
I’ve been getting some “how does Crowbar work” inquiries and wanted to take a shot at adding some technical detail. Before I launch into technical babble, there are some important things to note:
- Dell has committed to open source release the code for Crowbar (Apache 2)
- Crowbar is an extension of Chef Server – it does not function stand alone and uses Chef’s APIs to store all it’s data.
- The OpenStack components install is managed by Chef cookbooks & recipes jointly developed by Dell, Opscode and Rackspace.
- Crowbar can be used to simply bootstrap your data center; however, we believe it is the start of a cloud operational model that I described in the hyperscale cloud white paper.
LIVE DEMO (video via Barton George): If you’re at SXSW on 3/14 @ 2pm in Kung Fu Salon, you can ask Greg Althaus to explain it – he does a better job than I do.
Here’s what you need to know to understand Crowbar:
Crowbar is a PXE state machine.
The primary function of Crowbar is to get new hardware into a state where it can be managed by Chef. To get hardware into a “Chef Ready” state, there are several steps that must be performed. We need to setup the BIOS, RAID, figure out where the server is racked, install an operating system, assign IP networking and names, synchronize clocks (NTP) and setup a chef client linked to our server. That’s a lot of steps!
In order to do these steps, we need to boot the server through a series of controlled images (stages) and track the progress through each state. That means that each state corresponds to a PXE boot image. The images have a simple script that uses WGET to update the Crowbar server (which stores it’s data in Chef) when the script completes. When a state is finished, Crowbar will change the PXE server to provide the next image in the sequence.
During the Crowbar managed part of the install, the servers will reboot several times. Once all of the hardware configuration is complete, Crowbar will use an operating system install image to create the base configuration. For the first release, we are only planning to have a single Operating System (Ubuntu 10.10); however, we expect to be adding more operating system options.
The current architecture of Crowbar (and the Chef Server that it extends) is to use a dedicated server in the system for administration. Our default install adds PXE, DHCP, NTP, DNS, Nagios, & Ganglia to the admin server. For small systems, you can use Chef to add other infrastructure capabilities to the admin server; unfortunately, adding components makes it harder to redeploy the components. For dynamic configurations where you may want to rehearse deployments while building Chef recipes, we recommend installing other infrastructure services on the admin server.
Of course, the hardware configuration steps are vendor specific. We had to make the state machine (stored in Chef data bags) configurable so that you can add or omit steps. Since hardware config is slow, error prone and painful, we see this as a big value add. Making it work for open source will depend on community participation.
Once Chef has control of the servers, you can use Chef (on the local Chef Server) to complete the OpenStack installation. From there, you can continue to use Chef to deploy VMs into the environment. Because Chef encourages a DevOps automation mindset, I believe there is a significant ROI to your investment in learning how this tool operates if you want to manage hyperscale clouds.
Crowbar effectively extends the reach of Chef earlier into the cloud management life cycle.
3/21 Note: Updated graphic to show WGET.
[...] How OpenStack installer (crowbar + chefops) works [...]
[...] How OpenStack installer (crowbar + chefops) works [...]
[...] team’s first OpenStack project started as a cloud installer (aka Crowbar), but we’ve learned that it is more fundamental than that. To achieve “4 hours to cloud,” [...]
Hi ,
I have been Exploring OpenStack for a couple of weeks now, and Crowbar seems to be very existing and interesting . Can you please provide where I can get more info for trying it.
Regards
Kanthi – contact openstack@dell.com if you want early access to Crowbar via our starter solution (as per the white paper). We’re working on the internal legal process to open source our code (Apache2) and then anyone will be able to get the code. Until it’s open, I’m trying to get an contributor/beta agreement in place.
We’ll be bringing the latest code and equipment to the Cactus conference.
[...] utilisation and using cloud-bursting for extra capacity, as well as a great demo of Dell’s Crowbar, taking 5 completely blank servers and having a Nova install running on them all controlled by [...]
[...] server utilisation and using cloud-bursting for extra capacity, as well as a great demo of Dell's Crowbar, taking 5 completely blank servers and having a Nova install running on them all controlled by [...]
[...] team at Dell is working diligently to release Crowbar (Apache 2) to the [...]
[...] source (next few weeks) and want to show the community the design thinking behind the project. Crowbar currently only embodies a fraction of this scope but we have designed it looking [...]
[...] Recipes, Swift | Leave a Comment Our OpenStack team here at Dell has been busy getting Crowbar ready to open source and that does not leave much time for blog posts. We’re putting on a [...]
[...] As promised, here’s the other drop in advance of our OpenStack team’s Crowbar release. [...]
[...] target for our product range and he was enthusiastic about tackling this challenge after seeing the Dell crowbar solution take more than half and hour to deploy an Open Stack node at the [...]
[...] by the community. Chef also got a boost as the underlying configuration tool for Dell’s software to automatically deploy OpenStack-based clouds. Robbins hopes Chef becomes the default for OpenStack deployments in general with those deciding to [...]
[...] Opscode‘s satellite offices to talk to Matt Ray about the integration work he is doing with Crowbar (a soon to be open source bare-metal provisioning tool from Dell), Chef, and OpenStack. This [...]
[...] of the Dell OpenStack Cloud Solution along with the open sourcing of the code behind our Crowbar software [...]
[...] of the Dell OpenStack Cloud Solution along with the open sourcing of the code behind our Crowbar software [...]
what is the installation and configuration and component in crowbar server.how dese the admin server install PXE, DHCP, NTP, DNS, Nagios, & Ganglia to nodes?What do we need on admin server to support the installation on nodes?
All of these actions are part of the install script that’s included in the Crowbar source. Most of those feature are orchestrated by Chef and you can inspected them in the Cookbooks of the Chef Server that Crowbar deploys. Please review the install script.
[...] Crowbar is a solution linking Dell and the OpenStack project that we’ve been following for some time. I discussed the design of Orchestra at length with Crowbar’s chief architect, Rob Hirschfeld, in San Antonio at the 2nd OpenStack Developer Summit in San Antonio in November 2010. [...]
I’m posting this link, but his facts about Crowbar are wrong.
[...] Crowbar is a solution linking Dell and the OpenStack project that we’ve been following for some time. I discussed the design of Orchestra at length with Crowbar’s chief architect, Rob Hirschfeld, in San Antonio at the 2nd OpenStack Developer Summit in San Antonio in November 2010. I’ve also seen OpsCode Matt Ray‘s excellent presentation/demonstration on Crowbar at the Texas Linux Fest. [...]
[...] instructions on how OpenStack installer works here… Share this:TwitterFacebookLike this:LikeBe the first to like this post. This entry was [...]
[...] getting deeper in the details of the code. Dell’s been focusing their efforts on the amazing OpenStack deployment tool: Crowbar, and it seems clear to me that HP is taking their interest quite a bit deeper into the core [...]
[...] Crowbar is a solution linking Dell and the OpenStack project that we’ve been following for some time. I discussed the design of Orchestra at length with Crowbar’s chief architect, Rob Hirschfeld, in San Antonio at the 2nd OpenStack Developer Summit in San Antonio in November 2010. I’ve also seen OpsCode Matt Ray‘s excellent presentation/demonstration on Crowbar at the Texas Linux Fest. [...]
online scripts…
[...]How OpenStack installer (crowbar + chefops) works (video from 3/14 demo) « Rob Hirschfeld's Blog[...]…
[...] like Openstack from Dell is offered as the key to an successfull private cloud. Try take a look at this article written by Rob Hirschfeld. It outrules the baseline, including using Crowbar as the baremetal [...]
[...] target for our product range and he was enthusiastic about tackling this challenge after seeing the Dell crowbar solution take more than half an hour to deploy an Open Stack node at the [...]