Building Crowbar post-modularization (15 minute how-to video)

Posted on September 20, 2011 by Rob H

Note: I’m putting build ISOs and Sledgehammer TARs on crowbar.zehicle.com if you don’t want to follow these steps then download the ISO. We are updating the ISO daily, so don’t assume that you have that latest!

To build Crowbar, you need a Linux machine and access to the internet. The video shows how you can use an Ubuntu 10.10 Rackspace Cloud Server. We build Crowbar inside our firewall on our PCs too. No matter how you do it, Crowbar is full of fuzzily delicious cloud bits.

For up-to-date instructions, see the Crowbar wiki Build ISO page.

Crowbar modularized: latest changes that make clouds even easier to create, update, and maintain

Posted on September 19, 2011 by Rob H

In the last week, my team at Dell completed a major refactoring of Crowbar that significantly improves our ability to bring in community contributions and field customizations. Today, we merged it into Crowbar’s public repo(s).

From the very first versions, our objective for Crowbar was to create the fastest and most reliable cloud deployments. Along the way, we realized Crowbar’s true potential lay in embracing DevOps as an operational model for maintaining clouds. That meant building up cloud deployments in layers from pieces that we call barclamps (extensions of Chef cookbooks). Our first version, centered on OpenStack Cactus, leveraged barclamps but was still created as a single system. This unified system was a huge step forward in cloud deployments, but did not live up to our CloudOps vision of continuous delivery.

In this version, each Crowbar barclamp is an independent delivery unit that can be integrated before, while or after installing Crowbar.

The core of the change is each barclamp, including the most core ones, are stored in independent code repositories. Putting the code into distinct repos means that each barclamp can have its own life cycle, its own maintainer site and its own dependency tree. This modularization allows customers to manage their Crowbar deployments with a very fine brush: they may choose to customize parts of the system, they could lock components to specific tag and they can bring in barclamps from other vendors.

While the core barclamps are automatically integrated into the Crowbar build using git submodules; other barclamps are installed into the system as needed. This allows you to pull in the suite of OpenStack barclamps at build time or to wait until your Crowbar system is running before installing. Once you install a barclamp, you are able to retrieve an updated barclamp and reapply it to the system.

This feature gives you the ability to 1) choose exactly what you want to include and 2) perform field updates to a live Crowbar system.

Let’s look at some examples:

The Cloud Foundry barclamp can be sourced Cloud Foundry instead of bundled into the Crowbar repository. This allows the team working on the cloud application to take ownership for their own deployment. As a continuous delivery proponent, I believe strongly that the development team should be responsible for ensuring that their code is deployable (refer to my OpenStack “Deployer API” blue print attempting to codify this).
DreamHost, maintainers of Ceph Storage, can maintain their own local barclamp repos for OpenStack that are cloned from our community Swift barclamp. This allows them to innovate and customize OpenStack deployments for their business and choose which updates to merge back to the community.
Rackspace Cloud Builders can work on the most leading edge OpenStack features and maintaining workable deployments on branches. As the code stabilizes, they simply merge in their changes.
Dell BIOS and RAID barclamps only support the PowerEdge C line today. When we offer PowerEdge R support, you will be able to install or update the barclamps to add that capability. If another hardware vendor creates a barclamp for their hardware then you can install that into your existing system.

I believe that these changes to Crowbar are a huge step forwards on our journey of creating a community supportable Open Operations framework. I hope that you are as excited as I am about these changes.

I encourage you to take the first step by trying out Crowbar and, ultimately, writing your own barclamps.

Post Scripts:

In addition to the modularization, the updated code includes RHEL as a deployment platform. At present, you must choose to be either RHEL or Ubuntu at build time.
We have enhanced the network barclamp to describe connections as more abstract connections, called conduits, between nodes. This is a powerful change, but requires some understanding before you start making changes.
We have only begun testing the change as of 9/12, we expect the system to be fully stabilized by 10/3. If you are not willing to deal with bugs then I recommend building the Crowbar “v1.0” tag (or using the ISOs from our July launch).

Don’t fork it up. OpenStack needs community collaboration

Posted on September 12, 2011 by Rob H

Cant we just be friends?

We’re standing on the eve of the OpenStack 4th Design summit (aka Essex) and I’m watching a frenzy of IT Goliaths (Dell, Citrix, Cisco, HP, Rackspace) and some Cloud Davids (Nebula, Stackops) try to tangle revenue streams from an open source cloud project.

I was pleased to read GigaOM‘s Derrick Harris validation of Dell’s strategy which featured my team’s contributions (Crowbar, OpenStack & Hadoop). We are working hard to bring these technologies to our customers in an open and collaborative way.

Dell has substantial IT assets to bring to bear on cloud solutions. All of them are ultimately tied to products that generate revenue for Dell; however, that does not prevent our being able to collaborate and share. On the contrary, we benefiting from input from our partners, customers and community to determine which features are needed to accelerate adoption. Our recent decision to accelerate Crowbar modularization is a clear example of that process.

It is essential to understand that this is not just about cloud technologies! It is about the collaborative way we are promoting them and the processes we are using to deliver them.

With Dell’s cloud moving at hurricane speed, it has been interesting to watch how other companies are setting their own OpenStack initiatives. It seems to me that many of these efforts involve forks from OpenStack that cannot/will not be contributed back the community. One (but not the only) example is from HP’s Emil Sayegh who says that “HP developers … ideas will be shared…” He does not commit to sharing HP’s code in his post. I hope that is an oversight and not their plan.

In time, forking may be needed. Right now, we need to focus on building a strong foundation. Open contributions of code are the engine of that success.

WHIR Webinar Notes: Prying Open the Cloud with Dell Crowbar & OpenStack

Posted on September 1, 2011 by Rob H

Panelists: Me (@zehicle) & Joseph B. George (@jbgeorge), Director, Cloud and Big Data Solutions, Dell

Moderator: Liam Eagle (@theWHIR) , Editor-in-Chief, Web Host Industry Review

Wow, this Webinar was an hour of OpenStack insights (see the whole thing). If you don’t have the hour then you can use my time line nodes to jump to what you want to hear.

2:50: Presentation Starts (introductions are over)
3:40: Joseph coins the word “dynormous” for dynamic & large scale clouds
4:40: Customers want to know how they are going to maintain a cloud
4:50: Customers don’t want a 9 month cycle for features, want it faster. DevOps gives us the flexibility to meet our customer needs as quickly as they want to.
7:11: Massive scalability… their (Rackspace & NASA) business is about scale
8:00: Rackspace and NASA started from the beginning to build a community
8:50: We have the data that this has staying power
10:10: We see a lot of companies joining in the community
11:56: Shout out to Opscode Chef
12:40: From bare metal to a fully functioning cloud in under 2 hours. Crowbar allows you to introduce new elements into the environment
13:40: Crowbar leverages our experience with cloud deployments
14:33: Dell was the only provider there from day 1. We have the most experience.
17:27: DevOps Poll
18:40: DevOps is a significant trend that you should consider. Hosters have a lot of operational chops.
19:34: There are a lot of right ways to do cloud. You need to pick what’s best for your business model
20:23: We could get hardware and software, but operational expertise was missing.
21:33: We’re more making the complexity of a cloud go away. We are getting our customers a head start. We are chipping away at the learning curve.
22:05: The cloud is always ready, never finished. Cloud is an ongoing operational environment: DevOps!
23:30: Crowbar bakes a lot of operational experience into the deployment.
25:17: Core tenant of DevOps: there is no single OpenStack image. Cloud is too complex. We build it in layers.
26:26: Before you deploy, you can change the configuration.
27:30: Barclamps are modules that execute a function. We are inviting community participation
28:40: Crowbar process view – Crowbar is a “PXE state machine” is a very simplified description.
29:57: You can go through a tuning cycle where you can get it working, make sure it’s right, flush and reset. That ensures you have an automated system.
30:34: Screen shots with descriptions
33:25: Event the core state machine that runs Crowbar is deployed as a barclamp
35:00: You can download OpenStack and install it yourself from our github. We don’t want to talk about OpenStack, we want to DO OpenStack.
Poll Results (see to the right)
38:00: Online resources
40:00: Question 1: Timeline for RHEL. Answer: RHEL is part of Hadoop, will make it into OpenStack by end of year (or sooner based on market demand)
42:17: Question 2: What led Dell to get involved in OpenStack? Answer: It’s about experience. We like being able to fix and change if we needed. There is a lot of active community
45:30: Question 3: How does a hardware maker play with open source software? Answer: It’s a solution for us. We wanted to make sure that people cloud deploy the software. Adding DevOps takes it to another level.
48:00 Question 4: What elements of Diablo are most exciting? Answer: Keystone (centralized authentication) is a big deal. Networking changes that “bust the top” of the networking hurdles.
50:25: Question 5: Where is OpenStack going long term? Answer: We’re pleasantly surprised about how much it’s picked up. We’ll see more standards in the community. We have high hopes for OpenStack and have invested heavily. We’ll see more as-a-service capabilities to build on a common infrastructure: both open and commercial.
52:47: Question 6: What’s the biggest barrier to operating at scale? Answer: learning how to operate is the biggest hurdle. We took a learning approach to help customers get started. We are hosting a training with Rackspace.
55:00: Question 7: Where does Dell and Rackspace overlap? Answer: We see Rackspace Cloud Builders that the premier experts. Dell Services is involved with all of it. Dell takes the phone call and deals with our customers directly.

OpenStack Cloud Training announced by RackSpace. Sessions in London@Rackspace & Austin@Dell

Posted on August 26, 2011 by Rob H

Talking to WHIR today, Joseph George and I were asked what’s the biggest gap in OpenStack? Our answer: operator expertise!

That’s why I’m excited to post about hands on Cloud Builder Training sessions based on the Rackspace training announcement yesterday.

Dell is hosting one of the five-day sessions at our Austin campus (register) starting on October 24th. Other sessions are in Boston (9/26) and London (10/10).

If you come to the Austin session, I can guarantee you’ll get to meet some of our Austin team (Rob, Joseph, Greg, Victor, AD, Nick and Joey). I’ll try to setup a visit to the Boston sessions by some of our Nashua NH members (Dan, Scott, Andi, Randy, Audra and Paul).

Build Sledgehammer, the Crowbar discovery image / build prerequisite

Posted on August 1, 2011 by Rob H

Note: This content has been copied to the Crowbar Wiki.

Victor “got your back” Lowther, CI & build automation czar on our team at Dell, spent a lot of time cleaning up the open source build to make it MUCH easier. The latest build only requires ONE server for all components. To make it repeatable and fast, I’m using a hosted VM from Rackspace Cloud.

Here are the steps that you should follow (cool: if you build before the prereqs are in place, the script will tell you what’s missing).

Note: You must build the discovery image (build_sledgehammer.sh) before building Crowbar. This image does not change very often, so it’s helpful to cache it somewhere (like in the Crowbar cache where it normally lives) and save time.

Starting from a Rackspace Cloud Ubuntu 10.10 image (512 RAM is OK, $0.03/hr)
Get libraries for git, RPM, & Ruby: apt-get install git rpm ruby
Get the sledgehammer repo: git clone git://github.com/dellcloudedge/crowbar-sledgehammer.git
Go to sledgehammer: cd crowbar-sledgehammer
Download the CentOS image: curl -o ../CentOS-5.6-x86_64-bin-DVD-1of2.iso http://mirror.cs.vt.edu/pub/CentOS/5.6/isos/x86_64/CentOS-5.6-x86_64-bin-DVD-1of2.iso

takes some time (10+ mins) even in the cloud

Tell the build where to look for the CentOS image: CENTOS_ISO=~/CentOS-5.6-x86_64-bin-DVD-1of2.iso ./build_sledgehammer.sh

you may need to change the path of the image if you did not put it in your home directory

wait a long time while magic happens and the tar gets created
check out the tar ball in the /bin directory!

Create the cache location for Sledgehammer: mkdir -p ~/.crowbar-build-cache
Move the the cache location: cd ~/.crowbar-build-cache
Extract the Sledgehammer tar: tar xzvf ~/crowbar-sledgehammer/bin/sledgehammer-tftpboot.tar.gz

Or, use the tar copy that I’ve cached it on zehicle.com! Then you can start at step 8.

Now you can build crowbar as per instructions (duplicated below)

cd ~
git clone git://github.com/dellcloudedge/crowbar.git
apt-get update
apt-get install build-essential mkisofs debootstrap
crowbar/build_crowbar.sh

kicks off a long download to create the cache (first time only!)
look in the home directory for the openstack-dev.iso

Of course, you still need to INSTALL CROWBAR (as root, /tftpboot/ubuntu_dvd/extra/install) after you use the ISO to boot a VM. Instructions on that shortly…

Cybera’s OpenStack efforts (includes Dell xrefs)

Posted on June 11, 2011 by Rob H

It’s awesome to see new deployments of OpenStack so I wanted to point out Cybera’s post about their OpenStack efforts!

Everett Toews does a nice job talking about the rationale for their decisions including some analysis of their hardware and vendor selections. Of course, I’m also happy to have them posting links back to my Dell team’s white paper and content.

I wanted to highlight one point that Everett makes:

“Is this the best mix of hardware possible for OpenStack? As always the answer is, “It depends.” It depends primarily on your the use cases for your cloud. We think we got a good mix of hardware but time will truely tell if it was the best mix possible for DAIR.”

I strongly agree, we (Dell) are still recommending starting with a smaller set general purpose hardware config that can be easily repurposed. Once you’ve figured out how your application maps into OpenStack then we’ll be ready to work togther to tune that order for 1000s of servers.

The OpenStack Rocket: How Citrix, Dell & Rackspace collaboration propels OpenStack

Posted on May 25, 2011 by Rob H

OpenStack has grown amazingly and picked up serious corporate support in its first year. Understanding how helps to explain why the initiative has legs and where adopters should invest. While OpenStack has picked up a lot of industry partners, early participation by Dell and Citrix have been important to a meteoric trajectory.

So why are we (Dell) working so hard to light a fire under OpenStack?

To explain OpenStack support and momentum, we have to start with a self-reflective fact: Dell (my employer), Citrix and Rackspace are seeking to gain the dominate positions in their respective areas of “cloud.” Individually, we have competitors in more entrenched positions for cloud silos in solutions, ecosystem, and hypervisor; however, our competitors are not acting in concertto maintain their position.

OpenStack offers an opportunity for companies trying to gain market share to leverage the strengths of partners against their competitors.

The surprising aspect of OpenStack is how much better this collaboration is working in practice than in theory. This, dare I say it, synergistic benefit comes to OpenStack from all the partners (Dell, Citrix, Rackspace and others) working together because:

Real scale clouds are big and complex with a lot of moving pieces (too big for a single vendor)
A vibrant ecosystem needs to see commercial commitment from large players (and avoids lock-in)
Having alternatives drives innovation and manages costs (because differentiation is still needed)
Interlocking expertise is required (because no vendor has all the pieces)
Customer needs are diverse and changing (since cloud is accelerating the rate of innovation)

Today, three major cloud players are standing together to demonstrate commitment to the community. This announcement is a foundation for the OpenStack ecosystem. In the next few months, I expect to see more and more collaborative announcements as the community proves the value of working together.

By aligning around an open platform, we collectively out flank previously dominant players who choose to go it alone. The technology is promising; however, the power of OpenStack flows mainly from the ecosystem that we are building together.

Rackspace will balance control of OpenStack. It takes time & strong partners

Posted on March 24, 2011 by Rob H

Rick Clark’s post “Why I Left Rackspace and What About Openstack” (+ his softer post script) is part of a longer conversation that started when Rackspace acquired Anso Labs and was expanded with the resignation of Chris Kemp (NASA CTO & OpenStack #1 fanboy).

Building a community is a delicate balance: you need show leadership while you cultivate leadership.

Putting aside the context (resigning from Rackspace to join Cisco) of his post, I think that Rick’s comments do resonate with parts of the community. OpenStack goverance became unbalanced when Anso became Rackspace. The governance board formed at the Austin conference was dominated by a small number (2: NASA/Anso & Rackspace) of highly committed voices but there was no single master.

Considering OpenStack’s momentum, we are in a very good position to fix the single master problem. However, it takes time. While companies like Dell (my employer), NTT, Citrix, Cisco (Rick’s employer), and Microsoft are clearly investing in OpenStack, none have yet achieved NASA or Rackspace’s level of technical committment.

The challenge for Rackspace is to expand the OpenStack market and ecosystem so that partners are motivated to jump in more and more quickly. If my experiences inside Dell are indicative of the broader community, Rackspace’s leadership makes it much easier for partners to increase their own commitment. Like teaching my daughter to ride her bike, she needed to know that I was running next to her before she would pedal hard enough to balance by herself.

Like teaching bike riding – you can’t lead communities too hard or too lightly.

To build a community around OpenStack, we (the partners) need to stand up our own capability. Until we have demonstrated more leadership, Rackspace must cultivate both a community and a market. This is a challenging role to balance. While the community wants distributed ownership, the market wants leadership. Rick’s governance comments are evidence of this struggle and Rick’s move to Cisco is an indication of leadership diversification.

I believe that Rackspace is committed to distributed ownership – we, in the community, need to rise to the challenge!

OpenStack still needs strong leadership from Rackspace because the market needs someone to be accountable for releases and features. That allows new partners to depend on someone to run beside them while the wobble their way along to independence. As the community leaders stand up, we’ll see a balanced community emerge. The challenge is on us to make that happen (and happen quickly).

How OpenStack installer (crowbar + chefops) works (video from 3/14 demo)

Posted on March 14, 2011 by Rob H

July 24th 2012 Update:

This page is very very old and Crowbar has progressed significantly since this was posted. For better information, please visit the Crowbar wiki and review my Crowbar 2 writeups.

August 5th 2011 Update:

While still relevant and accurate, the information on this page does not reflect the latest information about the now Apache 2 released Crowbar code. In the 4+ months following this post, we substantially refactored the code make make it more modular (see Barclamps), better looking, and multi-vendor/multi-application (Hadoop & RHEL). If you want more information, I recommend that you try Crowbar for yourself.

Original March 14th 2011 Text:

I’ve been getting some “how does Crowbar work” inquiries and wanted to take a shot at adding some technical detail. Before I launch into technical babble, there are some important things to note:

Dell has committed to open source release the code for Crowbar (Apache 2)
Crowbar is an extension of Chef Server – it does not function stand alone and uses Chef’s APIs to store all it’s data.
The OpenStack components install is managed by Chef cookbooks & recipes jointly developed by Dell, Opscode and Rackspace.
Crowbar can be used to simply bootstrap your data center; however, we believe it is the start of a cloud operational model that I described in the hyperscale cloud white paper.

LIVE DEMO (video via Barton George): If you’re at SXSW on 3/14 @ 2pm in Kung Fu Salon, you can ask Greg Althaus to explain it – he does a better job than I do.

Here’s what you need to know to understand Crowbar:

Crowbar is a PXE state machine.

The primary function of Crowbar is to get new hardware into a state where it can be managed by Chef. To get hardware into a “Chef Ready” state, there are several steps that must be performed. We need to setup the BIOS, RAID, figure out where the server is racked, install an operating system, assign IP networking and names, synchronize clocks (NTP) and setup a chef client linked to our server. That’s a lot of steps!

In order to do these steps, we need to boot the server through a series of controlled images (stages) and track the progress through each state. That means that each state corresponds to a PXE boot image. The images have a simple script that uses WGET to update the Crowbar server (which stores it’s data in Chef) when the script completes. When a state is finished, Crowbar will change the PXE server to provide the next image in the sequence.

During the Crowbar managed part of the install, the servers will reboot several times. Once all of the hardware configuration is complete, Crowbar will use an operating system install image to create the base configuration. For the first release, we are only planning to have a single Operating System (Ubuntu 10.10); however, we expect to be adding more operating system options.

The current architecture of Crowbar (and the Chef Server that it extends) is to use a dedicated server in the system for administration. Our default install adds PXE, DHCP, NTP, DNS, Nagios, & Ganglia to the admin server. For small systems, you can use Chef to add other infrastructure capabilities to the admin server; unfortunately, adding components makes it harder to redeploy the components. For dynamic configurations where you may want to rehearse deployments while building Chef recipes, we recommend installing other infrastructure services on the admin server.

Of course, the hardware configuration steps are vendor specific. We had to make the state machine (stored in Chef data bags) configurable so that you can add or omit steps. Since hardware config is slow, error prone and painful, we see this as a big value add. Making it work for open source will depend on community participation.

Once Chef has control of the servers, you can use Chef (on the local Chef Server) to complete the OpenStack installation. From there, you can continue to use Chef to deploy VMs into the environment. Because Chef encourages a DevOps automation mindset, I believe there is a significant ROI to your investment in learning how this tool operates if you want to manage hyperscale clouds.

Crowbar effectively extends the reach of Chef earlier into the cloud management life cycle.

3/21 Note: Updated graphic to show WGET.

Rob Hirschfeld

On Computing, Containers, Cloud & Tech Culture

Tag Archives: rackspace