Build Sledgehammer, the Crowbar discovery image / build prerequisite

Note: This content has been copied to the Crowbar Wiki.
Victor “got your back” Lowther, CI & build automation czar on our team at Dell, spent a lot of time cleaning up the open source build to make it MUCH easier.  The latest build only requires ONE server for all components.  To make it repeatable and fast, I’m using a hosted VM from Rackspace Cloud.
Here are the steps that you should follow (cool: if you build before the prereqs are in place, the script will tell you what’s missing).
Note: You must build the discovery image (build_sledgehammer.sh) before building Crowbar.  This image does not change very often, so it’s helpful to cache it somewhere (like in the Crowbar cache where it normally lives) and save time.
  1. Starting from a Rackspace Cloud Ubuntu 10.10 image (512 RAM is OK, $0.03/hr)
  2. Get libraries for git, RPM, & Ruby: apt-get install git rpm ruby
  3. Get the sledgehammer repo: git clone git://github.com/dellcloudedge/crowbar-sledgehammer.git
  4. Go to sledgehammer: cd crowbar-sledgehammer
  5. Download the CentOS image: curl -o ../CentOS-5.6-x86_64-bin-DVD-1of2.iso http://mirror.cs.vt.edu/pub/CentOS/5.6/isos/x86_64/CentOS-5.6-x86_64-bin-DVD-1of2.iso
    1. takes some time (10+ mins) even in the cloud
  6. Tell the build where to look for the CentOS image: CENTOS_ISO=~/CentOS-5.6-x86_64-bin-DVD-1of2.iso ./build_sledgehammer.sh
    1. you may need to change the path of the image if you did not put it in your home directory
    1. wait a long time while magic happens and the tar gets created
    2. check out the tar ball in the /bin directory!
  7. Create the cache location for Sledgehammer: mkdir -p ~/.crowbar-build-cache
  8. Move the the cache location: cd ~/.crowbar-build-cache
  9. Extract the Sledgehammer tar: tar xzvf ~/crowbar-sledgehammer/bin/sledgehammer-tftpboot.tar.gz 
Or, use the tar copy that I’ve cached it on zehicle.com!  Then you can start at step 8.
Now you can build crowbar as per instructions (duplicated below)
  1. cd ~
  2. git clone git://github.com/dellcloudedge/crowbar.git
  3. apt-get update
  4. apt-get install build-essential mkisofs debootstrap
  5. crowbar/build_crowbar.sh
    1. kicks off a long download to create the cache (first time only!)
    2. look in the home directory for the openstack-dev.iso

Of course, you still need to INSTALL CROWBAR (as root, /tftpboot/ubuntu_dvd/extra/install) after you use the ISO to boot a VM.  Instructions on that shortly…

Crowbar build using Ubuntu 10.10 vm on Rackspace Cloud from Github Repo

Our OpenStack team at Dell (especially Victor Lowthor) has been working hard with the public Crowbar repos to make it possible for the community to build their own version of a Crowbar ISO.   When you build the ISO, you’ll be downloading a whole bunch (that’s the technical term) of open source licensed components to make it work: we’re trying to maintain a list of licenses on the Github wiki.

To make sure that it was possible for mortals, I signed up for a Ubuntu 10.10 VM (512 Mb RAM, $0.03/hr) at RackSpace Cloud.  I did this from a non-Dell to ensure that it was as independent from our source as possible.

Once I had my vm, there were just a few steps to follow (these are NOT verbatim):

  • apt-get install debootstrap, mkisofs, git, build-essential packages
  • git clone git://github.com/dellcloudedge/crowbar.git
  • Got the results from a sledgehammer build (a fresh sledgehammer tarball) and extracted it into $HOME/.crowbar-build-cache/tftpboot, which is where build_crowbar.sh expects to find it cached.
    • NOTE: I’m not ready to document sledgehammer builds yet, but I will tell you that you’d need a CentOS VM.
  • In the crowbar directory, ran ./build_crowbar.sh
  • The build will pull down all the packages that you need and cache them to the VM.  Subsequent builds will be much faster!

The end result of the build is an “openstack-dev.iso” that will install Crowbar with the OpenStack barclamps (here’s how to do it on VMs).  Just for fun, I copied _my build_ output ISO off the build VM and to my web server.

Please let me know if you have problems with this process, we want people to try Crowbar!

$$ Note: Turn off your VM when you’re done so you don’t incur extra expenses.  Since this process only took about 2 hours, the whole build cost me less than a dime.  Which is good, since I was building it on “my own dime” anyway.

The OpenStack Rocket: How Citrix, Dell & Rackspace collaboration propels OpenStack

OpenStack has grown amazingly and picked up serious corporate support in its first year.  Understanding how helps to explain why the initiative has legs and where adopters should invest.  While OpenStack has picked up a lot of industry partners, early participation by Dell and Citrix have been important to a meteoric trajectory.

So why are we (Dell) working so hard to light a fire under OpenStack?

To explain OpenStack support and momentum, we have to start with a self-reflective fact: Dell (my employer), Citrix and Rackspace are seeking to gain the dominate positions in their respective areas of “cloud.”  Individually, we have competitors in more entrenched positions for cloud silos in solutions, ecosystem, and hypervisor; however, our competitors are not acting in concertto maintain their position.   

OpenStack offers an opportunity for companies trying to gain market share to leverage the strengths of partners against their competitors.

The surprising aspect of OpenStack is how much better this collaboration is working in practice than in theory.  This, dare I say it, synergistic benefit comes to OpenStack from all the partners (Dell, Citrix, Rackspace and others) working together because:

  1. Real scale clouds are big and complex with a lot of moving pieces (too big for a single vendor)
  2. A vibrant ecosystem needs to see commercial commitment from large players (and avoids lock-in)
  3. Having alternatives drives innovation and manages costs (because differentiation is still needed)
  4. Interlocking expertise is required (because no vendor has all the pieces)
  5. Customer needs are diverse and changing (since cloud is accelerating the rate of innovation)

Today, three major cloud players are standing together to demonstrate commitment to the community.  This announcement is a foundation for the OpenStack ecosystem.  In the next few months, I expect to see more and more collaborative announcements as the community proves the value of working together.

By aligning around an open platform, we collectively out flank previously dominant players who choose to go it alone.  The technology is promising; however, the power of OpenStack flows mainly from the ecosystem that we are building together.

OpenStack Design Conference Observations (plus IPv6 thread)

I’m not going to post OpenStack full conference summary because I spent more time talking 1 on 1 with partners and customers than participating in sessions.  Other members of the Dell team (@galthaus) did spend more time (I’ll see if he’ll post his notes).

I did lead an IPv6 unconference and those notes are below.

Overall, my observations from the conference are:

  • A constantly level of healthy debate.  For OpenStack to thrive, the community must be able to disagree, discuss and reach consensus.   I saw that going in nearly every session and hallway.  There were some pitched battles with forks and branches but no injuries.
  • Lots of adopters.  For a project that’s months old, there were lots of companies that were making plans to use OpenStack in some way.
  • Everyone was in a rush.  There’s been something of a log jam for decision making because the market is changing so fast companies seem to delay committing waiting for the “next big thing.”
  • Service Providers and implementers were out in force.
  • IPv6 is interesting to a limited audience, but consistently injected.

While IPv6 deserves more coverage here, I thought it would be worthwhile to at least preserve my notes/tweets from the IPv6 unconference discussion (To IP or not to IPv6? That will be the question.) at the OpenStack Design Summit.

NOTE: My tweets for this topic are notes, not my own experience/opinions

  • RT @opnstk_com_mgr #openstack unconference in camino real today < #IPv6 session going now – good size crowd
  • #NTT has IPv6 for VMs and tests for IPv6. If you set the mac, then you will know what the address will be.
  • it will be helpful to break out VMs to multiple networks – could have a VM on both IPv6 & IPv4
  • @zehicle @sjensen1850 (Dell) if IPv6 100% then may break infrastructure products – inside, easier to stay v4
    • you don’t want to paint yourself into a corner – IPv6 should not become your major feature requirement
  • typing IPv6 address not that hard to remember. DNS helps, but not required if you want to get to machines.
  • using IPv6 not hard – issue is the policy to do it. Until it’s forced. We need to find a path for DUAL operation.
  • chicken/egg problem. Our primary job is to make sure it works and is easy to adopt.
    • we are missing information on what options we have for transforms
  • where is the responsibility to do the translation? floating IP scheme needs to be worked out. IPv6 can make this easier.
  • idea, IPv6 should be the default. Fill gap with IPv4 as a Service? Floating needs NAT – v4aaS is LB/Proxy
  • unconference session was great! Good participation and ideas. Lots of opinions.

We had a hallway conversation after the unconference about what would force the switch.  In a character, it’s $.

Votes for IPv6 during the keynote (tweet: I’d like to hear from audience here if that’s important to them. RT to vote).  Retweeters:

Rackspace will balance control of OpenStack. It takes time & strong partners

Rick Clark’s post “Why I Left Rackspace and What About Openstack” (+ his softer post script) is part of a longer conversation that started when Rackspace acquired Anso Labs and was expanded with the resignation of Chris Kemp (NASA CTO & OpenStack #1 fanboy).

Building a community is a delicate balance: you need show leadership while you cultivate leadership.

Putting aside the context (resigning from Rackspace to join Cisco) of his post, I think that Rick’s comments do resonate with parts of the community.  OpenStack goverance became unbalanced when Anso became Rackspace.  The governance board formed at the Austin conference was dominated by a small number (2: NASA/Anso & Rackspace) of highly committed voices but there was no single master.

Considering OpenStack’s momentum, we are in a very good position to fix the single master problem.  However, it takes time.  While companies like Dell (my employer), NTT, Citrix, Cisco (Rick’s employer), and Microsoft are clearly investing in OpenStack, none have yet achieved NASA or Rackspace’s level of technical committment.

The challenge for Rackspace is to expand the OpenStack market and ecosystem so that partners are motivated to jump in more and more quickly.  If my experiences inside Dell are indicative of the broader community, Rackspace’s leadership makes it much easier for partners to increase their own commitment.  Like teaching my daughter to ride her bike, she needed to know that I was running next to her before she would pedal hard enough to balance by herself.

Like teaching bike riding – you can’t lead communities too hard or too lightly.

To build a community around OpenStack, we (the partners) need to stand up our own capability.  Until we have demonstrated more leadership, Rackspace must cultivate both a community and a market.  This is a challenging role to balance.  While the community wants distributed ownership, the market wants leadership.  Rick’s governance comments are evidence of this struggle and Rick’s move to Cisco is an indication of leadership diversification.

I believe that Rackspace is committed to distributed ownership – we, in the community, need to rise to the challenge!

OpenStack still needs strong leadership from Rackspace because the market needs someone to be accountable for releases and features.  That allows new partners to depend on someone to run beside them while the wobble their way along to independence.  As the community leaders stand up, we’ll see a balanced community emerge.  The challenge is on us to make that happen (and happen quickly).

Demo Redux: OpenStack installer SXSW demo of Chef + Crowbar

If you missed the OpenStack installer demo at Cloud Connect Event then you’ll have another chance to see us go from bare iron to provisioning VMs in under 30 minutes at SXSW on Monday 3/14 from 2-4 pm at Kung Fu Saloon.

Note: Rackspace rented out the Kung Fu Saloon all day Monday, and are doing various events — from live webinars to a Scoble tweetup to a happy hour and more VIP after hours event.

The demo will be orchestrated by Greg Althaus from my team at Dell.  Greg is the primary architect for Crowbar and responsible for some of it’s amazing capabilities including the Chef integrations, network discovery and rockin’ PXE state machine.  Dell Cloud Evanglist, Barton George, will also be on hand.

Of course, our friends from Opscode & Rackspace will be there too – this is Rackspace’s party (they are a Platinum SXSW sponsor)

More more information (outside of this blog, of course), check out http://www.Dell.com/OpenStack.

Dell to spin bare iron into OpenStack gold

I’m at the CloudConnect conference today supporting my team’s initial OpenStack foray.   Our announcement part of the Rackspace Cloud Builders announcement.

Tonight (3/8), we’re at the Rackspace Launch with a pony rack of servers (6 nodes) where we will run a LIVE DEMO of our cloud installer (codename “Crowbar”).  The initial offer includes my hyperscale white paper and our cloud foundation kit.

Interested in the details?  Here are background posts that talk about the Lean/Agile process we use, what is Crowbar, and my write up about hyperscale (“flat edge”) data centers.

Added 3/9: Links to articles about the release:

Here’s what Dell is saying about OpenStack on Dell.com/openstack:

Dell is one of the original partners in the OpenStack community, which has now grown to more than 50 companies and participants. To accelerate adoption of this powerful platform, Dell has worked to develop an effortless out-of-box OpenStack experience with:
  • Optimized PowerEdge™ C-based hardware configurations
  • A technical whitepaper that details the design of an OpenStack hyperscale cloud on PowerEdge C server technology
  • An OpenStack installer that allows bare metal deployment of OpenStack clouds in a few hours (vs. a manual installation period of several days)

Read more about the steps to design an OpenStack hyperscale cloud in a Dell technical whitepaper entitled “Bootstrapping OpenStack Clouds.”

Interested?  Contact OpenStack@Dell.com.

Unboxing OpenStack clouds with Crowbar and Chef [in just over 9,000 seconds! ]

I love elegant actionable user requirements so it’s no wonder that I’m excited about how simply we have defined the deliverable for project Crowbar**, our OpenStack cloud installer.

On-site, go from 6+ servers in boxes to a fully working OpenStack cloud before lunch.

That’s pretty simple!  Our goal was to completely eliminate confusion, learning time and risk in setting up an OpenStack cloud.  So if you want to try OpenStack then our installer will save you weeks of effort in figuring out what to order, how to set it up and, most critically, how to install all myriad of pieces and parts required.

That means that the instructions + automation must be able to:

  • Starting with servers in boxes and without external connectivity
  • Setup the BIOS and RAID on all systems
  • Identify the networking topology
  • Install the base operating systems
  • Discover the resources available
  • Select resources for deployment
  • Install the OpenStack infrastructure appropriately on those resources
  • Validate the system is operating correctly
  • Deploy a reference application
  • In under 4 hours (or 14400 seconds).

That’s a lot of important and (normally) painful work!

Crowbar does not do all this lifting alone.  It is really an extension of Opscode’s Chef Server – an already awesome deployment management product.  The OpenStack deployment scripts that we include are collaborations between Dell, Opscode (@MattRay), and RackSpace (@JordanRinke, Wayne Wallis (@waynewalls)
& Jason Cannavale).

There are two critical points to understand about our OpenStack installer:

  1. It’s an open source collaboration* using proven tools (centrally Chef)
  2. It delivers an operational model to cloud management (really a DevOps model)

One of my team’s significant lessons learned about installing clouds is that current clouds are more about effective operations than software features.  We believe that helping customers succeed with OpenStack should focus more heavily on helping you become operationally capable of running a hyperscale system than on adding lots of features to the current code base.

That is why our cloud installer delivers a complete operational environment.

I believe that the heart of this environment must be a strong automated deployment system.  This translates into a core operational model for hyperscale cloud success.  The operational model says that

  1. Individual nodes are interchangeable (can be easily reimaged)
  2. Automation controls the configuration of each node
  3. Effort is invested to make the system deployment highly repeatable
  4. System selection favors general purpose (80% case)
  5. Exceptions should really be exceptions

Based on this model, I expect that cloud operators may rebuild their entire infrastructure on a weekly (even daily!) basis during the pre-production phase while your Ops team works to get their automation into a predictable and repeatable state.  This state provides a stable foundation for expansion.

My experience with Crowbar reinforces this attitude.  We started making choices that delivered a smooth out-of-box experience and then quickly learned that we had built something more powerful than an installer.  It was the concept that you could build and then rebuild your cloud in the time it takes to get a triple caramel mochachino.

Don’t believe me?  I’ve got a system with your name on it just waiting in the warehouse.

*Open source note: Dell has committed to open source release (Apache 2) the Crowbar code base as part of our ongoing engagement in the OpenStack community.

**Crowbar naming history.  The original code name for this project was offered by Greg Althaus as “you can name it purple fuzzy bunny for all I care.”  While excellent as a mascot, it was cumbersome to say quickly.  Crowbar was picked up as a code name because it is 1) easy to say, 2) used for unboxing things, 3) a powerful and fast tool and 4) the item you start with in a popular FPS.  Once properly equipped, our bunny (I call him “Mesa”) was ready to hit IT.

OpenStack Swift Demo (in a browser)

I’m working on mini-demo project for OpenStack Swift.  To keep things very simple and easy to understand, I decided that the whole demo would work in JavaScript in the browser.  I also choose to use RackSpace’s CloudFiles as a Swift target for testing since they have the same API are are universally available (unlike my lab systems).

One advantage of this approach is that FireBug makes it very nice to debug and check the activity of the code.  Unfortunately, FireBug also seems to eat the headers that I need.  *Let me phrase that in a google friend way so that someone else will not loose the 2 hours I just lost*

“XmlHttpRequest setRequestHeader FireFox Not Respected when using FireBug”

It works great in Safari. So onward and upward.  So far, I’ve got step #1 ready – getting the authorization token back from the cloud site.

Here’s the HTML page (you need jQuery too).  Basically, it uses the username and key from the inputs to set “x-auth-user” and “x-auth-key” header attributes.  These attributes will allow Swift to return a token that you can use on future requests when you want to do useful work.

<!DOCTYPE html>

<html>

<head>

<title>Dell Swift Demo [0.0]</title>

<script src=”jquery.js” type=”text/javascript”></script>

<script type=”text/javascript” charset=”utf-8″>

var xmlhttp = null;

function swiftLogin() {

var usr = $(‘input:text[name=usr]’).val();

var key = $(‘input:text[name=key]’).val();

// code for IE7+, Firefox, Chrome, Opera, Safari (UR SOL IE<7)

xmlhttp = new XMLHttpRequest();

xmlhttp.onreadystatechange=function() //callback

{

if (xmlhttp.readyState==2)

{

$(‘#status’).replaceWith(xmlhttp.getResponseHeader(“X-Auth-Token”));

}

}

xmlhttp.open(‘GET’,’https://auth.api.rackspacecloud.com/v1.0&#8242;, true);

xmlhttp.setRequestHeader(‘Host’, ‘auth.api.rackspacecloud.com’);

xmlhttp.setRequestHeader(‘X-Auth-User’, usr);

xmlhttp.setRequestHeader(‘X-Auth-Key’, key);

xmlhttp.send();

}

</script>

</head>

<body>

<div id=”credentials”>

<fieldset id=”credentials” class=””>

<legend>Swift Login</legend>

<label for=”user”>User: </label><input type=”text” name=”usr” value=”user” id=”user”>

<label for=”key”>Key: </label><input type=”text” name=”key” value=”key” id=”key”>

<input type=”button” name=”Login” value=”login” id=”Login” onclick=”swiftLogin();”>

</fieldset>

</div>

<div id=”status”>[pending]</div>

<div id=”footer”>Time?</div>

<script type=”text/javascript”>

$(‘#footer’).replaceWith((new Date).toString());

swiftLogin();

</script>

</body>

</html>