Videos about Crowbar, CloudOps, and Dell OpenStack Cloud

I’m not usually a big fan of launch videos (too much markitecture); however, these turned out to be nice and meaty.  The meaty part explains why it looks like I’m about to eat a big sandwich in the last video.  yum!

  • What is Crowbar: Dell Crowbar Software Overview  

Continue reading

Crowbar source released, includes OpenStack Cloud install

I’m delighted to announce (official version) that my team at Dell has opened the Crowbar source under the Apache 2 license. This action is part of the broader Dell OpenStack Cloud Solution which includes OpenStack install packages, Crowbar, reference hardware architectures, and services/consulting to support deployments.

There are two important components to this news:

  1. Dell is officially offering our OpenStack Solution and helping advance the community’s ability to implement OpenStack quickly and consistently.
  2. Dell is releasing the Crowbar code (which is included in the solution) as open source.

Both are significant items; however, my focus here is on the Crowbar release.

Crowbar started as a Dell OpenStack installer project and then grew beyond that in scope.  Now it can be extended to work with other vendors’ kits and other solutions bits.

We are contributing Crowbar to the community because we believe that everyone benefits by sharing in the operational practices that Crowbar embodies. These are rooted in Opscsode Chef (which Crowbar tightly integrates with) and the cloud & hyper-scale proven DevOps practices that are reflected in our deployment model.

Where to get it?

What’s included?

  • A comprehensive set of barclamps to set up an OpenStack cloud.
  • Crowbar UI and Remote APIs to make it easy to set up your cloud
  • Automated testing scripts for community members doing continuous integration with OpenStack.
  • Build scripts so you can create your own Crowbar install ISO
  • Switch discovery so you can create Chef Cookbooks that are network aware.
  • Open source Chef server that powers much of Crowar’s functionality

What’s not included?

  • Non-open source license components (BIOS+RAID config) that we could not distribute under the Apache 2 license.  We are working to address this and include them in our release.  They are available in the Dell Licensed version of Crowbar.
  • Dell Branded Components (skin + overview page).   Crowbar has an OpenSource skin with identical functionality.
  • Pre-built ISOs with install images (you must download the open source components yourself, we cannot redistribute them to you as a package)

Important notes:

  • Crowbar uses Chef Server as its database and relies on cookbooks for node deployments.  It is installed (using Chef Solo) automatically as part of the Crowbar install.
  • Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as barclamps.
  • Each barclamp has its own Chef configuration, UI sub-component, deployment configuration, and documentation.

On the project roadmap:

  • Hadoop support
  • Additional operating system support (specifically RHEL)
  • Barclamp version repository
  • Network configuration
  • We’d like suggestions!  Please comment!

Sites for more information: Joseph George, Barton George (launch day), Dell

Crowbar modules (aka barclamps) perform many functions and enable multi-vendor hardware

10/18 Update:

More recent information about Barclamps can be found at http://robhirschfeld.com/2011/09/14/details-of-crowbar-changes/.  We’ve also created videos showing how you can create your own barclamps.

Original Post

Just after we’d started deep Crowbar development, Andi Abes, Paul Webster and Victor Lowther joined the Dell Crowbar+OpenStack team.  They immediately started to dig into our Swift, BIOS/RAID, and Network components.  They also started to bump into each other in our original code base.  It quickly became apparent that we needed to modularize Crowbar.

Restructuring Crowbar into modules has proved essential as a method for safe community collaboration.

Greg Althaus coined the name “barclamps” during the modularization rearchitecture.  A barclamp is a class extension of the Crowbar ServiceObject that allows Crowbar to identify the Chef components used by the barclam

p (name p

attern in Chef is bc-template-[barclamp]) and provides capabilities that are specific to each barclamp.

  • In the simplest case, the barclamp is a minimal wrapper that just provides naming hooks for your Chef cookbooks.  This makes it very easily to adapt existing Chef work to work with Crowbar.
  • In more complex cases, the barclamp will help identity how nodes are allocated, interacts with other barclamps, extends the provisioner state machine and provides custom user interfaces.
  • In most cases, the barclamp’s generic integration and UI are sufficient.
Initially, barclamps were entirely exposed via REST using the ServiceObject.  We quickly wrapped those into a CLI for our continuous integration system.  Lately, we’ve expressed them in the user interface.
At launch, you’ll find all but two in the open source repository.  Unfortunately, we were not able to include BIOS and RAID barclamps in the open version because they use licensed components – we are working to correct this.  They are available in the Dell licensed version.
When looking at the barclamps, it is critical to understand that even the most core Crowbar functionality is expressed as a barclamp.
This exposure of Crowbar internals as barclamps is important because it
  1. helps modularize the code and
  2. reflects the deep integration between Chef and Crowbar.

Consequently, the core logic of the state machine, networking configuration, and provisioning are all exposed in barclamps.  This makes it possible to modify and extend the most basic Crowbar operations; however, there are currently no guards against breaking these barclamps either!

The following list includes all the barclamps that we’ve created for Crowbar.
Barclamp   Function  Included
Crowbar The roles and recipes to set up the barclamp framework.  Yes
Deployer Initial classification system for the Crowbar environment (aka the state machine)  Yes
Provisioner The roles and recipes to set up the provisioning server and a base environment for all nodes  Yes
Network Instantiates network interfaces on the crowbar managed systems. Also manages the address pool.  Yes
NTP Common NTP service for the cluster. An NTP server or servers can be specified and all other nodes will be clients of them.  Yes
DNS manages the DNS subsystem for the cluste  Yes
Logging centralized logging system based on syslog  Yes
IPMI Integrates with IP management to allow direct hardware control bypassing the operating system.  Yes
RAID LSI Licensed components.  Cannot be included in open source release at this time.  No
BIOS
PowerEdge C series: Dell License component.  Cannot be included in open source release at this time.  No
Ganglia Optional: a common Ganglia service for the cluster that can be used by other barclamps  Yes
Nagios Optional : common monitoring service for the cluster that can be used by other barclamps  Yes
Nova OpenStack: installs and configures the Openstack Nova (Cactus Release) component. It relies upon the network and glance barclamps for normal operation.  Yes
Swift OpenStack: part of Openstack (Cactus Release) , and provides a distributed blob storage  Yes
Glance OpenStack: Glance service (Cactus Release, Nova image management) for the cloud  Yes
Test provides a shell for writing tests against  Yes

OpenStack Crowbar User Guide: explaining how barclamps get deployed

My whole team is working feverishly on the final touches of Crowbar before we turn over the keys.  We’re putting it through a complete release cycle (extensive QA, customer pilots, documentation, etc) because internal Dell consumers are expecting that level of finish. 

For those in the community eagerly waiting to see the code, I hope you like the extra polish (for example: I18N, user & deployment guides, bundled continuous integration scripts, and months of testing).

RUMOR CONTROL NOTE: Crowbar is NOT limited to deployments on Dell products!!  Our BIOS and RAID barclamps are, of course, targeted and licensed for Dell customers.  The OpenStack and other barclamps will work on any gear that can run Chef Client.

Tonight I was working on the user guide and thought I would share the graphic and text describing how a barclamp gets deployed.

The figure shows the entire of a barclamp within the Crowbar user interface.  A Barclamp defines the capability for a service but cannot be deployed.  To deploy a barclamp, you must create a Proposal.  Once the proposal is created, you must selection nodes to operate on.  As discussed in the next sections, you may also edit the Proposal’s attributes as needed.  

Applying the Proposal tells Crowbar to deploy the proposal onto the nodes.  While deploying, nodes return to the Ready state when deployment is completed.  Once a proposal has become an Active Role, you cannot edit it.  You must delete the Role and repeat the Apply process

Cybera’s OpenStack efforts (includes Dell xrefs)

Cybera's Everett ToewsIt’s awesome to see new deployments of OpenStack so I wanted to point out Cybera’s post about their OpenStack efforts!

Everett Toews does a nice job talking about the rationale for their decisions including some analysis of their hardware and vendor selections.  Of course, I’m also happy to have them posting links back to my Dell team’s white paper and content. 

I wanted to highlight one point that Everett makes:

“Is this the best mix of hardware possible for OpenStack? As always the answer is, “It depends.” It depends primarily on your the use cases for your cloud. We think we got a good mix of hardware but time will truely tell if it was the best mix possible for DAIR.”

I strongly agree, we (Dell) are still recommending starting with a smaller set general purpose hardware config that can be easily repurposed.  Once you’ve figured out how your application maps into OpenStack then we’ll be ready to work togther to tune that order for 1000s of servers.

The OpenStack Rocket: How Citrix, Dell & Rackspace collaboration propels OpenStack

OpenStack has grown amazingly and picked up serious corporate support in its first year.  Understanding how helps to explain why the initiative has legs and where adopters should invest.  While OpenStack has picked up a lot of industry partners, early participation by Dell and Citrix have been important to a meteoric trajectory.

So why are we (Dell) working so hard to light a fire under OpenStack?

To explain OpenStack support and momentum, we have to start with a self-reflective fact: Dell (my employer), Citrix and Rackspace are seeking to gain the dominate positions in their respective areas of “cloud.”  Individually, we have competitors in more entrenched positions for cloud silos in solutions, ecosystem, and hypervisor; however, our competitors are not acting in concertto maintain their position.   

OpenStack offers an opportunity for companies trying to gain market share to leverage the strengths of partners against their competitors.

The surprising aspect of OpenStack is how much better this collaboration is working in practice than in theory.  This, dare I say it, synergistic benefit comes to OpenStack from all the partners (Dell, Citrix, Rackspace and others) working together because:

  1. Real scale clouds are big and complex with a lot of moving pieces (too big for a single vendor)
  2. A vibrant ecosystem needs to see commercial commitment from large players (and avoids lock-in)
  3. Having alternatives drives innovation and manages costs (because differentiation is still needed)
  4. Interlocking expertise is required (because no vendor has all the pieces)
  5. Customer needs are diverse and changing (since cloud is accelerating the rate of innovation)

Today, three major cloud players are standing together to demonstrate commitment to the community.  This announcement is a foundation for the OpenStack ecosystem.  In the next few months, I expect to see more and more collaborative announcements as the community proves the value of working together.

By aligning around an open platform, we collectively out flank previously dominant players who choose to go it alone.  The technology is promising; however, the power of OpenStack flows mainly from the ecosystem that we are building together.

OpenStack discussion at 5/19 Central Texas Linux Users Group (CTLUG ATX)

image

Greg Althaus (@glathaus) and I will be leading a discussion about OpenStack at the May CTLUG  on 5/19 at 7pm.  The location is Mangia Pizza on Burnet and Duval (In the strip mall where Taco Deli is).

We’ll talk about how OpenStack works, where we see it going, and what Dell is doing to participate in the community.

OpenStack should be very interesting to the CTLUG because of the technologies being used AND way that the community is engaged in helping craft the software.