So you want to create a Crowbar barclamp? Here’s what you have to know…

My team at Dell has created many barclamps to support OpenStack and Hadoop. One major objective of our recent modularization refactoring was to make it easier to the community to contribute barclamps. The Crowbar CloudOps approach is to build up a full cloud deployment using layers. So each barclamp represents a component of the overall deployment.

Note 9/21/12: Added Video Post showing the steps below.

Note 9/7/13: Reference to Crowbar Docs.

A barclamp is a deployment module that is imported from its own code repository into the Crowbar framework. A barclamp cannot operate without Crowbar, but you do not have to create a unique build of Crowbar in order to create a barclamp.

The first thing to know about barclamps is that most of the work (80%!) is building your Chef cookbooks. If you don’t have a cookbook that deploys your application then stop here and work on that first.

The second thing to know about barclamps is that there are a lot of them that you can study for examples. Check out the Glance barclamp if you have a single server deployment, Nagios if you have a service that needs to be integrated into every node, Nova if you have a complex multi-component system and Provisioner if you want to impact core Crowbar functionality.

We’ve done a lot of work to make it easy to create and install a stub barclamp. Our experience is that building a barclamp is a highly iterative exercise with a lot of testing. Luckily, Crowbar’s primary mission is to help you brush, rinse and repeat. From there, you can customize and extend your barclamp to deploy your application’s full untamed glory.

Before you try to create a new barclamp, you must install Crowbar.

Creating a barclamp

The following steps use the barclamp_model that included under /dell/opt and is described below.

  1. Figure out the name of your barclamp. I’m naming our example “foo barclamp”
    1. Barclamps must have unique names.
    2. Do not use spaces or hyphens.
  2. From the Crowbar server, become the super admin: sudo -i
  3. Create a directory for your barclamp: mkdir /barclamps
  4. Run the barclamp create script: /opt/dell/bin/barclamp_create.rb foo “Zehicle” /barclamps
    1. “foo” is our barclamp name [required]
    2. “Zehicle” is my company name for the copyright information [default is Dell]
    3. “/barclamps” is the path where we are putting the barclamp [default is /opt/dell/barclamps]
    4. Result will be a populated barclamp. In this example: /barclamps/foo

That’s it! If you want to plan ahead then you could use an initialized git repo as the target.

Reminder: In building your barclamp, you’ll need to learn about Chef, how Crowbar extends cookbooks and how barclamps interact. That’s beyond the scope of this post.

Importing a barclamp

Once you created a barclamp, you can import the barclamp into Crowbar & Chef.

Assuming that you already created the foo barclamp in /barclamps, here are the steps:

  1. From the Crowbar server, become the super admin: sudo –i
  2. Run the barclamp install script: /opt/dell/bin/barclamp_install /barclamps/foo
    1. “/barclamps/foo” is the path to your barclamp. If could be anything!
    2. The core barclamps are in /opt/dell/barclamps.
    3. In a vm, you could mount a shared folder to access the barclamp (e.g.: /mnt/hgfs/barclamps)

Your barclamp should now show up in the Crowbar UI! You can also see it in Chef under the Crowbar databag.

While barclamps are generally safe to install multiple times, you can uninstall a barclamp using “barclamp_uninstall.rb /path/to/barclamp”

Barclamp layout

A barclamp has the following core components:

  • crowbar.yml configuration file (documented below)
  • README.txt file (optional, recommended)
  • chef directory containing
    • Cookbooks directory with Chef cookbooks
    • Data_bags directory with Crowbar configuration files
    • Roles directory with Chef roles used by the cookbooks and data_bags
  • crowbar_framework directory
    • app directory with Crowbar model, controller, and view code
    • other optional directories to add components needed by the UI such as images

The barclamp_model has a functional layout that covers most configuration requirements. The string ==BC-MODEL== indicates places where the name of the barclamp must be substituted. It is critical to understand that the name of the barclamp is embedded into the barclamp path and file names! This is needed to avoid file collisions when the barclamp is imported.

Crowbar.yml

The crowbar.yml file is a required configuration file that gives direction to the installer. The file has the following components:

barclamp:
  name: name of your barclamp (required, do not use space or hyphens) 
  display: pretty name of your barclamp [optional for now]
  description: information about your barclamp your barclamp [optional for now]
  version: what you want to consider for versioning [optiona] 
crowbar:
  layout: 1 (use the # one. This is required because it tells the installer what to expect inside your barclamp)
  order: 1000 (if installing multiple barclamps in one pass, order tells the installer the, well, order) 
nav: (remove the nav section, advanced users only) 
locale_additions: (you must add UI localizations here if you have any custom UI components) 
  en: (entries in this file map directly to entries in the config/locales/en.yml file and are added during install) 

Crowbar modularized: latest changes that make clouds even easier to create, update, and maintain

In the last week, my team at Dell completed a major refactoring of Crowbar that significantly improves our ability to bring in community contributions and field customizations.  Today, we merged it into Crowbar’s public repo(s).

From the very first versions, our objective for Crowbar was to create the fastest and most reliable cloud deployments. Along the way, we realized Crowbar’s true potential lay in embracing DevOps as an operational model for maintaining clouds. That meant building up cloud deployments in layers from pieces that we call barclamps (extensions of Chef cookbooks). Our first version, centered on OpenStack Cactus, leveraged barclamps but was still created as a single system. This unified system was a huge step forward in cloud deployments, but did not live up to our CloudOps vision of continuous delivery.

In this version, each Crowbar barclamp is an independent delivery unit that can be integrated before, while or after installing Crowbar.

The core of the change is each barclamp, including the most core ones, are stored in independent code repositories. Putting the code into distinct repos means that each barclamp can have its own life cycle, its own maintainer site and its own dependency tree. This modularization allows customers to manage their Crowbar deployments with a very fine brush: they may choose to customize parts of the system, they could lock components to specific tag and they can bring in barclamps from other vendors.

While the core barclamps are automatically integrated into the Crowbar build using git submodules; other barclamps are installed into the system as needed. This allows you to pull in the suite of OpenStack barclamps at build time or to wait until your Crowbar system is running before installing. Once you install a barclamp, you are able to retrieve an updated barclamp and reapply it to the system.

This feature gives you the ability to 1) choose exactly what you want to include and 2) perform field updates to a live Crowbar system.

Let’s look at some examples:

  1. The Cloud Foundry barclamp can be sourced Cloud Foundry instead of bundled into the Crowbar repository. This allows the team working on the cloud application to take ownership for their own deployment. As a continuous delivery proponent, I believe strongly that the development team should be responsible for ensuring that their code is deployable (refer to my OpenStack “Deployer API” blue print attempting to codify this).
  2. DreamHost, maintainers of Ceph Storage, can maintain their own local barclamp repos for OpenStack that are cloned from our community Swift barclamp. This allows them to innovate and customize OpenStack deployments for their business and choose which updates to merge back to the community.
  3. Rackspace Cloud Builders can work on the most leading edge OpenStack features and maintaining workable deployments on branches. As the code stabilizes, they simply merge in their changes.
  4. Dell BIOS and RAID barclamps only support the PowerEdge C line today. When we offer PowerEdge R support, you will be able to install or update the barclamps to add that capability. If another hardware vendor creates a barclamp for their hardware then you can install that into your existing system.

I believe that these changes to Crowbar are a huge step forwards on our journey of creating a community supportable Open Operations framework. I hope that you are as excited as I am about these changes.

I encourage you to take the first step by trying out Crowbar and, ultimately, writing your own barclamps.

Post Scripts:

  • In addition to the modularization, the updated code includes RHEL as a deployment platform. At present, you must choose to be either RHEL or Ubuntu at build time.
  • We have enhanced the network barclamp to describe connections as more abstract connections, called conduits, between nodes. This is a powerful change, but requires some understanding before you start making changes.
  • We have only begun testing the change as of 9/12, we expect the system to be fully stabilized by 10/3. If you are not willing to deal with bugs then I recommend building the Crowbar “v1.0” tag (or using the ISOs from our July launch).

Cloudcast interview with Dell Cloud Solutions Team (quotes with time stamps)

Lloyds BarbershopTheCloudcast.net – Thank you for such a great series of questions. Wow, nearly 36+ minutes of cloudicious interview about the work my team at Dell is doing!

Thanks to our hosts for putting together a great series! They are:

Highlights from Episode 16: Dell, Dude you’re getting a cloud

  • 3:40 JBG “we are listening to our customers tell us what they want to accomplish”
  • 4:40 RAH “humility is part of [what we’re] doing … cloud is about learning and collaboration”
  • 6:40 RAH “OpenStack filled a niche. It was the first open source community cloud. … Not just open source, its open community.”
  • 7:15 RAH “We’re beyond critical mass. We’re seeing acceleration… we are transitioning into a community development.”
  • 7:30 RAH “It’s accelerating. It happening so fast.”
  • 8:00 RAH “We felt it was really important for people to be able to use it. We felt that it was important to get away from just people developing into people using. “
  • 8:57 – RAH “Cloud is not just one thing. You have to have all the pieces.”
  • 10: 15 – RAH “Cloud is always ready, never finished”
  • 10:50 – RAH “OpenStack is an alternative to public cloud including hosting providers seeking to offer their own cloud”
  • 12:40 – AD “Dell has been in the Big data space for many years now”
  • 20:15 – JBG “There’s a legacy of great partnerships that we leverage”
  • 20:48 – JBG “Conflicts have not come up because we are focused on the customer”
  • 21:30 – RAH “Shout out to Greg Althaus for solving these problems in such an elegant way. And we rewrote it 3 times”
  • 22:02 – RAH “Crowbar started from our frustration of bringing up a cloud quickly … so we took a DevOps approach.”
  • 22:41 – RAH “You had to have a system view AND a boot strapping view simultaneously”
  • 23:50 – JBG “Crowbar was born out of necessity because we were setting up and blowing away our clouds over and over and over again. “
  • 24:40 – JBG “We realized there were not many people thinking about all the pieces before OpenStack was installed”
  • 25:20 – RAH “We don’t think customers have all the answers before we show up. This is not unique to OpenStack.”
  • 28:20 – JBG “We’re seeing the community pick up Crowbar as a way to deploy”

Big Questions? Big Answers with Dell BigData solution (plus Crowbar gets RHEL)

In my enthusiasm for all things Dell + OpenStack, I have neglected to talk about my team’s interesting Big Data work with Apache Hadoop.  Hadoop is a suite of open source projects for analyzing large data sets of unstructured data.  Initially, Hadoop centered around use of the map-reduce algorithm; however, it’s grown way beyond that as the community has worked to solve problems related to data storage, discovery, and scheduling.

Big Data clouds are well suited to my team because the model (non-redundant/cloud) and scale (hyper) of their deployments.  It should be no surprise that builders of analysis clouds have the same goals (maximizing operational ROI per compute unit) as builders of other types of clouds.

Our Hadoop solution relies on the same core principles (CloudOps) and technologies (Crowbar) as our OpenStack solution.  Like our other cloud solutions, we are working closely with a proven leader: Cloudera.  Now that we’ve formally announced our solution and partnership, I can talk a about what we’re doing on the Big Data front.

One extra thing that I’m proud to announce, we’ll be adding Red Hat Enterprise Linux (RHEL) support to Crowbar to support our Hadoop solution.  This support is not just at the node level: we are making Crowbar admin run on either platform too!  This is significant for two reasons:

  1. It expands the number of platforms and support options for Crowbar users
  2. It provides the framework to support more varieties of node operating environment (e.g.: XenServer, BSD, DRDOS, etc)

For more information, check out: