Cote & Rob interview: Crowbar+OpenStack Summit/Conference Reflections (40 mins)

I’m working on a larger post about the OpenStack Summit around API Implementation vs. Specification. You can have a preview of that AND A LOT OF OTHER STUFF (OpenStack, Crowbar, lunch) in this 40 minute interview w/ Michael Cote.

Setting: Dell World
Interview w/ @Cote at the Hilton Hotel Lobby on 6th street in Austin.

I know that Cote’s post does not have a time marker for easy navigation; however, I added them to help guide your navigation in the interview (link for audio) if you want to jump around.

  • 0:00 Introductions
  • 1:00 OpenStack
    • 1:00 Essex Conference – what is it, naming conventions
    • 2:45 Diablo is adding projects from incubation (Keystone, Dashboard,Quantum,
    • 5:30 OpenStack vs. Amazon – “OpenStack has ambitions.” We see it as a “platform for innovation.”
    • 6:30 OpenStack is a competitor for Amazon. It implements the EC2 APIs.
    • 7:30 How are people managing the evolving nature?
    • 8:20 We’re going to see OpenStack in production for the next release based on what we see in our deal flow.
    • 9:00 Every user that comes on adds momentum
    • 9:30 Rackspace setting up the OpenStack foundation is a reflection of the speed of adoption
    • 11:15 Our message is “we’re doing it, we’re in the field.” We are very hands on
    • 11:15 We chose early on to focus on helping deployment to help drive adoption
    • 12:00 “Our first test for partners is: Are you contributing back to the community?”
    • 12:44 The community told us “if you are participating then you are going to open source.” Our commits for OpenStack are live and in the open on our github.
    • 13:40 Why Github? We’re happy with it.
    • 14:20 OpenStack is using Gerrit because they have a gated trunk. They are migrated to Github
    • 15:20 APIs have been a big topic for OpenStack
    • 16:00 Do you track who is forking and following? Yes. We also have a listserv. We are trying to do a better job managing the Crowbar community. We know we need to do a better job.
    • 17:30 OpenStack is defined by its Implementation. That’s “an effective way to move the project forward quickly;” however, we’re getting to a point where people want to use alternate implementations.
    • 19:20 Implementation vs Specification is like the SOAP vs REST debate
    • 20:05 This is something the community needs to wrestle with
    • 21:45 Specification would allow the efforts to scale. The more people consume the API, the more people care about how it operates
    • 22:30 “Bugs can become the API”
    • 23:10 Asia and Europe are very active. We are seeing a ton of activity overseas.
  • 23:30 Crowbar
    • 24:00 Crowbar arose out of our need to deploy cloud software regardless of customer infrastructure
    • 24:45 We would show up and the customer needed all this cloud infrastructure. We created Crowbar because we always needed this
    • 26:00 We extended Chef because we had to do the initial bring-up including BIOS and RAID
    • 26:45 We added a state machine and an orchestration layer
    • 27:45 Updating the system is a huge component. Every month you may be upgrading the infrastructure!
    • 28:30 In our lab, we build whole clouds multiple times a day
    • 29:45 Crowbar is the “cloud unboxer”
    • 30:00 We modularized Crowbar with barclamps. Hadoop and OpenStack are a series of barclamps. Over 5 for each
    • 31:00 Barclamps are applied as layers. We are using that as a term to define DevOps
    • 31:15 We are using Crowbar to help message that we understand DevOps
    • 31:45 Soup vs Sandwich analogy – Images are like soup while DevOps is like a sandwich.
    • 32:45 If you don’t want something in a 1000 server deployment, DevOps lets you make a small change. Gives you flexibility.
    • 33:45 We added Cloud Foundry
    • 34:00 We’ve made it so easy with barclamps that partners are coming to us with ideas for barclamps. It’s like “changing the meat for the sandwich.”
    • 34:30 Dreamhost Ceph team created a barclamp and was actually running a majority of the Crowbar demos at the OpenStack conference
  • 35:25 What’s the future for Crowbar?
    • 35:30 More aspects of the infrastructure as open source
    • 35:45 More Hardware
    • 36:00 Multiple operating systems at the same time (XenServer, ESX, etc)
    • 36:30 Larger scale
    • 36:50 More types of infrastructure: storage & network
    • 37:40 Scalr shout out
    • 38:00 We know we need to collaborate more with our community
    • 38:30 The first step is to download it and try. Read my blog and sign up for the list serve
    • 39:00 CROWBAR IS NOT DELL SPECIFIC – we are working with people who want to create support for other vendor’s hardware. This benefits Dell.
    • 39:40 We don’t pretend that our customers are single vendor


Crowbar OpenStack deployment video (15 mins): Diablo + Keystone + Dashboard

This week at the OpenStack Design Summit and Conference in Boston, my team unveiled the Diablo+ Crowbar deployments. The OpenStack deployment that’s included with Crowbar reflects a collaborative effort between Dell, Opscode, and Rackspace and pulls packages from the Rackspace repository. It was important for us to use the Rackspace repos so that we could include integrated Keystone and Dashboard components that were omitted from the Diablo (current) release. Our decision to include these Essex (coming) components is based on customer feedback.

Since some of you cannot make it to the show and see the demo in person, we’ve captured it as a video for your enjoyment. The OpenStack deployment is available in our open source distribution. We are currently in QA for the overall solution so expect additional refinement as we progress towards our next OpenStack solution release.

REMINDER: Dell Hardware is NOT required to use Crowbar for OpenStack.  The open source version has everything you need – the BIOS and RAID barclamps are optional (but handy).

Shout, chat and whisper with Dell at OpenStack Design Summit & Conference

My team at Dell has been very (very) busy delivering a lot of great materials for the Fall 2011 OpenStack Design Summit & Conference in Boston MA next week.

Our motto for this conference is “DOING IS DOING” or, perhaps, “DIABLO IS DOING.”

You can count on Dell to be walking the walk with deliverables that advance OpenStack.  In fact, you can watch what we’re doing because it’s posted live as we work with the community to build it.

First, we’ll have our Crowbar demo rack showcasing LIVE MULTI-NODE DIABLO DEPLOYMENTS and some IMPORTANT FEATURE AND COMMUNITY ADDITIONS.  No spoilers here – you’ll have to come by.  Of course, it’s in the git hub too, but we’ve put a bow on it.

Second, there’s a DEPLOYMENT BLUE PRINT discussion about getting better interlocks between OpenStack development and deployment.  We really need to reduce the pain and lag between adding great features and using those features.

Next, we’ve got a limited audience CONCEPT SNEAK PEEK for something from our labs that we think is very interesting and we’d like to get input about.  Unfortunately, we’re very limited with space & time for this whisper session so you’ll need to contact OpenStack@Dell.com to request an invitation.

Finally, at the Conference, you can see OUR TEAM IN ACTION:

  • Thurs 11:30 – Dell Keynote by John Igoe
  • Thurs 3:30 – Private Cloud Panel w/ Rob Hirschfeld
  • Thurs 4:30 – Hardware Infrastructure w/ Rob Hirschfeld & Greg Althaus
  • Friday 11:00 – Deployments w/ Greg Althaus
  • Friday 3:15 – Crowbar!! w/ Scott Jensen (yes, he does it with the !!)
  • Friday 4:15 – KVM & OpenStack

More conference posts: JB Gorge & Barton George.

Dell Crowbar to deploy OpenStack Diablo Cloud

Direction in the Cloud

Photo by JB George

This week, some of the Crowbar/Dell OpenStack-Powered Cloud team, plus Matt Ray from Opscode, have been working with our partners at Rackspace in San Antonio (see Opscode post about collaboration). Our target is to have Crowbar deliver a core Diablo deployment by the October 2011 design conference (sponsored in part by Dell). This is a collaborative effort and we invite community participation – we are trying to be open and communicative (via the Crowbar listserv) while also respecting that there is a mountain of work if we are to meet deadlines.

We are doing the work in the open on the Crowbar Github so you have access to the very latest capabilities and it also means that the head the Crowbar may be unstable while we add capabilities. We feel like this is an important trade off because it allows us to keep up with the rapid pace of development in OpenStack (and other projects). This is the motivation for the recent modularization work and will continue to be a feature driver for Crowbar enhancements because it allows Crowbar users to easily bring in updated bits.

 

Building Crowbar post-modularization (15 minute how-to video)

Note: I’m putting build ISOs and Sledgehammer TARs on crowbar.zehicle.com if you don’t want to follow these steps then download the ISO. We are updating the ISO daily, so don’t assume that you have that latest!

To build Crowbar, you need a Linux machine and access to the internet. The video shows how you can use an Ubuntu 10.10 Rackspace Cloud Server.  We build Crowbar inside our firewall on our PCs too. No matter how you do it, Crowbar is full of fuzzily delicious cloud bits.

For up-to-date instructions, see the Crowbar wiki Build ISO page.


So you want to create a Crowbar barclamp? Here’s what you have to know…

My team at Dell has created many barclamps to support OpenStack and Hadoop. One major objective of our recent modularization refactoring was to make it easier to the community to contribute barclamps. The Crowbar CloudOps approach is to build up a full cloud deployment using layers. So each barclamp represents a component of the overall deployment.

Note 9/21/12: Added Video Post showing the steps below.

Note 9/7/13: Reference to Crowbar Docs.

A barclamp is a deployment module that is imported from its own code repository into the Crowbar framework. A barclamp cannot operate without Crowbar, but you do not have to create a unique build of Crowbar in order to create a barclamp.

The first thing to know about barclamps is that most of the work (80%!) is building your Chef cookbooks. If you don’t have a cookbook that deploys your application then stop here and work on that first.

The second thing to know about barclamps is that there are a lot of them that you can study for examples. Check out the Glance barclamp if you have a single server deployment, Nagios if you have a service that needs to be integrated into every node, Nova if you have a complex multi-component system and Provisioner if you want to impact core Crowbar functionality.

We’ve done a lot of work to make it easy to create and install a stub barclamp. Our experience is that building a barclamp is a highly iterative exercise with a lot of testing. Luckily, Crowbar’s primary mission is to help you brush, rinse and repeat. From there, you can customize and extend your barclamp to deploy your application’s full untamed glory.

Before you try to create a new barclamp, you must install Crowbar.

Creating a barclamp

The following steps use the barclamp_model that included under /dell/opt and is described below.

  1. Figure out the name of your barclamp. I’m naming our example “foo barclamp”
    1. Barclamps must have unique names.
    2. Do not use spaces or hyphens.
  2. From the Crowbar server, become the super admin: sudo -i
  3. Create a directory for your barclamp: mkdir /barclamps
  4. Run the barclamp create script: /opt/dell/bin/barclamp_create.rb foo “Zehicle” /barclamps
    1. “foo” is our barclamp name [required]
    2. “Zehicle” is my company name for the copyright information [default is Dell]
    3. “/barclamps” is the path where we are putting the barclamp [default is /opt/dell/barclamps]
    4. Result will be a populated barclamp. In this example: /barclamps/foo

That’s it! If you want to plan ahead then you could use an initialized git repo as the target.

Reminder: In building your barclamp, you’ll need to learn about Chef, how Crowbar extends cookbooks and how barclamps interact. That’s beyond the scope of this post.

Importing a barclamp

Once you created a barclamp, you can import the barclamp into Crowbar & Chef.

Assuming that you already created the foo barclamp in /barclamps, here are the steps:

  1. From the Crowbar server, become the super admin: sudo –i
  2. Run the barclamp install script: /opt/dell/bin/barclamp_install /barclamps/foo
    1. “/barclamps/foo” is the path to your barclamp. If could be anything!
    2. The core barclamps are in /opt/dell/barclamps.
    3. In a vm, you could mount a shared folder to access the barclamp (e.g.: /mnt/hgfs/barclamps)

Your barclamp should now show up in the Crowbar UI! You can also see it in Chef under the Crowbar databag.

While barclamps are generally safe to install multiple times, you can uninstall a barclamp using “barclamp_uninstall.rb /path/to/barclamp”

Barclamp layout

A barclamp has the following core components:

  • crowbar.yml configuration file (documented below)
  • README.txt file (optional, recommended)
  • chef directory containing
    • Cookbooks directory with Chef cookbooks
    • Data_bags directory with Crowbar configuration files
    • Roles directory with Chef roles used by the cookbooks and data_bags
  • crowbar_framework directory
    • app directory with Crowbar model, controller, and view code
    • other optional directories to add components needed by the UI such as images

The barclamp_model has a functional layout that covers most configuration requirements. The string ==BC-MODEL== indicates places where the name of the barclamp must be substituted. It is critical to understand that the name of the barclamp is embedded into the barclamp path and file names! This is needed to avoid file collisions when the barclamp is imported.

Crowbar.yml

The crowbar.yml file is a required configuration file that gives direction to the installer. The file has the following components:

barclamp:
  name: name of your barclamp (required, do not use space or hyphens) 
  display: pretty name of your barclamp [optional for now]
  description: information about your barclamp your barclamp [optional for now]
  version: what you want to consider for versioning [optiona] 
crowbar:
  layout: 1 (use the # one. This is required because it tells the installer what to expect inside your barclamp)
  order: 1000 (if installing multiple barclamps in one pass, order tells the installer the, well, order) 
nav: (remove the nav section, advanced users only) 
locale_additions: (you must add UI localizations here if you have any custom UI components) 
  en: (entries in this file map directly to entries in the config/locales/en.yml file and are added during install) 

Crowbar modularized: latest changes that make clouds even easier to create, update, and maintain

In the last week, my team at Dell completed a major refactoring of Crowbar that significantly improves our ability to bring in community contributions and field customizations.  Today, we merged it into Crowbar’s public repo(s).

From the very first versions, our objective for Crowbar was to create the fastest and most reliable cloud deployments. Along the way, we realized Crowbar’s true potential lay in embracing DevOps as an operational model for maintaining clouds. That meant building up cloud deployments in layers from pieces that we call barclamps (extensions of Chef cookbooks). Our first version, centered on OpenStack Cactus, leveraged barclamps but was still created as a single system. This unified system was a huge step forward in cloud deployments, but did not live up to our CloudOps vision of continuous delivery.

In this version, each Crowbar barclamp is an independent delivery unit that can be integrated before, while or after installing Crowbar.

The core of the change is each barclamp, including the most core ones, are stored in independent code repositories. Putting the code into distinct repos means that each barclamp can have its own life cycle, its own maintainer site and its own dependency tree. This modularization allows customers to manage their Crowbar deployments with a very fine brush: they may choose to customize parts of the system, they could lock components to specific tag and they can bring in barclamps from other vendors.

While the core barclamps are automatically integrated into the Crowbar build using git submodules; other barclamps are installed into the system as needed. This allows you to pull in the suite of OpenStack barclamps at build time or to wait until your Crowbar system is running before installing. Once you install a barclamp, you are able to retrieve an updated barclamp and reapply it to the system.

This feature gives you the ability to 1) choose exactly what you want to include and 2) perform field updates to a live Crowbar system.

Let’s look at some examples:

  1. The Cloud Foundry barclamp can be sourced Cloud Foundry instead of bundled into the Crowbar repository. This allows the team working on the cloud application to take ownership for their own deployment. As a continuous delivery proponent, I believe strongly that the development team should be responsible for ensuring that their code is deployable (refer to my OpenStack “Deployer API” blue print attempting to codify this).
  2. DreamHost, maintainers of Ceph Storage, can maintain their own local barclamp repos for OpenStack that are cloned from our community Swift barclamp. This allows them to innovate and customize OpenStack deployments for their business and choose which updates to merge back to the community.
  3. Rackspace Cloud Builders can work on the most leading edge OpenStack features and maintaining workable deployments on branches. As the code stabilizes, they simply merge in their changes.
  4. Dell BIOS and RAID barclamps only support the PowerEdge C line today. When we offer PowerEdge R support, you will be able to install or update the barclamps to add that capability. If another hardware vendor creates a barclamp for their hardware then you can install that into your existing system.

I believe that these changes to Crowbar are a huge step forwards on our journey of creating a community supportable Open Operations framework. I hope that you are as excited as I am about these changes.

I encourage you to take the first step by trying out Crowbar and, ultimately, writing your own barclamps.

Post Scripts:

  • In addition to the modularization, the updated code includes RHEL as a deployment platform. At present, you must choose to be either RHEL or Ubuntu at build time.
  • We have enhanced the network barclamp to describe connections as more abstract connections, called conduits, between nodes. This is a powerful change, but requires some understanding before you start making changes.
  • We have only begun testing the change as of 9/12, we expect the system to be fully stabilized by 10/3. If you are not willing to deal with bugs then I recommend building the Crowbar “v1.0” tag (or using the ISOs from our July launch).

Technical details of pending Crowbar changes

We’re testing a HUGE batch of changes to Crowbar before we commit them. The changes support the barclamp modularization work and also include the addition of RHEL and network barclamp update.

You may be eager to dig in; however, disruptiveness of these changes means that we are taking extra time to make sure that the build and install still work.

Here’s what you’ll see when we commit the changes:

  • Changes in naming to be more generic
    • Crowbar server user/pass is now crowbar/crowbar (was openstack/openstack)
    • Rails app path now crowbar_framework (was openstack_manager )
  • The pre-split barclamps (/change-image/dell/barclamps/*) have been moved into individual github repos (barclamp-*).
    • Barclamps are pulled into the build using “git submodule”
    • Chef scripts for barclamps are no longer copied and comingled together in the chef directory. They remain in their source directories (default /opt/dell/barclamps)
  • Inside the barclamps, you’ll find
    • A crowbar configuration file to direct the barclamp installer including localization and menu extensions.
    • Path changes to better align with the destination paths (command_line -> bin, app ->crowbar_framework)
    • App views moved under subdirectories
  • Changes to installation scripts
    • Barclamp installation changed to a ruby library so it can do more and be used individually outside of the install process. This allows barclamps to be imported or updated after installation.
    • Changes to create accommodate multiple operating systems
  • Addition of a “redhat-5.6-extra” directory with the RHEL 5.6 installation build components.
    • The RHEL version installs Opcode Chef Server 0.10 (Ubuntu is still 0.9 – community help here?)
  • Crowbar framework Rails app runs under Rainbow instead of Apache.
  • The code for the framework and the barclamp installer has been moved into the crowbar barclamp.
    • The installer bootstraps the crowbar barclamp to install itself.
  • The network barclamp has been substantially changed – that will require additional documentation. Features include
    • Concept of “conduits” that are constructed on nodes to be shared between barclamps
    • Ability to map adapters in a general way to deal with inconsistent enumeration
    • Mapping conduits to adapters allows for new teaming and multiple teaming configurations

We’ll post to the Crowbar listserv when changes. They will be posted to Crowbar HEAD. If you want the current build, we have created a “v1.0” tag.

Crowbar modularization work begins

I shared the following with the Crowbar listserv and wanted to post it for the larger audience.  If you want the latest on Crowbar then subscribe!

We’ve been getting questions and defects (thanks Matt Ray) about how we are going to allow you to update and add barclamps to Crowbar.  We’re working on that exact issue right now – you can watch me on the “modules” branch of the github.

NOTE TO CROWBAR FOLLOWERS: we are moving some items around in the repo!  There are “cactus” and “v1.0” tags in place so you can still build the current trees after we start the refactor.

We’ve got some big plans that I’ll outline on the list and earlier posts.

Right now, we’re working to modularize barclamps so that each one is in its own github repo.   This will allow you to pull in barclamps at build time or live on site. We’re also creating import/update routines that work for live systems to make it easier to develop barclamps.  Once again, that’s on the github modules branch. These will be exposed as rake barclamp:create[“foo”] and rake barclamp:install[“../foo”] type commands and I’ve committed to create some “how to make barclamps” videos.

That work is a prelude for a hard push on OpenStack Diablo before the design conference.  All that work will also be done in the github but the Diablo barclamps will be in independent repos from the Crowbar framework.

If you want to get started early.  80% of a barclamp effort is around the Chef Cookbooks.  Keith Hudgins with DTO did a great job writing up barclamps here: http://kb.dtosolutions.com/wiki/Deploying_the_cloudfoundry_barclamp.  We’re changing some of it to make it much easier and more modular.