jump to navigation

CloudOps white paper explains “cloud is always ready, never finished” January 16, 2012

Posted by Rob H in CloudFoundry, CloudOps, Crowbar, DevOps, Greg Althaus, Hadoop, Linux, Open source, OpenStack.
2 comments

I don’t usually call out my credentials, but knowing the I have a Masters in Industrial Engineering helps (partially) explain my passion for process as being essential to successful software delivery. One of my favorite authors, Mary Poppendiek, explains undeployed code as perishable inventory that you need to get to market before it loses value. The big lessons (low inventory, high quality, system perspective) from Lean manufacturing translate directly into software and, lately, into operation as DevOps.

What we have observed from delivering our own cloud products, and working with customers on thier’s, is that the operations process for deployment is as important as the software and hardware. It is simply not acceptable for us to market clouds without a compelling model for maintaining the solution into the future. Clouds are simply moving too fast to be delivered without a continuous delivery story.

This white paper [link here!] has been available since the OpenStack conference, but not linked to the rest of our OpenStack or Crowbar content.

Dell Crowbar Project: Open Source Cloud Deployer expands into the Community October 18, 2011

Posted by Rob H in CloudFoundry, Crowbar, Hadoop, Open source, OpenStack, Opscode, RackSpace, VMware.
Tags: , , , , , , , , , , ,
1 comment so far

Note: Cross posted on Dell Tech Center Blogs.

Background: Crowbar is an open source cloud deployment framework originally developed by Dell to support our OpenStack and Hadoop powered solutions.  Recently, it’s scope has increased to include a DevOps operations model and other deployments for additional cloud applications.

It’s only been a matter of months since we open sourced the Dell Crowbar Project at OSCON in June 2011; however, the progress and response to the project has been over whelming.  Crowbar is transforming into a community tool that is hardware, operating system, and application agnostic.  With that in mind, it’s time for me to provide a recap of Crowbar for those just learning about the project.

Crowbar started out simply as an installer for the “Dell OpenStack™-Powered Cloud Solution” with the objective of deploying a cloud from unboxed servers to a completely functioning system in under four hours.  That meant doing all the BIOS, RAID, Operations services (DNS, NTP, DHCP, etc.), networking, O/S installs and system configuration required creating a complete cloud infrastructure.  It was a big job, but one that we’d been piecing together on earlier cloud installation projects.  A key part of the project involved collaborating with Opscode Chef Server on the many system configuration tasks.  Ultimately, we met and exceeded the target with a complete OpenStack install in less than two hours.

In the process of delivering Crowbar as an installer, we realized that Chef, and tools like it, were part of a larger cloud movement known as DevOps.

The DevOps approach to deployment builds up systems in a layered model rather than using packaged images.  This layered model means that parts of the system are relatively independent and highly flexible.  Users can choose which components of the system they want to deploy and where to place those components.  For example, Crowbar deploys Nagios by default, but users can disable that component in favor of their own monitoring system.  It also allows for new components to identify that Nagios is available and automatically register themselves as clients and setup application specific profiles.  In this way, Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.

We believe that operations that embrace layered deployments are essential for success because they allow our customers to respond to the accelerating pace of change.  We call this model for cloud data centers “CloudOps.”

Based on the flexibility of Crowbar, our team decided to use it as the deployment model for our Apache™ Hadoop™ project (“Dell | Apache Hadoop Solution”).  While a good fit, adding Hadoop required expanding Crowbar in several critical ways.

  1. We had to make major changes in our installation and build processes to accommodate multi-operating system support (RHEL 5.6 and Ubuntu 10.10 as of Oct 2011).
  2. We introduced a modularization concept that we call “barclamps” that package individual layers of the deployment infrastructure.  These barclamps reach from the lowest system levels (IPMI, BIOS, and RAID) to the highest (OpenStack and Hadoop).

Barclamps are a very significant architecture pattern for Crowbar:

  1. They allow other applications to plug into the framework and leverage other barclamps in the solution.  For example, VMware created a Cloud Foundry barclamp and Dream Host has created a Ceph barclamp.  Both barclamps are examples of applications that can leverage Crowbar for a repeatable and predictable cloud deployment.
  2. They are independent modules with their own life cycle.  Each one has its own code repository and can be imported into a live system after initial deployment.  This allows customers to expand and manage their system after initial deployment.
  3. They have many components such as Chef Cookbooks, custom UI for configuration, dependency graphs, and even localization support.
  4. They offer services that other barclamps can consume.  The Network barclamp delivers many essential services for bootstrapping clouds including IP allocation, NIC teaming, and node VLAN configuration.
  5. They can provide extensible logic to evaluate a system and make deployment recommendations.  So far, no barclamps have implemented more than the most basic proposals; however, they have the potential for much richer analysis.

Making these changes was a substantial investment by Dell, but it greatly expands the community’s ability to participate in Crowbar development.  We believe these changes were essential to our team’s core values of open and collaborative development.

Most recently, our team moved Crowbar development into the open.  This change was reflected in our work on OpenStack Diablo (+ Keystone and Dashboard) with contributions by Opscode and Rackspace Cloud Builders.  Rather than work internally and push updates at milestones, we are now coding directly from the Crowbar repositories on Github.  It is important to note that for licensing reasons, Dell has not open sourced the optional BIOS and RAID barclamps.  This level of openness better positions us to collaborate with the crowbar community.

For a young project, we’re very proud of the progress that we’ve made with Crowbar.  We are starting a new chapter that brings new challenges such as expanding community involvement, roadmap transparency, and growing Dell support capabilities.  You will also begin to see optional barclamps that interact with proprietary and licensed hardware and software.  All of these changes are part of growing Crowbar in framework that can support a vibrant and rich ecosystem.

We are doing everything we can to make it easy to become part of the Crowbar community.  Please join our mailing list, download the open source code or ISO, create a barclamp, and make your voice heard.  Since Dell is funding the core development on this project, contacting your Dell salesperson and telling them how much you appreciate our efforts goes a long way too.

So you want to create a Crowbar barclamp? Here’s what you have to know… September 19, 2011

Posted by Rob H in CloudFoundry, Crowbar, Hadoop, OpenStack, Opscode.
3 comments

My team at Dell has created many barclamps to support OpenStack and Hadoop. One major objective of our recent modularization refactoring was to make it easier to the community to contribute barclamps. The Crowbar CloudOps approach is to build up a full cloud deployment using layers. So each barclamp represents a component of the overall deployment.

Note 9/21: Added Video Post showing the steps below.

A barclamp is a deployment module that is imported from its own code repository into the Crowbar framework. A barclamp cannot operate without Crowbar, but you do not have to create a unique build of Crowbar in order to create a barclamp.

The first thing to know about barclamps is that most of the work (80%!) is building your Chef cookbooks. If you don’t have a cookbook that deploys your application then stop here and work on that first.

The second thing to know about barclamps is that there are a lot of them that you can study for examples. Check out the Glance barclamp if you have a single server deployment, Nagios if you have a service that needs to be integrated into every node, Nova if you have a complex multi-component system and Provisioner if you want to impact core Crowbar functionality.

We’ve done a lot of work to make it easy to create and install a stub barclamp. Our experience is that building a barclamp is a highly iterative exercise with a lot of testing. Luckily, Crowbar’s primary mission is to help you brush, rinse and repeat. From there, you can customize and extend your barclamp to deploy your application’s full untamed glory.

Before you try to create a new barclamp, you must install Crowbar.

Creating a barclamp

The following steps use the barclamp_model that included under /dell/opt and is described below.

  1. Figure out the name of your barclamp. I’m naming our example “foo barclamp”
    1. Barclamps must have unique names.
    2. Do not use spaces or hyphens.
  2. From the Crowbar server, become the super admin: sudo -i
  3. Create a directory for your barclamp: mkdir /barclamps
  4. Run the barclamp create script: /opt/dell/bin/barclamp_create.rb foo “Zehicle” /barclamps
    1. “foo” is our barclamp name [required]
    2. “Zehicle” is my company name for the copyright information [default is Dell]
    3. “/barclamps” is the path where we are putting the barclamp [default is /opt/dell/barclamps]
    4. Result will be a populated barclamp. In this example: /barclamps/foo

That’s it! If you want to plan ahead then you could use an initialized git repo as the target.

Reminder: In building your barclamp, you’ll need to learn about Chef, how Crowbar extends cookbooks and how barclamps interact. That’s beyond the scope of this post.

Importing a barclamp

Once you created a barclamp, you can import the barclamp into Crowbar & Chef.

Assuming that you already created the foo barclamp in /barclamps, here are the steps:

  1. From the Crowbar server, become the super admin: sudo –i
  2. Run the barclamp install script: /opt/dell/bin/barclamp_install /barclamps/foo
    1. “/barclamps/foo” is the path to your barclamp. If could be anything!
    2. The core barclamps are in /opt/dell/barclamps.
    3. In a vm, you could mount a shared folder to access the barclamp (e.g.: /mnt/hgfs/barclamps)

Your barclamp should now show up in the Crowbar UI! You can also see it in Chef under the Crowbar databag.

While barclamps are generally safe to install multiple times, you can uninstall a barclamp using “barclamp_uninstall.rb /path/to/barclamp”

Barclamp layout

A barclamp has the following core components:

  • crowbar.yml configuration file (documented below)
  • README.txt file (optional, recommended)
  • chef directory containing
    • Cookbooks directory with Chef cookbooks
    • Data_bags directory with Crowbar configuration files
    • Roles directory with Chef roles used by the cookbooks and data_bags
  • crowbar_framework directory
    • app directory with Crowbar model, controller, and view code
    • other optional directories to add components needed by the UI such as images

The barclamp_model has a functional layout that covers most configuration requirements. The string ==BC-MODEL== indicates places where the name of the barclamp must be substituted. It is critical to understand that the name of the barclamp is embedded into the barclamp path and file names! This is needed to avoid file collisions when the barclamp is imported.

Crowbar.yml

The crowbar.yml file is a required configuration file that gives direction to the installer. The file has the following components:

barclamp:
  name: name of your barclamp (required, do not use space or hyphens) 
  display: pretty name of your barclamp [optional for now]
  description: information about your barclamp your barclamp [optional for now]
  version: what you want to consider for versioning [optiona] 
crowbar:
  layout: 1 (use the # one. This is required because it tells the installer what to expect inside your barclamp)
  order: 1000 (if installing multiple barclamps in one pass, order tells the installer the, well, order) 
nav: (remove the nav section, advanced users only) 
locale_additions: (you must add UI localizations here if you have any custom UI components) 
  en: (entries in this file map directly to entries in the config/locales/en.yml file and are added during install)

Collaboration between Dell Crowbar & VMware Cloud Foundry – unleashes your inner cloud August 17, 2011

Posted by Rob H in CloudFoundry, Crowbar, Dave McCrory, PaaS, VMware.
Tags: , , , , , ,
11 comments

Sometimes a single sprint can deliver magic: when I signed up to document how to create a Crowbar module (aka a barclamp) two weeks ago, I had no idea that it would add a new flavor to Crowbar .

I’m proud to announce that the first public non-Dell Crowbar module will be supporting the VMware Cloud Foundry Open PaaS project.

Development is still in progress (on the Crowbar “CF” branch) and you’ll be able to watch us (even help!) collaborate on this project.  Initially, the deployment will be to single server but we’re hoping to quickly expand to a distributed install that fully leverages the capabilities of both projects.

By creating a Crowbar module, Cloud Foundry™ is able to leverage the cloud deployment capabilities that allow it to be setup on any physical or virtualized data center.  This is core to the Crowbar message: the value of a cloud solution can best be realized when it’s coupled with open practices for deploying it.

There are many significant aspects of this collaboration:

  1. Cloud Foundry is taking the right approach to PaaS.  Their team’s perspective on PaaS mirrors my own: A PaaS is a collection of application services.  That approach makes it extensible and flexible.  Plus, they are also multi-language and multi-platform.
  2. Crowbar is proving our breadth of support.  Last week we announced coming RHEL support and now adding Cloud Foundry is a natural extension.  We did not design Crowbar to be a one-trick pony.  It’s modular design makes it easy to extend while leveraging the existing body of work.
  3. Big companies are acting like start-ups.  Both Crowbar and Cloud Foundry are projects that focus on putting the core functionality out quickly to prove their value proposition, get feedback, and change the game.  This collaboration is positive proof of these companies being Agile and starting a project Lean.
  4. Big companies are acting in the open.  Both Dell via Crowbar and VMware via Cloud Foundry are contributing their source and working on it in the open.

Stay tuned for that “how to create a barclamp” post (or check out the barclamp rake task).

For more information:

Follow

Get every new post delivered to your Inbox.

Join 980 other followers