Category Archives: Open source

Cloudcast interview with Dell Cloud Solutions Team (quotes with time stamps)

Posted on by
Reply

Lloyds BarbershopTheCloudcast.net – Thank you for such a great series of questions. Wow, nearly 36+ minutes of cloudicious interview about the work my team at Dell is doing!

Thanks to our hosts for putting together a great series! They are:

Highlights from Episode 16: Dell, Dude you’re getting a cloud

  • 3:40 JBG “we are listening to our customers tell us what they want to accomplish”
  • 4:40 RAH “humility is part of [what we’re] doing … cloud is about learning and collaboration”
  • 6:40 RAH “OpenStack filled a niche. It was the first open source community cloud. … Not just open source, its open community.”
  • 7:15 RAH “We’re beyond critical mass. We’re seeing acceleration… we are transitioning into a community development.”
  • 7:30 RAH “It’s accelerating. It happening so fast.”
  • 8:00 RAH “We felt it was really important for people to be able to use it. We felt that it was important to get away from just people developing into people using. “
  • 8:57 – RAH “Cloud is not just one thing. You have to have all the pieces.”
  • 10: 15 – RAH “Cloud is always ready, never finished”
  • 10:50 – RAH “OpenStack is an alternative to public cloud including hosting providers seeking to offer their own cloud”
  • 12:40 – AD “Dell has been in the Big data space for many years now”
  • 20:15 – JBG “There’s a legacy of great partnerships that we leverage”
  • 20:48 – JBG “Conflicts have not come up because we are focused on the customer”
  • 21:30 – RAH “Shout out to Greg Althaus for solving these problems in such an elegant way. And we rewrote it 3 times”
  • 22:02 – RAH “Crowbar started from our frustration of bringing up a cloud quickly … so we took a DevOps approach.”
  • 22:41 – RAH “You had to have a system view AND a boot strapping view simultaneously”
  • 23:50 – JBG “Crowbar was born out of necessity because we were setting up and blowing away our clouds over and over and over again. “
  • 24:40 – JBG “We realized there were not many people thinking about all the pieces before OpenStack was installed”
  • 25:20 – RAH “We don’t think customers have all the answers before we show up. This is not unique to OpenStack.”
  • 28:20 – JBG “We’re seeing the community pick up Crowbar as a way to deploy”

Big Questions? Big Answers with Dell BigData solution (plus Crowbar gets RHEL)

Posted on by
6

In my enthusiasm for all things Dell + OpenStack, I have neglected to talk about my team’s interesting Big Data work with Apache Hadoop.  Hadoop is a suite of open source projects for analyzing large data sets of unstructured data.  Initially, Hadoop centered around use of the map-reduce algorithm; however, it’s grown way beyond that as the community has worked to solve problems related to data storage, discovery, and scheduling.

Big Data clouds are well suited to my team because the model (non-redundant/cloud) and scale (hyper) of their deployments.  It should be no surprise that builders of analysis clouds have the same goals (maximizing operational ROI per compute unit) as builders of other types of clouds.

Our Hadoop solution relies on the same core principles (CloudOps) and technologies (Crowbar) as our OpenStack solution.  Like our other cloud solutions, we are working closely with a proven leader: Cloudera.  Now that we’ve formally announced our solution and partnership, I can talk a about what we’re doing on the Big Data front.

One extra thing that I’m proud to announce, we’ll be adding Red Hat Enterprise Linux (RHEL) support to Crowbar to support our Hadoop solution.  This support is not just at the node level: we are making Crowbar admin run on either platform too!  This is significant for two reasons:

  1. It expands the number of platforms and support options for Crowbar users
  2. It provides the framework to support more varieties of node operating environment (e.g.: XenServer, BSD, DRDOS, etc)

For more information, check out:

Crowbar build using Ubuntu 10.10 vm on Rackspace Cloud from Github Repo

Posted on by
11

Our OpenStack team at Dell (especially Victor Lowthor) has been working hard with the public Crowbar repos to make it possible for the community to build their own version of a Crowbar ISO.   When you build the ISO, you’ll be downloading a whole bunch (that’s the technical term) of open source licensed components to make it work: we’re trying to maintain a list of licenses on the Github wiki.

To make sure that it was possible for mortals, I signed up for a Ubuntu 10.10 VM (512 Mb RAM, $0.03/hr) at RackSpace Cloud.  I did this from a non-Dell to ensure that it was as independent from our source as possible.

Once I had my vm, there were just a few steps to follow (these are NOT verbatim):

  • apt-get install debootstrap, mkisofs, git, build-essential packages
  • git clone git://github.com/dellcloudedge/crowbar.git
  • Got the results from a sledgehammer build (a fresh sledgehammer tarball) and extracted it into $HOME/.crowbar-build-cache/tftpboot, which is where build_crowbar.sh expects to find it cached.
    • NOTE: I’m not ready to document sledgehammer builds yet, but I will tell you that you’d need a CentOS VM.
  • In the crowbar directory, ran ./build_crowbar.sh
  • The build will pull down all the packages that you need and cache them to the VM.  Subsequent builds will be much faster!

The end result of the build is an “openstack-dev.iso” that will install Crowbar with the OpenStack barclamps (here’s how to do it on VMs).  Just for fun, I copied _my build_ output ISO off the build VM and to my web server.

Please let me know if you have problems with this process, we want people to try Crowbar!

$$ Note: Turn off your VM when you’re done so you don’t incur extra expenses.  Since this process only took about 2 hours, the whole build cost me less than a dime.  Which is good, since I was building it on “my own dime” anyway.

Crowbar source released, includes OpenStack Cloud install

Posted on by
32

I’m delighted to announce (official version) that my team at Dell has opened the Crowbar source under the Apache 2 license. This action is part of the broader Dell OpenStack Cloud Solution which includes OpenStack install packages, Crowbar, reference hardware architectures, and services/consulting to support deployments.

There are two important components to this news:

  1. Dell is officially offering our OpenStack Solution and helping advance the community’s ability to implement OpenStack quickly and consistently.
  2. Dell is releasing the Crowbar code (which is included in the solution) as open source.

Both are significant items; however, my focus here is on the Crowbar release.

Crowbar started as a Dell OpenStack installer project and then grew beyond that in scope.  Now it can be extended to work with other vendors’ kits and other solutions bits.

We are contributing Crowbar to the community because we believe that everyone benefits by sharing in the operational practices that Crowbar embodies. These are rooted in Opscsode Chef (which Crowbar tightly integrates with) and the cloud & hyper-scale proven DevOps practices that are reflected in our deployment model.

Where to get it?

What’s included?

  • A comprehensive set of barclamps to set up an OpenStack cloud.
  • Crowbar UI and Remote APIs to make it easy to set up your cloud
  • Automated testing scripts for community members doing continuous integration with OpenStack.
  • Build scripts so you can create your own Crowbar install ISO
  • Switch discovery so you can create Chef Cookbooks that are network aware.
  • Open source Chef server that powers much of Crowar’s functionality

What’s not included?

  • Non-open source license components (BIOS+RAID config) that we could not distribute under the Apache 2 license.  We are working to address this and include them in our release.  They are available in the Dell Licensed version of Crowbar.
  • Dell Branded Components (skin + overview page).   Crowbar has an OpenSource skin with identical functionality.
  • Pre-built ISOs with install images (you must download the open source components yourself, we cannot redistribute them to you as a package)

Important notes:

  • Crowbar uses Chef Server as its database and relies on cookbooks for node deployments.  It is installed (using Chef Solo) automatically as part of the Crowbar install.
  • Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as barclamps.
  • Each barclamp has its own Chef configuration, UI sub-component, deployment configuration, and documentation.

On the project roadmap:

  • Hadoop support
  • Additional operating system support (specifically RHEL)
  • Barclamp version repository
  • Network configuration
  • We’d like suggestions!  Please comment!

Sites for more information: Joseph George, Barton George (launch day), Dell

Crowbar’s surprise value proposition: continous integration (#ci) testing

Posted on by
4

As part of our Agile/Lean methodologies, our team at Dell is highly invested in automated testing and continuous integration.  We’re running Jenkins to coordinate builds and EVERY CHECK-IN launches our full integration suite that tests our system end-to-end.  It may not be typical, but I don’t consider that to be particularly note worthy because it’s best practice.    (Rob’s note: if you write code and don’t think you have the authority then you need to geek-up and just do it – that’s our MO at Dell)

It’s important to understand that since Crowbar is an installer, every check-in does a FULL CLEAN INSTALL of all the Cactus OpenStack components.  Our verification requires that we test OpenStack because that’s our #1 exit requirement.  Consequently, we have built an automated build system that does a continuous integration test of a full, multi-node Nova/Glance/Swift deployment.

Automated end-to-end integration tests of OpenStack are a very handy thing!

In the last few weeks, we’ve heard from Dell internal groups and partners who are contributing to OpenStack Diablo that they want to leverage our work in continuous integration.  This will allow them to make sure that their development work does not regress other functions.  It’s a significant opportunity to ensure that we can collaborate between organizations.  It also promotes early development and distribution of Diablo installation scripts.

To support this in Crowbar, we are already planning incorporate more sophisticated revision control (likely based on Git) into Crowbar.

Note: YES, we consider our CI scripts to be part of our open source code.

I’ll be at OSCON 7/25-29/11 (Dell=sponsor & speaking w/ @jbgeorge)

Posted on by
Reply

As part of our commitment to open source, Dell is a sponsor of OSCON 2011.  The Dell OpenStack Cloud team will have a booth presence with our well-travelled Crowbar Install rack (now with BOTH PowerEdge C6100 & C6105s).  We’re doing our famous 30 minute OpenStack installs and handing out goodies including USB keys. 

Joseph George (@jbgeorge) and I are speaking:

We’ll be giving specifics about how Crowbar works to deliver the Dell OpenStack Cloud Solution including a narrated demo and details about how the community can extend Crowbar using barclamps.

Note:  Stephen Spector (opnstk_com_mgr), the amazing OpenStack community manager, wanted me to remind everyone that we’re celebrating OpenStack’s 1 year anniversary with activites at OSCON.  He’s asking for video commentary about OpenStack and RSVPs if you can attend the events.  More at OpenStack Blog.

OpenStack Crowbar User Guide: explaining how barclamps get deployed

Posted on by
1

My whole team is working feverishly on the final touches of Crowbar before we turn over the keys.  We’re putting it through a complete release cycle (extensive QA, customer pilots, documentation, etc) because internal Dell consumers are expecting that level of finish. 

For those in the community eagerly waiting to see the code, I hope you like the extra polish (for example: I18N, user & deployment guides, bundled continuous integration scripts, and months of testing).

RUMOR CONTROL NOTE: Crowbar is NOT limited to deployments on Dell products!!  Our BIOS and RAID barclamps are, of course, targeted and licensed for Dell customers.  The OpenStack and other barclamps will work on any gear that can run Chef Client.

Tonight I was working on the user guide and thought I would share the graphic and text describing how a barclamp gets deployed.

The figure shows the entire of a barclamp within the Crowbar user interface.  A Barclamp defines the capability for a service but cannot be deployed.  To deploy a barclamp, you must create a Proposal.  Once the proposal is created, you must selection nodes to operate on.  As discussed in the next sections, you may also edit the Proposal’s attributes as needed.  

Applying the Proposal tells Crowbar to deploy the proposal onto the nodes.  While deploying, nodes return to the Ready state when deployment is completed.  Once a proposal has become an Active Role, you cannot edit it.  You must delete the Role and repeat the Apply process

Preview Crowbar GUI (OpenStack Installer by Dell for Cloud)

Posted on by
4

I can’t show you the really cool Overview screen yet, but here’s the one that replaces the one we’ve demo’ed before.  The nodes are grouped by switch and ordered by port so it creates a very nice “rack” layout if your wiring is organized.

Props to Jon Roberts (@emptyflask) for his excellent UI work!

Hungry for Nova Cuisine? Adding Chef recipes for OpenStack Nova

Posted on by
1

As promised, here’s the other drop in advance of our OpenStack team’s Crowbar release. 

This is the second part of the Swift and Nova recipes that we are intentionally leaking out to the community.

USAGE NOTE: These recipes are designed to work with Crowbar!  They are not intended to stand alone.

As part of our collaboration with Opscode, Matt Ray, has been merging our recipes into his most excellent OpenStack cookbook tree.  If you want to see our unmerged recipes, we’re also posting those to our github. 

In addition to our Swift recipes, you can now check out the Nova recipes.

ADDITIONAL USAGE NOTE: The Matt’s tree is more complete – these are released for reference only.  They will ultimately be maintained as part of the Crowbar.

Cooking up OpenStack Chef recipes with Opscode

Posted on by
3

Our OpenStack team here at Dell has been busy getting Crowbar ready to open source and that does not leave much time for blog posts.  We’re putting on a new UI, modularizing with barclamps and creating network options for Nova Cactus.

Sharing is goodHowever, I wanted to take a minute to update the community about Swift and Nova recipes that we are intenionally leaking out to the community in advance of the larger Crowbar code drop.

As part of our collaboration with Opscode, Matt Ray, has been merging our recipes into his most excellent OpenStack cookbook tree.  If you want to see our unmerged recipes, we’re also posting those to our github.  So far, we have the Swift recipes available (thanks to Andi Abes!) with Nova to follow soon.

5/31 Update: These are now online.