Austin OpenStack Cloud Meetup: Thursday 10/27 6:30 PM at TechRanch Austin

OpenStack Enthusiasts, you are OFFICIALLY INVITED to Austin’s first post-Diablo OpenStack community event.

Dell is sponsoring an Austin OpenStack Meet Up help connect the Austin community around OpenStack and open source clouds!

Link: http://www.meetup.com/OpenStack-Austin/events/37908242/

We’ve got members of the Rackspace Cloud Builders Training team in town and Dell’s own Crowbar team attending.  We’re planning to do OpenStack demos and talk about the project in detail – and we’ll have plenty of pizza and sodas to keep the cloud juices flowing.

This is a great way to learn about the OpenStack cloud project and meet other people who are developing/deploying the hottest open source cloud around.

PLEASE SPREAD THE WORD – we’re trying to make this inaugural OpenStack meetup a big success!

See you there,

Joseph @jbgeorge George & Rob @Zehicle Hirschfeld

Dell Crowbar Project: Open Source Cloud Deployer expands into the Community

Note: Cross posted on Dell Tech Center Blogs.

Background: Crowbar is an open source cloud deployment framework originally developed by Dell to support our OpenStack and Hadoop powered solutions.  Recently, it’s scope has increased to include a DevOps operations model and other deployments for additional cloud applications.

It’s only been a matter of months since we open sourced the Dell Crowbar Project at OSCON in June 2011; however, the progress and response to the project has been over whelming.  Crowbar is transforming into a community tool that is hardware, operating system, and application agnostic.  With that in mind, it’s time for me to provide a recap of Crowbar for those just learning about the project.

Crowbar started out simply as an installer for the “Dell OpenStack™-Powered Cloud Solution” with the objective of deploying a cloud from unboxed servers to a completely functioning system in under four hours.  That meant doing all the BIOS, RAID, Operations services (DNS, NTP, DHCP, etc.), networking, O/S installs and system configuration required creating a complete cloud infrastructure.  It was a big job, but one that we’d been piecing together on earlier cloud installation projects.  A key part of the project involved collaborating with Opscode Chef Server on the many system configuration tasks.  Ultimately, we met and exceeded the target with a complete OpenStack install in less than two hours.

In the process of delivering Crowbar as an installer, we realized that Chef, and tools like it, were part of a larger cloud movement known as DevOps.

The DevOps approach to deployment builds up systems in a layered model rather than using packaged images.  This layered model means that parts of the system are relatively independent and highly flexible.  Users can choose which components of the system they want to deploy and where to place those components.  For example, Crowbar deploys Nagios by default, but users can disable that component in favor of their own monitoring system.  It also allows for new components to identify that Nagios is available and automatically register themselves as clients and setup application specific profiles.  In this way, Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.

We believe that operations that embrace layered deployments are essential for success because they allow our customers to respond to the accelerating pace of change.  We call this model for cloud data centers “CloudOps.”

Based on the flexibility of Crowbar, our team decided to use it as the deployment model for our Apache™ Hadoop™ project (“Dell | Apache Hadoop Solution”).  While a good fit, adding Hadoop required expanding Crowbar in several critical ways.

  1. We had to make major changes in our installation and build processes to accommodate multi-operating system support (RHEL 5.6 and Ubuntu 10.10 as of Oct 2011).
  2. We introduced a modularization concept that we call “barclamps” that package individual layers of the deployment infrastructure.  These barclamps reach from the lowest system levels (IPMI, BIOS, and RAID) to the highest (OpenStack and Hadoop).

Barclamps are a very significant architecture pattern for Crowbar:

  1. They allow other applications to plug into the framework and leverage other barclamps in the solution.  For example, VMware created a Cloud Foundry barclamp and Dream Host has created a Ceph barclamp.  Both barclamps are examples of applications that can leverage Crowbar for a repeatable and predictable cloud deployment.
  2. They are independent modules with their own life cycle.  Each one has its own code repository and can be imported into a live system after initial deployment.  This allows customers to expand and manage their system after initial deployment.
  3. They have many components such as Chef Cookbooks, custom UI for configuration, dependency graphs, and even localization support.
  4. They offer services that other barclamps can consume.  The Network barclamp delivers many essential services for bootstrapping clouds including IP allocation, NIC teaming, and node VLAN configuration.
  5. They can provide extensible logic to evaluate a system and make deployment recommendations.  So far, no barclamps have implemented more than the most basic proposals; however, they have the potential for much richer analysis.

Making these changes was a substantial investment by Dell, but it greatly expands the community’s ability to participate in Crowbar development.  We believe these changes were essential to our team’s core values of open and collaborative development.

Most recently, our team moved Crowbar development into the open.  This change was reflected in our work on OpenStack Diablo (+ Keystone and Dashboard) with contributions by Opscode and Rackspace Cloud Builders.  Rather than work internally and push updates at milestones, we are now coding directly from the Crowbar repositories on Github.  It is important to note that for licensing reasons, Dell has not open sourced the optional BIOS and RAID barclamps.  This level of openness better positions us to collaborate with the crowbar community.

For a young project, we’re very proud of the progress that we’ve made with Crowbar.  We are starting a new chapter that brings new challenges such as expanding community involvement, roadmap transparency, and growing Dell support capabilities.  You will also begin to see optional barclamps that interact with proprietary and licensed hardware and software.  All of these changes are part of growing Crowbar in framework that can support a vibrant and rich ecosystem.

We are doing everything we can to make it easy to become part of the Crowbar community.  Please join our mailing list, download the open source code or ISO, create a barclamp, and make your voice heard.  Since Dell is funding the core development on this project, contacting your Dell salesperson and telling them how much you appreciate our efforts goes a long way too.

Dell Crowbar to deploy OpenStack Diablo Cloud

Direction in the Cloud

Photo by JB George

This week, some of the Crowbar/Dell OpenStack-Powered Cloud team, plus Matt Ray from Opscode, have been working with our partners at Rackspace in San Antonio (see Opscode post about collaboration). Our target is to have Crowbar deliver a core Diablo deployment by the October 2011 design conference (sponsored in part by Dell). This is a collaborative effort and we invite community participation – we are trying to be open and communicative (via the Crowbar listserv) while also respecting that there is a mountain of work if we are to meet deadlines.

We are doing the work in the open on the Crowbar Github so you have access to the very latest capabilities and it also means that the head the Crowbar may be unstable while we add capabilities. We feel like this is an important trade off because it allows us to keep up with the rapid pace of development in OpenStack (and other projects). This is the motivation for the recent modularization work and will continue to be a feature driver for Crowbar enhancements because it allows Crowbar users to easily bring in updated bits.

 

Building Crowbar post-modularization (15 minute how-to video)

Note: I’m putting build ISOs and Sledgehammer TARs on crowbar.zehicle.com if you don’t want to follow these steps then download the ISO. We are updating the ISO daily, so don’t assume that you have that latest!

To build Crowbar, you need a Linux machine and access to the internet. The video shows how you can use an Ubuntu 10.10 Rackspace Cloud Server.  We build Crowbar inside our firewall on our PCs too. No matter how you do it, Crowbar is full of fuzzily delicious cloud bits.

For up-to-date instructions, see the Crowbar wiki Build ISO page.


Crowbar modularized: latest changes that make clouds even easier to create, update, and maintain

In the last week, my team at Dell completed a major refactoring of Crowbar that significantly improves our ability to bring in community contributions and field customizations.  Today, we merged it into Crowbar’s public repo(s).

From the very first versions, our objective for Crowbar was to create the fastest and most reliable cloud deployments. Along the way, we realized Crowbar’s true potential lay in embracing DevOps as an operational model for maintaining clouds. That meant building up cloud deployments in layers from pieces that we call barclamps (extensions of Chef cookbooks). Our first version, centered on OpenStack Cactus, leveraged barclamps but was still created as a single system. This unified system was a huge step forward in cloud deployments, but did not live up to our CloudOps vision of continuous delivery.

In this version, each Crowbar barclamp is an independent delivery unit that can be integrated before, while or after installing Crowbar.

The core of the change is each barclamp, including the most core ones, are stored in independent code repositories. Putting the code into distinct repos means that each barclamp can have its own life cycle, its own maintainer site and its own dependency tree. This modularization allows customers to manage their Crowbar deployments with a very fine brush: they may choose to customize parts of the system, they could lock components to specific tag and they can bring in barclamps from other vendors.

While the core barclamps are automatically integrated into the Crowbar build using git submodules; other barclamps are installed into the system as needed. This allows you to pull in the suite of OpenStack barclamps at build time or to wait until your Crowbar system is running before installing. Once you install a barclamp, you are able to retrieve an updated barclamp and reapply it to the system.

This feature gives you the ability to 1) choose exactly what you want to include and 2) perform field updates to a live Crowbar system.

Let’s look at some examples:

  1. The Cloud Foundry barclamp can be sourced Cloud Foundry instead of bundled into the Crowbar repository. This allows the team working on the cloud application to take ownership for their own deployment. As a continuous delivery proponent, I believe strongly that the development team should be responsible for ensuring that their code is deployable (refer to my OpenStack “Deployer API” blue print attempting to codify this).
  2. DreamHost, maintainers of Ceph Storage, can maintain their own local barclamp repos for OpenStack that are cloned from our community Swift barclamp. This allows them to innovate and customize OpenStack deployments for their business and choose which updates to merge back to the community.
  3. Rackspace Cloud Builders can work on the most leading edge OpenStack features and maintaining workable deployments on branches. As the code stabilizes, they simply merge in their changes.
  4. Dell BIOS and RAID barclamps only support the PowerEdge C line today. When we offer PowerEdge R support, you will be able to install or update the barclamps to add that capability. If another hardware vendor creates a barclamp for their hardware then you can install that into your existing system.

I believe that these changes to Crowbar are a huge step forwards on our journey of creating a community supportable Open Operations framework. I hope that you are as excited as I am about these changes.

I encourage you to take the first step by trying out Crowbar and, ultimately, writing your own barclamps.

Post Scripts:

  • In addition to the modularization, the updated code includes RHEL as a deployment platform. At present, you must choose to be either RHEL or Ubuntu at build time.
  • We have enhanced the network barclamp to describe connections as more abstract connections, called conduits, between nodes. This is a powerful change, but requires some understanding before you start making changes.
  • We have only begun testing the change as of 9/12, we expect the system to be fully stabilized by 10/3. If you are not willing to deal with bugs then I recommend building the Crowbar “v1.0” tag (or using the ISOs from our July launch).

Don’t fork it up. OpenStack needs community collaboration

Cant we just be friends?

We’re standing on the eve of the OpenStack 4th Design summit (aka Essex) and I’m watching a frenzy of IT Goliaths (Dell, Citrix, Cisco, HP, Rackspace) and some Cloud Davids (Nebula, Stackops) try to tangle revenue streams from an open source cloud project.

I was pleased to read GigaOM‘s Derrick Harris validation of Dell’s strategy which featured my team’s contributions (Crowbar, OpenStack & Hadoop).  We are working hard to bring these technologies to our customers in an open and collaborative way.

Dell has substantial IT assets to bring to bear on cloud solutions.  All of them are ultimately tied to products that generate revenue for Dell; however, that does not prevent our being able to collaborate and share.  On the contrary, we benefiting from input from our partners, customers and community to determine which features are needed to accelerate adoption.  Our recent decision to accelerate Crowbar modularization is a clear example of that process.

It is essential to understand that this is not just about cloud technologies!  It is about the collaborative way we are promoting them and the processes we are using to deliver them.

With Dell’s cloud moving at hurricane speed, it has been interesting to watch how other companies are setting their own OpenStack initiatives.  It seems to me that many of these efforts involve forks from OpenStack that cannot/will not be contributed back the community.  One (but not the only) example is from HP’s Emil Sayegh who says that “HP developers … ideas will be shared…”  He does not commit to sharing HP’s code in his post.  I hope that is an oversight and not their plan.

In time, forking may be needed.  Right now, we need to focus on building a strong foundation.  Open contributions of code are the engine of that success.

WHIR Webinar Notes: Prying Open the Cloud with Dell Crowbar & OpenStack

Panelists: Me (@zehicle) & Joseph B. George (@jbgeorge), Director, Cloud and Big Data Solutions, Dell

Moderator: Liam Eagle (@theWHIR) , Editor-in-Chief, Web Host Industry Review

Wow, this Webinar was an hour of OpenStack insights (see the whole thing). If you don’t have the hour then you can use my time line nodes to jump to what you want to hear.

  • 2:50: Presentation Starts (introductions are over)
  • 3:40: Joseph coins the word “dynormous” for dynamic & large scale clouds
  • 4:40: Customers want to know how they are going to maintain a cloud
  • 4:50: Customers don’t want a 9 month cycle for features, want it faster. DevOps gives us the flexibility to meet our customer needs as quickly as they want to.
  • 7:11: Massive scalability… their (Rackspace & NASA) business is about scale
  • 8:00: Rackspace and NASA started from the beginning to build a community
  • 8:50: We have the data that this has staying power
  • 10:10: We see a lot of companies joining in the community
  • 11:56: Shout out to Opscode Chef
  • 12:40: From bare metal to a fully functioning cloud in under 2 hours. Crowbar allows you to introduce new elements into the environment
  • 13:40: Crowbar leverages our experience with cloud deployments
  • 14:33: Dell was the only provider there from day 1. We have the most experience.
  • 17:27: DevOps Poll
  • 18:40: DevOps is a significant trend that you should consider. Hosters have a lot of operational chops.
  • 19:34: There are a lot of right ways to do cloud. You need to pick what’s best for your business model
  • 20:23: We could get hardware and software, but operational expertise was missing.
  • 21:33: We’re more making the complexity of a cloud go away. We are getting our customers a head start. We are chipping away at the learning curve.
  • 22:05: The cloud is always ready, never finished. Cloud is an ongoing operational environment: DevOps!
  • 23:30: Crowbar bakes a lot of operational experience into the deployment.
  • 25:17: Core tenant of DevOps: there is no single OpenStack image. Cloud is too complex. We build it in layers.
  • 26:26: Before you deploy, you can change the configuration.
  • 27:30: Barclamps are modules that execute a function. We are inviting community participation
  • 28:40: Crowbar process view – Crowbar is a “PXE state machine” is a very simplified description.
  • 29:57: You can go through a tuning cycle where you can get it working, make sure it’s right, flush and reset. That ensures you have an automated system.
  • 30:34: Screen shots with descriptions
  • 33:25: Event the core state machine that runs Crowbar is deployed as a barclamp
  • 35:00: You can download OpenStack and install it yourself from our github. We don’t want to talk about OpenStack, we want to DO OpenStack.
  • Poll Results (see to the right)
  • 38:00: Online resources
  • 40:00: Question 1: Timeline for RHEL. Answer: RHEL is part of Hadoop, will make it into OpenStack by end of year (or sooner based on market demand)
  • 42:17: Question 2: What led Dell to get involved in OpenStack? Answer: It’s about experience. We like being able to fix and change if we needed. There is a lot of active community
  • 45:30: Question 3: How does a hardware maker play with open source software? Answer: It’s a solution for us. We wanted to make sure that people cloud deploy the software. Adding DevOps takes it to another level.
  • 48:00 Question 4: What elements of Diablo are most exciting? Answer: Keystone (centralized authentication) is a big deal. Networking changes that “bust the top” of the networking hurdles.
  • 50:25: Question 5: Where is OpenStack going long term? Answer: We’re pleasantly surprised about how much it’s picked up. We’ll see more standards in the community. We have high hopes for OpenStack and have invested heavily. We’ll see more as-a-service capabilities to build on a common infrastructure: both open and commercial.
  • 52:47: Question 6: What’s the biggest barrier to operating at scale? Answer: learning how to operate is the biggest hurdle. We took a learning approach to help customers get started. We are hosting a training with Rackspace.
  • 55:00: Question 7: Where does Dell and Rackspace overlap? Answer: We see Rackspace Cloud Builders that the premier experts. Dell Services is involved with all of it. Dell takes the phone call and deals with our customers directly.