Category Archives: Clouds

Crowbar modularization work begins

Posted on by
3

I shared the following with the Crowbar listserv and wanted to post it for the larger audience.  If you want the latest on Crowbar then subscribe!

We’ve been getting questions and defects (thanks Matt Ray) about how we are going to allow you to update and add barclamps to Crowbar.  We’re working on that exact issue right now – you can watch me on the “modules” branch of the github.

NOTE TO CROWBAR FOLLOWERS: we are moving some items around in the repo!  There are “cactus” and “v1.0” tags in place so you can still build the current trees after we start the refactor.

We’ve got some big plans that I’ll outline on the list and earlier posts.

Right now, we’re working to modularize barclamps so that each one is in its own github repo.   This will allow you to pull in barclamps at build time or live on site. We’re also creating import/update routines that work for live systems to make it easier to develop barclamps.  Once again, that’s on the github modules branch. These will be exposed as rake barclamp:create[“foo”] and rake barclamp:install[“../foo”] type commands and I’ve committed to create some “how to make barclamps” videos.

That work is a prelude for a hard push on OpenStack Diablo before the design conference.  All that work will also be done in the github but the Diablo barclamps will be in independent repos from the Crowbar framework.

If you want to get started early.  80% of a barclamp effort is around the Chef Cookbooks.  Keith Hudgins with DTO did a great job writing up barclamps here: http://kb.dtosolutions.com/wiki/Deploying_the_cloudfoundry_barclamp.  We’re changing some of it to make it much easier and more modular.

WHIR Webinar Notes: Prying Open the Cloud with Dell Crowbar & OpenStack

Posted on by
2

Panelists: Me (@zehicle) & Joseph B. George (@jbgeorge), Director, Cloud and Big Data Solutions, Dell

Moderator: Liam Eagle (@theWHIR) , Editor-in-Chief, Web Host Industry Review

Wow, this Webinar was an hour of OpenStack insights (see the whole thing). If you don’t have the hour then you can use my time line nodes to jump to what you want to hear.

  • 2:50: Presentation Starts (introductions are over)
  • 3:40: Joseph coins the word “dynormous” for dynamic & large scale clouds
  • 4:40: Customers want to know how they are going to maintain a cloud
  • 4:50: Customers don’t want a 9 month cycle for features, want it faster. DevOps gives us the flexibility to meet our customer needs as quickly as they want to.
  • 7:11: Massive scalability… their (Rackspace & NASA) business is about scale
  • 8:00: Rackspace and NASA started from the beginning to build a community
  • 8:50: We have the data that this has staying power
  • 10:10: We see a lot of companies joining in the community
  • 11:56: Shout out to Opscode Chef
  • 12:40: From bare metal to a fully functioning cloud in under 2 hours. Crowbar allows you to introduce new elements into the environment
  • 13:40: Crowbar leverages our experience with cloud deployments
  • 14:33: Dell was the only provider there from day 1. We have the most experience.
  • 17:27: DevOps Poll
  • 18:40: DevOps is a significant trend that you should consider. Hosters have a lot of operational chops.
  • 19:34: There are a lot of right ways to do cloud. You need to pick what’s best for your business model
  • 20:23: We could get hardware and software, but operational expertise was missing.
  • 21:33: We’re more making the complexity of a cloud go away. We are getting our customers a head start. We are chipping away at the learning curve.
  • 22:05: The cloud is always ready, never finished. Cloud is an ongoing operational environment: DevOps!
  • 23:30: Crowbar bakes a lot of operational experience into the deployment.
  • 25:17: Core tenant of DevOps: there is no single OpenStack image. Cloud is too complex. We build it in layers.
  • 26:26: Before you deploy, you can change the configuration.
  • 27:30: Barclamps are modules that execute a function. We are inviting community participation
  • 28:40: Crowbar process view – Crowbar is a “PXE state machine” is a very simplified description.
  • 29:57: You can go through a tuning cycle where you can get it working, make sure it’s right, flush and reset. That ensures you have an automated system.
  • 30:34: Screen shots with descriptions
  • 33:25: Event the core state machine that runs Crowbar is deployed as a barclamp
  • 35:00: You can download OpenStack and install it yourself from our github. We don’t want to talk about OpenStack, we want to DO OpenStack.
  • Poll Results (see to the right)
  • 38:00: Online resources
  • 40:00: Question 1: Timeline for RHEL. Answer: RHEL is part of Hadoop, will make it into OpenStack by end of year (or sooner based on market demand)
  • 42:17: Question 2: What led Dell to get involved in OpenStack? Answer: It’s about experience. We like being able to fix and change if we needed. There is a lot of active community
  • 45:30: Question 3: How does a hardware maker play with open source software? Answer: It’s a solution for us. We wanted to make sure that people cloud deploy the software. Adding DevOps takes it to another level.
  • 48:00 Question 4: What elements of Diablo are most exciting? Answer: Keystone (centralized authentication) is a big deal. Networking changes that “bust the top” of the networking hurdles.
  • 50:25: Question 5: Where is OpenStack going long term? Answer: We’re pleasantly surprised about how much it’s picked up. We’ll see more standards in the community. We have high hopes for OpenStack and have invested heavily. We’ll see more as-a-service capabilities to build on a common infrastructure: both open and commercial.
  • 52:47: Question 6: What’s the biggest barrier to operating at scale? Answer: learning how to operate is the biggest hurdle. We took a learning approach to help customers get started. We are hosting a training with Rackspace.
  • 55:00: Question 7: Where does Dell and Rackspace overlap? Answer: We see Rackspace Cloud Builders that the premier experts. Dell Services is involved with all of it. Dell takes the phone call and deals with our customers directly.

Please support me for the OpenStack Policy Board

Posted on by
4

I’m posting my OpenStack bio here and asking for support putting me on the Policy Board by voting for me.  NOTE: You can only vote if you’re registered and you got the “Poll: OpenStack Governance Elections” email.

Project Policy Board Objective

I am seeking a role on the OpenStack Policy Board to further the adoption of OpenStack within and beyond the community.  As the OpenStack technology lead within Dell, I am the engineer who is most actively engaged with field deployments; consequently, I am uniquely positioned to represent our development community, hosters and enterprise user bases.  I bring substantial process experience (Agile/Lean/CI) into my decision making.  My focus will be on ensuring OpenStack is deployable and ready for use.

Background

I am a Principal Engineer at Dell working as the lead for our OpenStack cloud initiative (http://dell.com/openstack).  My team at Dell is responsible for bringing hyper-scale cloud solutions to market and works closely with our cloud optimized hardware division (DCS).  Before working on the OpenStack project, I was involved in cloud projects for Azure, Eucalyptus, and Joyent at Dell.
My involvement with OpenStack goes back to the very earliest days before the project was launched where I was part of the evaluation team that advocated for Dell to join the project.  Since then, I have been active participant at every design conference.  It was my recommendation that Dell focus on making deployment capabilities for OpenStack and to ensure that those contributions are open sourced (Apache 2).  At this point in the project, I am Dell’s technical authority on OpenStack for community and customer interactions.
My team is responsible for the Crowbar cloud deployer (http://github.com/dellcloudedge/crowbar).  The purpose of this project is to ensure that OpenStack is be quickly and reliably deployed in a wide range of configurations on any hardware platform.  I believe that ease of deployability is essential for the success of OpenStack as a project because it ensure adoption by non-developers.  I also believe strongly in continuous integration and am working to adapt Crowbar as a CI platform.  I have been the primary driver to ensure that the Crowbar project is open sourced and accepting of input from the community.
My team also designs technical reference architectures (RAs) for OpenStack.  These RAs help drive adoption by providing crisp guidance on how deploy OpenStack.  I am a vocal proponent of open operations (keeping best practices public) and following a DevOps approach for ongoing cloud deployment life-cycles.
In addition to my work at Dell, I work to ensure community access and communication.  My independent blog provides technical detail and insights about the OpenStack and other cloud initiatives.  My blog also focuses on Agile and Lean practices that I believe are essential to success in technology innovation.
I have been working with cloud computing since 2001.  The company I founded with Dave McCrory (@mccrory), now owned by Quest, was the first multi-server VMware ESX deployment ouside of VMware.  We pioneered the concept of elastic vm management (look up the patents!) so I have a very deep understanding of the problems and architectures required.

OpenStack Design Summit regististration open for DEVS! See you in October

Posted on by
Reply

Registration for the OpenStack Conference (which my team at Dell is co-sponsoring), which is now a separate event (from the User Conference the same week) with a separate registration mechanism.

From the OpenStack List:

The Essex Design Summit is an event for the OpenStack developers community (existing and prospective coders) to gather and discuss the upcoming development cycle. It is highly technical in nature and is not targeted to the general OpenStack public, which will find the OpenStack Conference much more enjoyable.

If you think you belong in the former category, please join us. The Summit is free to attend, but with limited attendance, so we require registration.

More details at: http://wiki.openstack.org/Summit/Essex

Crowbar near-term features: increasing DevOps mojo and brewing Diablo

Posted on by
3

We’ve been so busy working on getting RHEL support ready to drop into the Crowbar repos that I have not had time to post about what’s coming next for Crowbar. The RHEL addition has required a substantial amount of work to accommodate different packaging models and capabilities. This change moves Crowbar closer to being able allow nodes’ operating systems (the allocated TFTP Boot Image) to be unique per node.

I will post more forward looking details soon but wanted to prime the pump and invite suggestions from our community.

We are tracking two major features for delivery by the OpenStack October Design Conference

  1. OpenStack Diablo Barclamps. Expect to see individual barclamps for various components like Keystone, Dashboard, Glace, Nova, Swift, etc)
  2. Barclamp versioning / connected imports. This feature will enable Crowbar to pull in the latest components for barclamps from remote repositories. I consider this a critical feature for Crowbar’s core DevOps/CloudOps capabilities and to support more community development for barclamps.

We are also working on some UI enhancements

  • Merging together the barclamps/proposals/active views into a single view
  • Enabling bulk actions for nodes (description, BIOS types, and allocate)
  • Allowing users to set node names and showing the names throughout the UI
  • More clarity on state of proposal application process (stretch goal)

I am planning to post more about our design ideas as work begins.

If you want to help with Diablo barclamps, these will be worked in the open and we’d be happy to collaborate. We’re also open to suggestion for what’s next.

Cloudcast interview with Dell Cloud Solutions Team (quotes with time stamps)

Posted on by
Reply

Lloyds BarbershopTheCloudcast.net – Thank you for such a great series of questions. Wow, nearly 36+ minutes of cloudicious interview about the work my team at Dell is doing!

Thanks to our hosts for putting together a great series! They are:

Highlights from Episode 16: Dell, Dude you’re getting a cloud

  • 3:40 JBG “we are listening to our customers tell us what they want to accomplish”
  • 4:40 RAH “humility is part of [what we’re] doing … cloud is about learning and collaboration”
  • 6:40 RAH “OpenStack filled a niche. It was the first open source community cloud. … Not just open source, its open community.”
  • 7:15 RAH “We’re beyond critical mass. We’re seeing acceleration… we are transitioning into a community development.”
  • 7:30 RAH “It’s accelerating. It happening so fast.”
  • 8:00 RAH “We felt it was really important for people to be able to use it. We felt that it was important to get away from just people developing into people using. “
  • 8:57 – RAH “Cloud is not just one thing. You have to have all the pieces.”
  • 10: 15 – RAH “Cloud is always ready, never finished”
  • 10:50 – RAH “OpenStack is an alternative to public cloud including hosting providers seeking to offer their own cloud”
  • 12:40 – AD “Dell has been in the Big data space for many years now”
  • 20:15 – JBG “There’s a legacy of great partnerships that we leverage”
  • 20:48 – JBG “Conflicts have not come up because we are focused on the customer”
  • 21:30 – RAH “Shout out to Greg Althaus for solving these problems in such an elegant way. And we rewrote it 3 times”
  • 22:02 – RAH “Crowbar started from our frustration of bringing up a cloud quickly … so we took a DevOps approach.”
  • 22:41 – RAH “You had to have a system view AND a boot strapping view simultaneously”
  • 23:50 – JBG “Crowbar was born out of necessity because we were setting up and blowing away our clouds over and over and over again. “
  • 24:40 – JBG “We realized there were not many people thinking about all the pieces before OpenStack was installed”
  • 25:20 – RAH “We don’t think customers have all the answers before we show up. This is not unique to OpenStack.”
  • 28:20 – JBG “We’re seeing the community pick up Crowbar as a way to deploy”

Collaboration between Dell Crowbar & VMware Cloud Foundry – unleashes your inner cloud

Posted on by
12

Sometimes a single sprint can deliver magic: when I signed up to document how to create a Crowbar module (aka a barclamp) two weeks ago, I had no idea that it would add a new flavor to Crowbar .

I’m proud to announce that the first public non-Dell Crowbar module will be supporting the VMware Cloud Foundry Open PaaS project.

Development is still in progress (on the Crowbar “CF” branch) and you’ll be able to watch us (even help!) collaborate on this project.  Initially, the deployment will be to single server but we’re hoping to quickly expand to a distributed install that fully leverages the capabilities of both projects.

By creating a Crowbar module, Cloud Foundry™ is able to leverage the cloud deployment capabilities that allow it to be setup on any physical or virtualized data center.  This is core to the Crowbar message: the value of a cloud solution can best be realized when it’s coupled with open practices for deploying it.

There are many significant aspects of this collaboration:

  1. Cloud Foundry is taking the right approach to PaaS.  Their team’s perspective on PaaS mirrors my own: A PaaS is a collection of application services.  That approach makes it extensible and flexible.  Plus, they are also multi-language and multi-platform.
  2. Crowbar is proving our breadth of support.  Last week we announced coming RHEL support and now adding Cloud Foundry is a natural extension.  We did not design Crowbar to be a one-trick pony.  It’s modular design makes it easy to extend while leveraging the existing body of work.
  3. Big companies are acting like start-ups.  Both Crowbar and Cloud Foundry are projects that focus on putting the core functionality out quickly to prove their value proposition, get feedback, and change the game.  This collaboration is positive proof of these companies being Agile and starting a project Lean.
  4. Big companies are acting in the open.  Both Dell via Crowbar and VMware via Cloud Foundry are contributing their source and working on it in the open.

Stay tuned for that “how to create a barclamp” post (or check out the barclamp rake task).

For more information:

Video of Crowbar Install & Introduction (two 15 min videos)

Posted on by
17

These Crowbar videos are the first two in a series of how to setup and use your own local Crowbar dev environment (here’s more info & the ISO).   I used VMware Workstation, but any virtual hosts that support Ubuntu 10.10 will work fine.  We use ESX, KVM and Xen for testing too.

Creating this environment is the basis for learning Crowbar, experimenting with OpenStack and creating your own barclamps.  It’s also a handy way to play with Opscode Chef since it includes a stand alone Chef server.

Some items from the install video that I want to re-emphasize:

  • The admin server REQUIRES >1 GB of RAM (more is better)
  • All other nodes REQUIRE > 512 MB of RAM (more depending on what you install)
  • At least TWO NICS are required.
  • You must disable DHCP on the virtual network because Crowbar has a DHCP server and they will conflict.
  • The login for the Crowbar admin server is openstack/openstack.  You must “sudo -i” before you run the install script.
  • The default url for Crowbar is http://192.168.124.10:3000  (crowbar/crowbar)

Installing Crowbar (15 mins)

Introduction to Crowbar (15 mins)

9/30 Bonus Video: How to change the default networking

Continue reading →

Big Questions? Big Answers with Dell BigData solution (plus Crowbar gets RHEL)

Posted on by
6

In my enthusiasm for all things Dell + OpenStack, I have neglected to talk about my team’s interesting Big Data work with Apache Hadoop.  Hadoop is a suite of open source projects for analyzing large data sets of unstructured data.  Initially, Hadoop centered around use of the map-reduce algorithm; however, it’s grown way beyond that as the community has worked to solve problems related to data storage, discovery, and scheduling.

Big Data clouds are well suited to my team because the model (non-redundant/cloud) and scale (hyper) of their deployments.  It should be no surprise that builders of analysis clouds have the same goals (maximizing operational ROI per compute unit) as builders of other types of clouds.

Our Hadoop solution relies on the same core principles (CloudOps) and technologies (Crowbar) as our OpenStack solution.  Like our other cloud solutions, we are working closely with a proven leader: Cloudera.  Now that we’ve formally announced our solution and partnership, I can talk a about what we’re doing on the Big Data front.

One extra thing that I’m proud to announce, we’ll be adding Red Hat Enterprise Linux (RHEL) support to Crowbar to support our Hadoop solution.  This support is not just at the node level: we are making Crowbar admin run on either platform too!  This is significant for two reasons:

  1. It expands the number of platforms and support options for Crowbar users
  2. It provides the framework to support more varieties of node operating environment (e.g.: XenServer, BSD, DRDOS, etc)

For more information, check out:

Build Sledgehammer, the Crowbar discovery image / build prerequisite

Posted on by
2
Note: This content has been copied to the Crowbar Wiki.
Victor “got your back” Lowther, CI & build automation czar on our team at Dell, spent a lot of time cleaning up the open source build to make it MUCH easier.  The latest build only requires ONE server for all components.  To make it repeatable and fast, I’m using a hosted VM from Rackspace Cloud.
Here are the steps that you should follow (cool: if you build before the prereqs are in place, the script will tell you what’s missing).
Note: You must build the discovery image (build_sledgehammer.sh) before building Crowbar.  This image does not change very often, so it’s helpful to cache it somewhere (like in the Crowbar cache where it normally lives) and save time.
  1. Starting from a Rackspace Cloud Ubuntu 10.10 image (512 RAM is OK, $0.03/hr)
  2. Get libraries for git, RPM, & Ruby: apt-get install git rpm ruby
  3. Get the sledgehammer repo: git clone git://github.com/dellcloudedge/crowbar-sledgehammer.git
  4. Go to sledgehammer: cd crowbar-sledgehammer
  5. Download the CentOS image: curl -o ../CentOS-5.6-x86_64-bin-DVD-1of2.iso http://mirror.cs.vt.edu/pub/CentOS/5.6/isos/x86_64/CentOS-5.6-x86_64-bin-DVD-1of2.iso
    1. takes some time (10+ mins) even in the cloud
  6. Tell the build where to look for the CentOS image: CENTOS_ISO=~/CentOS-5.6-x86_64-bin-DVD-1of2.iso ./build_sledgehammer.sh
    1. you may need to change the path of the image if you did not put it in your home directory
    1. wait a long time while magic happens and the tar gets created
    2. check out the tar ball in the /bin directory!
  7. Create the cache location for Sledgehammer: mkdir -p ~/.crowbar-build-cache
  8. Move the the cache location: cd ~/.crowbar-build-cache
  9. Extract the Sledgehammer tar: tar xzvf ~/crowbar-sledgehammer/bin/sledgehammer-tftpboot.tar.gz 
Or, use the tar copy that I’ve cached it on zehicle.com!  Then you can start at step 8.
Now you can build crowbar as per instructions (duplicated below)
  1. cd ~
  2. git clone git://github.com/dellcloudedge/crowbar.git
  3. apt-get update
  4. apt-get install build-essential mkisofs debootstrap
  5. crowbar/build_crowbar.sh
    1. kicks off a long download to create the cache (first time only!)
    2. look in the home directory for the openstack-dev.iso

Of course, you still need to INSTALL CROWBAR (as root, /tftpboot/ubuntu_dvd/extra/install) after you use the ISO to boot a VM.  Instructions on that shortly…