This release raises the bar on open Hadoop deployments by making them faster, scalable, more integrated and repeatable.
These barclamps were developed in conjunction with our licensed Dell | Cloudera Solution. The licensed solution is for customers seeking large scale and professionally supported big data solutions. The purpose of the open barclamps (which pull the open source parts from the Cloudera distro) is to help you get started with Hadoop and reduce your learning curve. Our team invested significant testing effort in ensuring that these barclamps work smoothly because they are the foundational layer of our for-pay Hadoop solution.
Included in the Hadoop barclamp suite are Hadoop Map Reduce, Hive, Pig, ZooKeeper and Sqoop running on RHEL 5.7. These barclamps cover the core parts of the Hadoop suite. Like other Crowbar deployments (see OpenStack), the barclamps automatically discover the service configurations and interoperate. One of our team members (call him Scott Jensen) said it very simply “I can deploy a fully an integrated Hadoop cluster in a few hours. That friggin’ rocks!” I just can’t put it more eloquently than that!
I’ll post again when we flip the “open” bit and invite our community to dig in and help us continue to set the standards on open Hadoop deployments.
Since some of you cannot make it to the show and see the demo in person, we’ve captured it as a video for your enjoyment. The OpenStack deployment is available in our open source distribution. We are currently in QA for the overall solution so expect additional refinement as we progress towards our next OpenStack solution release.
REMINDER: Dell Hardware is NOT required to use Crowbar for OpenStack. The open source version has everything you need – the BIOS and RAID barclamps are optional (but handy).
Note: I’m putting build ISOs and Sledgehammer TARs on crowbar.zehicle.com if you don’t want to follow these steps then download the ISO. We are updating the ISO daily, so don’t assume that you have that latest!
To build Crowbar, you need a Linux machine and access to the internet. The video shows how you can use an Ubuntu 10.10 Rackspace Cloud Server. We build Crowbar inside our firewall on our PCs too. No matter how you do it, Crowbar is full of fuzzily delicious cloud bits.
These Crowbar videos are the first two in a series of how to setup and use your own local Crowbar dev environment (here’s more info & the ISO). I used VMware Workstation, but any virtual hosts that support Ubuntu 10.10 will work fine. We use ESX, KVM and Xen for testing too.
Creating this environment is the basis for learning Crowbar, experimenting with OpenStack and creating your own barclamps. It’s also a handy way to play with Opscode Chef since it includes a stand alone Chef server.
Some items from the install video that I want to re-emphasize:
The admin server REQUIRES >1 GB of RAM (more is better)
All other nodes REQUIRE > 512 MB of RAM (more depending on what you install)
At least TWO NICS are required.
You must disable DHCP on the virtual network because Crowbar has a DHCP server and they will conflict.
The login for the Crowbar admin server is openstack/openstack. You must “sudo -i” before you run the install script.