For the Paris summit, the OpenCrowbar team delivered a PackStack demo that leveraged Crowbar’s ability to create a OpenStack ready state environment. For the Vancouver summit, we did something even bigger: we updated the OpenCrowbar Ceph workload.
Ceph is the leading open source block storage back-end for OpenStack; however, it’s tricky to install and few vendors invest the effort to hardware optimize their configuration. Like any foundation layer, configuration or performance errors in the storage layer will impact the entire system. Further, the Ceph infrastructure needs to be built before OpenStack is installed.
OpenCrowbar was designed to deploy platforms like Ceph. It has detailed knowledge of the physical infrastructure and sufficient orchestration to synchronize Ceph Mon cluster bring-up.
We are only at the start of the Ceph install journey. Today, you can use the open source components to bring up a Ceph cluster in a reliable way that works across hardware vendors. Much remains to optimize and tune this configuration to take advantage of SSDs, non-Centos environments and more.
We’d love to work with you to tune and extend this workload! Please join us in the OpenCrowbar community.
Overall, I’m happy with our three days of hacking on Crowbar 2. We’ve reached the critical “deploys workload” milestone and I’m excited about well the design is working and how clearly we’ve been able to articulate our approach in code & UI.
Of course, it’s worth noting again that Crowbar 1 has also had significant progress on OpenStack Havana workloads running on Ubuntu, Centos/RHEL, and SUSE/SLES
Here are the focus items from the hack:
- Documentation – cleaned up documentation specifically by updating the README in all the projects to point to the real documentation in an effort to help people find useful information faster. Reminder: if unsure, put documentation in barclamp-crowbar/doc!
- Docker Integration for Crowbar 2 progress. You can now install Docker from internal packages on an admin node. We have a strategy for allowing containers be workload nodes.
- Ceph installed as workload is working. This workload revealed the need for UI improvements and additional flags for roles (hello “cluster”)
- Progress on OpenSUSE and Fedora as Crowbar 2 install targets. This gets us closer to true multi-O/S support.
- OpenSUSE 13.1 setup as a dev environment including tests. This is a target working environment.
- Being 12 hours offset from the US really impacted remote participation.
One thing that became obvious during the hack is that we’ve reached a point in Crowbar 2 development where it makes sense to move the work into distinct repositories. There are build, organization and packaging changes that would simplify Crowbar 2 and make it easier to start using; however, we’ve been trying to maintain backwards compatibility with Crowbar 1. This is becoming impossible; consequently, it appears time to split them. Here are some items for consideration:
- Crowbar 2 could collect barclamps into larger “workload” repos so there would be far fewer repos (although possibly still barclamps within a workload). For example, there would be a “core” set that includes all the current CB2 barclamps. OpenStack, Ceph and Hadoop would be their own sets.
- Crowbar 2 would have a clearly named “build” or “tools” repo instead of having it called “crowbar”
- Crowbar 2 framework would be either part of “core” or called “framework”
- We would put these in a new organization (“Crowbar2” or “Crowbar-2”) so that the clutter of Crowbar’s current organization is avoided.
While we clearly need to break apart the repo, this suggestion needs community more discussion!
The Crowbar community has a tradition of “day zero ops” community support for the latest OpenStack release at the summit using our pull-from-source capability. This release we’ve really gone the extra mile by doing it one THREE Linux distros (Ubuntu, RHEL & SLES) in parallel with a significant number of projects and new capabilities included.
I’m especially excited about Crowbar implementation of Havana Docker support which required advanced configuration with Nova and Glance. The community also added Heat and Celiometer in the last release cycle plus High Availability (“Titanium”) deployment work is in active development. Did I mention that Crowbar is rocking OpenStack deployments? No, because it’s redundant to mention that. We’ll upload ISOs of this work for easy access later in the week.
While my team at Dell remains a significant contributor to this work, I’m proud to point out to SUSE Cloud leadership and contributions also (including the new Ceph barclamp & integration). Crowbar has become a true multi-party framework!
Want to learn more? If you’re in Hong Kong, we are hosting a Crowbar Developer Community Meetup on Monday, November 4, 2013, 9:00 AM to 12:00 PM (HKT) in the SkyCity Marriott SkyZone Meeting Room. Dell, dotCloud/Docker, SUSE and others will lead a lively technical session to review and discuss the latest updates, advantages and future plans for the Crowbar Operations Platform. You can expect to see some live code demos, and participate in a review of the results of a recent Crowbar 2 hackathon. Confirm your seat here – space is limited! (I expect that we’ll also stream this event using Google Hangout, watch Twitter #Crowbar for the feed)
My team at Dell has a significant presence at the OpenStack Summit in Hong Kong (details about activities including sponsored parties). Be sure to seek out my fellow OpenStack Board Member Joseph George, Dell OpenStack Product Manager Kamesh Pemmaraju and Enstratius/Dell Multi-Cloud Manager Founder George Reese.
Note: The work referenced in this post is about Crowbar v1. We’ve also reached critical milestones with Crowbar v2 and will begin implementing Havana on that platform shortly.
SUSE’s be a great development partner for Crowbar and an active member of the OpenStack community. I’m excited to see them giving a live demo today about their OpenStack technology stack (which includes Crowbar and Ceph).
Register for the Live Demo on Wed 06-26-2013 at 3.00 – 4.00 pm GMT to “learn about SUSE’s OpenStack distribution: SUSE Cloud with Dell Crowbar as the deployment mechanism and advanced features such as Ceph unified storage platform for object, block and file storage in the cloud.”
The presenter, Rick Ashford, lives in Austin and is a regular at the OpenStack Austin Meetups. He has been working with Linux and open-source software since 1998 and currently specializes in the OpenStack cloud platform and the SUSE ecosystem surrounding it.
Whew….Yesterday, Dell announced TWO OpenStack block storage capabilities (Equallogic & Ceph) for our OpenStack Essex Solution (I’m on the Dell OpenStack/Crowbar team) and community edition. The addition of block storage effectively fills the “persistent storage” gap in the solution. I’m quadrupally excited because we now have:
- both open source (Ceph) and enterprise (Equallogic) choices
- both Nova drivers’ code is in the open at part of our open source Crowbar work
Frankly, I’ve been having trouble sitting on the news until Dell World because both features have been available in Github before the announcement (EQLX and Ceph-Barclamp). Such is the emerging intersection of corporate marketing and open source.
As you may expect, we are delivering them through Crowbar; however, we’ve already had customers pickup the EQLX code and apply it without Crowbar.
The Equallogic+Nova Connector
If you are using Crowbar 1.5 (Essex 2) then you already have the code! Of course, you still need to have the admin information for your SAN – we did not automate the configuration of the storage system, but the Nova Volume integration.
We have it under a split test so you need to do the following to enable the configuration options:
- Install OpenStack as normal
- Create the Nova proposal
- Enter “Raw” Attribute Mode
- Change the “volume_type” to “eqlx”
- The Equallogic options should be available in the custom attribute editor! (of course, you can edit in raw mode too)
Want Docs? Got them! Check out these > EQLX Driver Install Addendum
Usage note: the integration uses SSH sessions. It has been performance tested but not been tested at scale.
The Ceph+Nova Connector
The Ceph capability includes a Ceph barclamp! That means that all the work to setup and configure Ceph is done automatically done by Crowbar. Even better, their Nova barclamp (Ceph provides it from their site) will automatically find the Ceph proposal and link the components together!
Note: Cross posted on Dell Tech Center Blogs.
Background: Crowbar is an open source cloud deployment framework originally developed by Dell to support our OpenStack and Hadoop powered solutions. Recently, it’s scope has increased to include a DevOps operations model and other deployments for additional cloud applications.
It’s only been a matter of months since we open sourced the Dell Crowbar Project at OSCON in June 2011; however, the progress and response to the project has been over whelming. Crowbar is transforming into a community tool that is hardware, operating system, and application agnostic. With that in mind, it’s time for me to provide a recap of Crowbar for those just learning about the project.
Crowbar started out simply as an installer for the “Dell OpenStack™-Powered Cloud Solution” with the objective of deploying a cloud from unboxed servers to a completely functioning system in under four hours. That meant doing all the BIOS, RAID, Operations services (DNS, NTP, DHCP, etc.), networking, O/S installs and system configuration required creating a complete cloud infrastructure. It was a big job, but one that we’d been piecing together on earlier cloud installation projects. A key part of the project involved collaborating with Opscode Chef Server on the many system configuration tasks. Ultimately, we met and exceeded the target with a complete OpenStack install in less than two hours.
In the process of delivering Crowbar as an installer, we realized that Chef, and tools like it, were part of a larger cloud movement known as DevOps.
The DevOps approach to deployment builds up systems in a layered model rather than using packaged images. This layered model means that parts of the system are relatively independent and highly flexible. Users can choose which components of the system they want to deploy and where to place those components. For example, Crowbar deploys Nagios by default, but users can disable that component in favor of their own monitoring system. It also allows for new components to identify that Nagios is available and automatically register themselves as clients and setup application specific profiles. In this way, Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.
We believe that operations that embrace layered deployments are essential for success because they allow our customers to respond to the accelerating pace of change. We call this model for cloud data centers “CloudOps.”
Based on the flexibility of Crowbar, our team decided to use it as the deployment model for our Apache™ Hadoop™ project (“Dell | Apache Hadoop Solution”). While a good fit, adding Hadoop required expanding Crowbar in several critical ways.
- We had to make major changes in our installation and build processes to accommodate multi-operating system support (RHEL 5.6 and Ubuntu 10.10 as of Oct 2011).
- We introduced a modularization concept that we call “barclamps” that package individual layers of the deployment infrastructure. These barclamps reach from the lowest system levels (IPMI, BIOS, and RAID) to the highest (OpenStack and Hadoop).
Barclamps are a very significant architecture pattern for Crowbar:
- They allow other applications to plug into the framework and leverage other barclamps in the solution. For example, VMware created a Cloud Foundry barclamp and Dream Host has created a Ceph barclamp. Both barclamps are examples of applications that can leverage Crowbar for a repeatable and predictable cloud deployment.
- They are independent modules with their own life cycle. Each one has its own code repository and can be imported into a live system after initial deployment. This allows customers to expand and manage their system after initial deployment.
- They have many components such as Chef Cookbooks, custom UI for configuration, dependency graphs, and even localization support.
- They offer services that other barclamps can consume. The Network barclamp delivers many essential services for bootstrapping clouds including IP allocation, NIC teaming, and node VLAN configuration.
- They can provide extensible logic to evaluate a system and make deployment recommendations. So far, no barclamps have implemented more than the most basic proposals; however, they have the potential for much richer analysis.
Making these changes was a substantial investment by Dell, but it greatly expands the community’s ability to participate in Crowbar development. We believe these changes were essential to our team’s core values of open and collaborative development.
Most recently, our team moved Crowbar development into the open. This change was reflected in our work on OpenStack Diablo (+ Keystone and Dashboard) with contributions by Opscode and Rackspace Cloud Builders. Rather than work internally and push updates at milestones, we are now coding directly from the Crowbar repositories on Github. It is important to note that for licensing reasons, Dell has not open sourced the optional BIOS and RAID barclamps. This level of openness better positions us to collaborate with the crowbar community.
For a young project, we’re very proud of the progress that we’ve made with Crowbar. We are starting a new chapter that brings new challenges such as expanding community involvement, roadmap transparency, and growing Dell support capabilities. You will also begin to see optional barclamps that interact with proprietary and licensed hardware and software. All of these changes are part of growing Crowbar in framework that can support a vibrant and rich ecosystem.
We are doing everything we can to make it easy to become part of the Crowbar community. Please join our mailing list, download the open source code or ISO, create a barclamp, and make your voice heard. Since Dell is funding the core development on this project, contacting your Dell salesperson and telling them how much you appreciate our efforts goes a long way too.