Getting the core Crowbar 2.0 changes working is not a major refactoring effort in calendar time; however, it will impact current Crowbar developers by changing improving the programming APIs. The Dell Crowbar team decided to treat this as a focused refactoring effort because several important changes are tightly coupled. We cannot solve them independently without causing a larger disruption.
All of the Crowbar 2.0 changes address issues and concerns raised in the community and are needed to support expanding of our OpenStack and Hadoop application deployments.
Our technical objective for Crowbar 2.0 is to simplify and streamline development efforts as the development and user community grows. We are seeking to:
simplify our use of Chef and eliminate Crowbar requirements in our Opscode Chef recipes.
reduce the initial effort required to leverage Crowbar
provide heterogeneous / multiple operating system deployments. This enables:
multiple versions of the same OS running for upgrades
different operating systems operating simultaneously (and deal with heterogeneous packaging issues)
accommodation of no-agent systems like locked systems (e.g.: virtualization hosts) and switches (aka external entities)
UEFI booting in Sledgehammer
strengthen networking abstractions
allow networking configurations to be created dynamically (so that users are not locked into choices made before Crowbar deployment)
better manage connected operations
enable pull-from-source deployments that are ahead of (or forked from) available packages.
improvements in Crowbar’s core database and state machine to enable
larger scale concerns
controlled production migrations and upgrades
other important items
make documentation more coupled to current features and easier to maintain
upgrade to Rails 3 to simplify code base, security and performance
deepen automated test coverage and capabilities
Beyond these great technical targets, we want Crowbar 2.0 is to address barriers to adoption that have been raised by our community, customers and partners. We have been tracking concerns about the learning curve for adding barclamps, complexity of networking configuration and packaging into a single ISO.
My team at Dell does not take on any refactoring changes lightly because they are disruptive to our community; however, a convergence of requirements has made it necessary to update several core components simultaneously. Specifically, we found that desired changes in networking, operating systems, packaging, configuration management, scale and hardware support all required interlocked changes. We have been bringing many of these changes into the code base in preparation and have reached a point where the next steps require changing Crowbar 1.0 semantics.
We are first and foremost an incremental architecture & lean development team – Crowbar 2.0 will have the smallest footprint needed to begin the transformations that are currently blocking us. There is significant room during and after the refactor for the community to shape Crowbar.
The response to Crowbar has been exciting and humbling. I most appreciate those who looked at Crowbar and saw more than a bare metal installer. They are the ones who recognized that we are trying to solve a bigger problem: it has been too difficult to cope with change in IT operations.
During this year, we have made many changes. Many have been driven by customer, user and partner feedback while others support Dell product delivery needs. Happily, these inputs are well aligned in intent if not always in timing.
Introduction of barclamps as modular components
Expansion into multiple applications (most notably OpenStack and Apache Hadoop)
Working in the open (with public commits)
Collaborative License Agreements
Dell‘s understanding of open source and open development has made a similar transformation. Crowbar was originally Apache 2 open sourced because we imagined it becoming part of the OpenStack project. While that ambition has faded, the practical benefits of open collaboration have proven to be substantial.
The results from this first year are compelling:
For OpenStack Diablo, coordination with the Rackspace Cloud Builder team enabled Crowbar to include the Keystone and Dashboard projects into Dell’s solution
We’ve amassed hundreds of mail subscribers and Github followers
Support for multiple releases of RHEL, Centos & Ubuntu including Ubuntu 12.04 while it was still in beta.
SuSE does their own port of Crowbar to SuSE with important advances in Crowbar’s install model (from ISO to package).
We stand on the edge of many exciting transformations for Crowbar’s second year. Based on the amount of change from this year, I’m hesitant to make long term predictions. Yet, just within next few months there are significant plans based on Crowbar 2.0 refactor. We have line of site to changes that expand our tool choices, improve networking, add operating systems and become more even production ops capable.
With the GA drop, the Crowbar Cloudera Barclamps are effectively at release candidate state (ISO). The Cloudera Barclamps include a freemium version of Cloudera Enterprise 4 that supports up to 50 nodes.
Last week, my team at Dell led a world-wide OpenStack Essex Deploy event. Kamesh Pemmaraju, our OpenStack-powered solution product manager, did a great summary of the event results (200+ attendees!). What started as a hack-a-thon for deploy scripts morphed into a stunning 14+ hour event with rotating intro content and an ecosystem showcase (videos). Special kudos to Kamesh, Andi Abes, Judd Maltin, Randy Perryman & Mike Pittaro for leadership at our regional sites.
Clearly, OpenStack is attracting a lot of interest. We’ve been investing time in content to help people who are curious about OpenStack to get started.
On that measure, we have room for improvement. We had some great discussions about how to handle upgrades and market drivers for OpenStack; however, we did not spend the time improving Essex deployments that I was hoping to achieve. I know it’s possible – I’ve talked with developers in the Crowbar community who want this.
If you wanted more expert interaction, here are some of my thoughts for future events.
Expert track did not get to deploy coding. I think that we need to simply focus more even tightly on to Crowbar deployments. That means having a Crowbar Hack with an OpenStack focus instead of vice versa.
Efforts to serve OpenStack n00bs did not protect time for experts. If we offer expert sessions then we won’t try to have parallel intro sessions. We’ll simply have to direct novices to the homework pages and videos.
Combining on-site and on-line is too confusing. As much as I enjoy meeting people face-to-face, I think we’d have a more skilled audience if we kept it online only.
Connectivity! Dropped connections, sigh.
Better planning for videos (not by the presenters) to make sure that we have good results on the expert track.
This event was too long. It’s just not practical to serve Europe, US and Asia in a single event. I think that 2-3 hours is a much more practical maximum. 10-12am Eastern or 6-8pm Pacific would be much more manageable.
Do you have other comments and suggestions? Please let me know!
One of my team at Dell’s most critical lessons from hyperscale cloud deployments was the DevOps tooling and operations processes are key to success. Our crowbar project was born out of this realization.
I have been tracking the progress the Copper ARM-based server from design to implementation internally. Now, I’m excited to see it getting some deserved attention.
The Copper platform is really cool because the cost, power, and density ratios of the nodes are unparalleled. This makes it an ideal platform for distributed mixed compute/store workloads like Hadoop. The nodes in the platform have excellent RAM/CPU/Spindle ratios.
While Copper is driving huge density, it also drives forward the same hyperscale challenges that we’ve been trying to address with Crowbar; consequently, we’re already working to ensure that we can deploy and manage Copper with Crowbar at scale.
Copper and Crowbar make a natural team and we’re excited to be part of today’s announcement:
Dell is staging clusters of the Dell “Copper” ARM server within the Dell Solution Centers and with TACC so developers may book time on the platforms. Dell also will deliver an ARM-supported version of Crowbar, Dell’s open-source management infrastructure software, to the industry in the future.
We are RECORDING everything and will link posts from the event page.
There is HOMEWORK if you want to get ahead by installing OpenStack yourself.
For last minute updates about the event, I recommend that you join the Crowbar Listserver.
Content Logistics work like this.
Everything will be available ONLINE. We are also coordinating many physical sites as rally points.
Introductory: FOUR 3-hour sessions for people who do not have OpenStack or Crowbar experience. These sessions will show how to install OpenStack using Crowbar, discuss DevOps and showcase companies that are in the OpenStack ecosystem. They are planned to have 2 European slots (afternoon & evening), 3 US slots (morning, afternoon & evening), and 1 Asian slot (morning).
Expert: ON-GOING deep technical sessions for engineers who have OpenStack and/or Crowbar experience. There will be one main screen and voice channel in which we are planning to highlight and discuss these topics in blocks throughout the day. We have a long list of topics to discuss and will maintain an ongoing Google Hangout for each topic. Depending on interest, we will jump back and forth to different hangouts.
Intro/Overview Session Logistics work like this
We’re planning FOUR introductory sessions throughout the day (read ahead?). Each session should be approximately 3 hours. The first hour of the sessions will be about OpenStack Essex and installing it using Crowbar. After some Q&A, we’re going to highlight the OpenStack ecosystem. The schedule for the ecosystem is in flux and will likely shift even during the event.
The Session start times for Overview & Ecosystem content
6/1 10 am
* There are no planned live venues at this time/region. You are always welcome to join online!
Experts Track Logistics
Note: we expect experts to have already installed OpenStack (see homework page). Ideally, an expert has already setup a build environment.
We have a list of topics (Essex, Quantum, Networking, Pull from Source, Documentation, etc) that we plan to cover on a 30-60 minute rotation.
We will cover the OpenStack Essex deploy at the start of each planned session (9am, Noon, 3pm & 8pm EDT). Before we cover the OpenStack deploy, we’ll spend 10 minutes setting (and posting) the agenda for the next three hours based on attendee input.
Even if we are not talking about a topic on the main channel, we will keep a dialog going on topic specific Google hangouts. The links to the hangouts will be posted with the Expert track agenda.
If you want to read more about the event, check out my event logistics post (link pending).
I do not apologize for my promotion of the Dell-lead open source Crowbar as the deployment tool for the OpenStack Essex Deploy. For a community to focus on improving deployment tooling, there must be a stable reference infrastructure. Crowbar provides a fast and repeatable multi-node environment with scriptable networking and packaging.
I believe that OpenStack benefits from a repeatable multi-node reference deployment. I’ll go further and state that this requires DevOps tooling to ensure consistency both within and between deployments.
DevStack makes trunk development more canonical between different developers. I hope that Crowbar will help provide a similar experience for operators so that we can truly share deployment experience and troubleshooting. I think it’s already realistic for Crowbar deployments to a repeatable enough deployment that they provide a reference for defect documentation and reproduction.
Said more plainly, it’s a good thing if a lot of us use OpenStack in the same way so that we can help each out.
We are planning many improvements to our OpenStack Essex and Crowbar Framework; however, most deployments can proceed without these enhancements. This also enables participants in the 5/31 OpenStack Essex Deploy Day.
By releasing a core stable Essex reference deployment, we are accelerating field deployments and enabling the OpenStack ecosystem. In terms of previous posts, we are eliminating release interlocks to enable more downstream development. Ultimately, we hope that we are also creating a baseline OpenStack deployment.
We are also reducing the pressure to rush more disruptive Crowbar changes (like enabling high availability, adding multiple operating systems, moving to Rails 3, fewer crowbarisms in cookbooks and streamlining networking). With this foundational Essex release behind us (we call it an MVP), we can work on more depth and breadth of capability in OpenStack.
One small challenge, some of the changes that we’d expected to drop have been postponed slightly. Specifically, markdown based documentation (/docs) and some new UI pages (/network/nodes, /nodes/families). All are already in the product under but not wired into the default UI (basically, a split test).
On the bright side, we did manage to expose 10g networking awareness for barclamps; however, we have notyet refactored to barclamps to leverage the change.
An OpenStack Deploy Hack-a-thon is like 3-liter bottle of distilled open source community love. Do you want direct access to my Dell team of OpenStack/Crowbar/Hadoop engineers? Are you just getting started and want training about OpenStack and DevOps? This is the event for you!
Here’s the official overview:
The OpenStack Deploy hack-a-thon focuses on automation for deploying OpenStack Essex with Dell Crowbar and Opscode Chef. This is a day-long, world-wide event bringing together developers, operators, users, ecosystem vendors and the open source cloud curious. (read below: We are looking for global sites and leaders to extend the event hours!)
OpenStack is the fastest growing open source cloud infrastructure project with broad market adoption from major hardware and software vendors. Crowbar is an Apache 2 licensed, open infrastructure deployment tool and is one of the leading multi-node deployers for OpenStack and Hadoop.
Learn first-hand how OpenStack and Crowbar can make it easy to deploy and operate your own cloud environments.
The Deploy day will offer two individual parallel tracks with something for both experts and beginners:
Newbies n00bs will learn the basics of OpenStack, Crowbar and DevOps and how they can benefit your organization. We’ll also have time for ecosystem vendors to discuss how they are leveraging OpenStack.
Experts l33ts will take a deep dive into new features of OpenStack Essex and Crowbar, and learn how Crowbar works under the hood, which will enable them to extend the product using Crowbar Barclamps.
Note: If you’re a n00b but want l33t content, we’ll be offering online training materials and videos to help get you up to speed.
Why now? We’ve validated our OpenStack Essex deployment against the latest release bits from Ubuntu. Now it’s time to reach out to the OpenStack and Crowbar communities for training, testing and collaborative development.