I’m not usually a big fan of launch videos (too much markitecture); however, these turned out to be nice and meaty. The meaty part explains why it looks like I’m about to eat a big sandwich in the last video. yum!
I’m delighted to announce (official version) that my team at Dell has opened the Crowbar source under the Apache 2 license. This action is part of the broader Dell OpenStack Cloud Solution which includes OpenStack install packages, Crowbar, reference hardware architectures, and services/consulting to support deployments.
There are two important components to this news:
Both are significant items; however, my focus here is on the Crowbar release.
Crowbar started as a Dell OpenStack installer project and then grew beyond that in scope. Now it can be extended to work with other vendors’ kits and other solutions bits.
We are contributing Crowbar to the community because we believe that everyone benefits by sharing in the operational practices that Crowbar embodies. These are rooted in Opscsode Chef (which Crowbar tightly integrates with) and the cloud & hyper-scale proven DevOps practices that are reflected in our deployment model.
Where to get it?
- Code: Github (Apache 2 license)
- Community: Dell Crowbar listserv,
- Information: wiki on Crowbar Github, Crowbar User’s Guide and RobHirschfeld.com web site.
What’s included?
- A comprehensive set of barclamps to set up an OpenStack cloud.
- Crowbar UI and Remote APIs to make it easy to set up your cloud
- Automated testing scripts for community members doing continuous integration with OpenStack.
- Build scripts so you can create your own Crowbar install ISO
- Switch discovery so you can create Chef Cookbooks that are network aware.
- Open source Chef server that powers much of Crowar’s functionality
What’s not included?
- Non-open source license components (BIOS+RAID config) that we could not distribute under the Apache 2 license. We are working to address this and include them in our release. They are available in the Dell Licensed version of Crowbar.
- Dell Branded Components (skin + overview page). Crowbar has an OpenSource skin with identical functionality.
- Pre-built ISOs with install images (you must download the open source components yourself, we cannot redistribute them to you as a package)
Important notes:
On the project roadmap:
Sites for more information: Joseph George, Barton George (launch day), Dell
Original PostJust after we’d started deep Crowbar development, Andi Abes, Paul Webster and Victor Lowther joined the Dell Crowbar+OpenStack team. They immediately started to dig into our Swift, BIOS/RAID, and Network components. They also started to bump into each other in our original code base. It quickly became apparent that we needed to modularize Crowbar.
Restructuring Crowbar into modules has proved essential as a method for safe community collaboration.
Greg Althaus coined the name “barclamps” during the modularization rearchitecture. A barclamp is a class extension of the Crowbar ServiceObject that allows Crowbar to identify the Chef components used by the barclam
p (name p
attern in Chef is bc-template-[barclamp]) and provides capabilities that are specific to each barclamp.
When looking at the barclamps, it is critical to understand that even the most core Crowbar functionality is expressed as a barclamp.
Consequently, the core logic of the state machine, networking configuration, and provisioning are all exposed in barclamps. This makes it possible to modify and extend the most basic Crowbar operations; however, there are currently no guards against breaking these barclamps either!
| Barclamp | Function | Included |
| Crowbar | The roles and recipes to set up the barclamp framework. | Yes |
| Deployer | Initial classification system for the Crowbar environment (aka the state machine) | Yes |
| Provisioner | The roles and recipes to set up the provisioning server and a base environment for all nodes | Yes |
| Network | Instantiates network interfaces on the crowbar managed systems. Also manages the address pool. | Yes |
| NTP | Common NTP service for the cluster. An NTP server or servers can be specified and all other nodes will be clients of them. | Yes |
| DNS | manages the DNS subsystem for the cluste | Yes |
| Logging | centralized logging system based on syslog | Yes |
| IPMI | Integrates with IP management to allow direct hardware control bypassing the operating system. | Yes |
| RAID | LSI Licensed components. Cannot be included in open source release at this time. | No |
| BIOS |
PowerEdge C series: Dell License component. Cannot be included in open source release at this time. | No |
| Ganglia | Optional: a common Ganglia service for the cluster that can be used by other barclamps | Yes |
| Nagios | Optional : common monitoring service for the cluster that can be used by other barclamps | Yes |
| Nova | OpenStack: installs and configures the Openstack Nova (Cactus Release) component. It relies upon the network and glance barclamps for normal operation. | Yes |
| Swift | OpenStack: part of Openstack (Cactus Release) , and provides a distributed blob storage | Yes |
| Glance | OpenStack: Glance service (Cactus Release, Nova image management) for the cloud | Yes |
| Test | provides a shell for writing tests against | Yes |
My whole team is working feverishly on the final touches of Crowbar before we turn over the keys. We’re putting it through a complete release cycle (extensive QA, customer pilots, documentation, etc) because internal Dell consumers are expecting that level of finish.
For those in the community eagerly waiting to see the code, I hope you like the extra polish (for example: I18N, user & deployment guides, bundled continuous integration scripts, and months of testing).
RUMOR CONTROL NOTE: Crowbar is NOT limited to deployments on Dell products!! Our BIOS and RAID barclamps are, of course, targeted and licensed for Dell customers. The OpenStack and other barclamps will work on any gear that can run Chef Client.
Tonight I was working on the user guide and thought I would share the graphic and text describing how a barclamp gets deployed.
The figure shows the entire of a barclamp within the Crowbar user interface. A Barclamp defines the capability for a service but cannot be deployed. To deploy a barclamp, you must create a Proposal. Once the proposal is created, you must selection nodes to operate on. As discussed in the next sections, you may also edit the Proposal’s attributes as needed.
Applying the Proposal tells Crowbar to deploy the proposal onto the nodes. While deploying, nodes return to the Ready state when deployment is completed. Once a proposal has become an Active Role, you cannot edit it. You must delete the Role and repeat the Apply process
It’s awesome to see new deployments of OpenStack so I wanted to point out Cybera’s post about their OpenStack efforts!
Everett Toews does a nice job talking about the rationale for their decisions including some analysis of their hardware and vendor selections. Of course, I’m also happy to have them posting links back to my Dell team’s white paper and content.
I wanted to highlight one point that Everett makes:
“Is this the best mix of hardware possible for OpenStack? As always the answer is, “It depends.” It depends primarily on your the use cases for your cloud. We think we got a good mix of hardware but time will truely tell if it was the best mix possible for DAIR.”
I strongly agree, we (Dell) are still recommending starting with a smaller set general purpose hardware config that can be easily repurposed. Once you’ve figured out how your application maps into OpenStack then we’ll be ready to work togther to tune that order for 1000s of servers.
OpenStack has grown amazingly and picked up serious corporate support in its first year. Understanding how helps to explain why the initiative has legs and where adopters should invest. While OpenStack has picked up a lot of industry partners, early participation by Dell and Citrix have been important to a meteoric trajectory.
So why are we (Dell) working so hard to light a fire under OpenStack?
To explain OpenStack support and momentum, we have to start with a self-reflective fact: Dell (my employer), Citrix and Rackspace are seeking to gain the dominate positions in their respective areas of “cloud.” Individually, we have competitors in more entrenched positions for cloud silos in solutions, ecosystem, and hypervisor; however, our competitors are not acting in concertto maintain their position.
OpenStack offers an opportunity for companies trying to gain market share to leverage the strengths of partners against their competitors.
The surprising aspect of OpenStack is how much better this collaboration is working in practice than in theory. This, dare I say it, synergistic benefit comes to OpenStack from all the partners (Dell, Citrix, Rackspace and others) working together because:
Today, three major cloud players are standing together to demonstrate commitment to the community. This announcement is a foundation for the OpenStack ecosystem. In the next few months, I expect to see more and more collaborative announcements as the community proves the value of working together.
By aligning around an open platform, we collectively out flank previously dominant players who choose to go it alone. The technology is promising; however, the power of OpenStack flows mainly from the ecosystem that we are building together.
Greg Althaus (@glathaus) and I will be leading a discussion about OpenStack at the May CTLUG on 5/19 at 7pm. The location is Mangia Pizza on Burnet and Duval (In the strip mall where Taco Deli is).
We’ll talk about how OpenStack works, where we see it going, and what Dell is doing to participate in the community.
OpenStack should be very interesting to the CTLUG because of the technologies being used AND way that the community is engaged in helping craft the software.
During the OpenStack Design Conference, Forrester’s James Staten (@Staten7) raved about OpenStack’s transparency compared to AWS. Within the enclave of OpenStack fan boys supports (Dell alone sent >14 people to the summit), his post drew a considerable attention but did little to really further the value proposition.
“Open deployments” are a much more significant value to implementors than transparency from open source code.
For any technology solution, there are significant challenges that will only be understood when the system is under stress. In some cases, these challenges are code defects; however, many will be related to configuration and deployment choices that are site specific. It is correcting these issues that result in design patterns and practices that create a robust infrastructure; consequently, the process of hardening a solution is critical to its ultimate stability and success.
When a solution, like AWS, is deployed and managed by a single entity, it is extremely rare for operational lessons learned and best practices to make it to the larger community. Amazon’s recent post mortem is a welcome exception. This is not a bad thing (Roman Stanek’s contrasting point), it is just the reality of a proprietary cloud. AWS operates as a black box and I don’t believe that Amazon’s operational experience would be relevant to others unless they were also operationally transparent.
While it makes business sense to remain operationally opaque, service providers lose the benefit of external lessons learned when there is no community working in parallel with them.
OpenStack’s community has an opportunity to iterate on CloudOps patterns and practices at a dramatically faster rate than any single provider. This creates distinct value for OpenStack adopters because they can shorten or eliminate their own challenges because other adopters will have the same pains and benefit from the same fixes.
It is critical to understand that the benefit is conferred to both the party sharing the problem (they get advice and support) and the party lending assistance (they avoid the problem). This is distinctly different from proprietary clouds where sharing is likely to cause embarrassment unlikely to create helpful outcomes.
I am not advocating that all OpenStack deployments be the same or follow a prescriptive patterns.
I believe that each installation will be unique in some way; however, there will be enough commonalities and shared code to make sharing worthwhile. This is especially true for adopters who start with tools like Crowbar that leverage community based Chef Recipes and automating scripts. Tools that encourage automation and shared scripts help accelerate the establishment of robust deployment patterns and practices.
Ultimately, the ability to collaborate on cloud operation practice does more to strengthen OpenStack than developers, code reviews or corporate endorsements.
Today, I gave the a presentation that is the sequel to the “bootstrapping the hyperscale clouds.” The topic addresses concerns raised by the BlackOps concept but brings up history and perspectives that drive Dell’s Solution focus for OpenStack.
You can view in slide share or here’s a PDF of the preso: Operating the Hyperscale Cloud 4-27
I tend to favor pictures over text, so you may get more when OpenStack posts the video [link pending].
The talk is about our journey to build an OpenStack solution starting from our drive to build a cloud installer (Crowbar). The critical learning is that success for clouds is the adoption of an “operational model” (aka Cloud Ops) that creates cloud agility.
Ultimately, the presentation is the basis for another white paper.
Appending:
Rick Clark’s post “Why I Left Rackspace and What About Openstack” (+ his softer post script) is part of a longer conversation that started when Rackspace acquired Anso Labs and was expanded with the resignation of Chris Kemp (NASA CTO & OpenStack #1 fanboy).
Building a community is a delicate balance: you need show leadership while you cultivate leadership.
Putting aside the context (resigning from Rackspace to join Cisco) of his post, I think that Rick’s comments do resonate with parts of the community. OpenStack goverance became unbalanced when Anso became Rackspace. The governance board formed at the Austin conference was dominated by a small number (2: NASA/Anso & Rackspace) of highly committed voices but there was no single master.
Considering OpenStack’s momentum, we are in a very good position to fix the single master problem. However, it takes time. While companies like Dell (my employer), NTT, Citrix, Cisco (Rick’s employer), and Microsoft are clearly investing in OpenStack, none have yet achieved NASA or Rackspace’s level of technical committment.
The challenge for Rackspace is to expand the OpenStack market and ecosystem so that partners are motivated to jump in more and more quickly. If my experiences inside Dell are indicative of the broader community, Rackspace’s leadership makes it much easier for partners to increase their own commitment. Like teaching my daughter to ride her bike, she needed to know that I was running next to her before she would pedal hard enough to balance by herself.
Like teaching bike riding – you can’t lead communities too hard or too lightly.
To build a community around OpenStack, we (the partners) need to stand up our own capability. Until we have demonstrated more leadership, Rackspace must cultivate both a community and a market. This is a challenging role to balance. While the community wants distributed ownership, the market wants leadership. Rick’s governance comments are evidence of this struggle and Rick’s move to Cisco is an indication of leadership diversification.
I believe that Rackspace is committed to distributed ownership – we, in the community, need to rise to the challenge!
OpenStack still needs strong leadership from Rackspace because the market needs someone to be accountable for releases and features. That allows new partners to depend on someone to run beside them while the wobble their way along to independence. As the community leaders stand up, we’ll see a balanced community emerge. The challenge is on us to make that happen (and happen quickly).
Rob Hirschfeld 