Greg Althaus at 11/15 Austin Cloud User Group meeting (annotated 90 min audio) November 18, 2011
Posted by Rob H in CloudOps, Crowbar, Crowbar, Greg Althaus, Joseph George, Meetup, Video.Tags: ACUG, cloud, Crowbar, Dell, Greg Althaus, OpenStack
add a comment
Greg Althaus did a 90 minute Crowbar deep dive at this week’s Austin Cloud User Group. Brad Knowles recorded audio and posted it so I thought I’d share the link and my annotations. There are a lot more times to catch up with our team at Dell in Austin, Boston, and Seattle.
Video Annotations - (##:## time stamp)
- 00:00 Intros & Meeting Management
- 12:00 Joseph George Introduction / Sponsorship
- 16:00 Greg Starts – why Crowbar
- 19:00 DevOps slides
- 21:00 What does Crowbar do for DevOps
- make it easier to manage
- make it easier to repeat
- 24:00 What’s included – how we grow / where to start
- 27:20 Starting to show crowbar – what’s included as barclamps
- pluggable / configuration
- Barclamps!
- 28:10 What is a barclamp
- discussion about the barclamps in the base
- 34:30: We <3 Chef. Puppet vs Chef
- 36:00 Why barclamps are more than cookbooks
- 36:30 State machine & transitions
- Q&A Section
- 38:50 Reference Architectures
- 43:00 Barclamps work outside of Crowbar?
- 44:15 Hardware models supported
- 47:30 Storage Queston
- 49:00 HA progress
- 53:00 Ceph as a distributed cloud on all nodes
- 56:20 Deployer has a map of how to give out roles
- 58:00 Demo Fails
- 58:30 Crowbar Architecture
- 62:00 How Crowbar can be extended
- 63:00 Workflow & Proposals
- 65:40 Meta Barclamps
- 71:10 Chef Environments
- 73:40 Taking OpenStack releases and Environments
- 75:00 The case for remove recipes
- 77:33 Git Hub Tour
- 79:00 Network Barclamp deep dive
- 84:00 Adding switch config (roadmap topic)
- 86:30 Conduits
- 87:40 Barclamp Extensions / Services
- 89:00 Questions
- 89:20 Proposal operations
- 93:30 OpenStack Readiness & Crowbar Design Approach
- 93:10 Network Teaming
- 94:30 Which OS & Hypervisors
- 96:30 Continuous Integration & Tools
- 98:40 BDD (“cucumberesque”) & Testing
- 99:40 Build approach & barclamp construction
- 100:00 Wrap up by Joseph
Cote & Rob interview: Crowbar+OpenStack Summit/Conference Reflections (40 mins) October 16, 2011
Posted by Rob H in Amazon, CloudOps, Crowbar, DevOps, Michael Cote, Open source, OpenStack, OpenStack Design Summit, Opscode.2 comments
I’m working on a larger post about the OpenStack Summit around API Implementation vs. Specification. You can have a preview of that AND A LOT OF OTHER STUFF (OpenStack, Crowbar, lunch) in this 40 minute interview w/ Michael Cote.
Setting: Dell World
Interview w/ @Cote at the Hilton Hotel Lobby on 6th street in Austin.
I know that Cote’s post does not have a time marker for easy navigation; however, I added them to help guide your navigation in the interview (link for audio) if you want to jump around.
- 0:00 Introductions
-
1:00 OpenStack
- 1:00 Essex Conference – what is it, naming conventions
- 2:45 Diablo is adding projects from incubation (Keystone, Dashboard,Quantum,
- 5:30 OpenStack vs. Amazon – “OpenStack has ambitions.” We see it as a “platform for innovation.”
- 6:30 OpenStack is a competitor for Amazon. It implements the EC2 APIs.
- 7:30 How are people managing the evolving nature?
- 8:20 We’re going to see OpenStack in production for the next release based on what we see in our deal flow.
- 9:00 Every user that comes on adds momentum
- 9:30 Rackspace setting up the OpenStack foundation is a reflection of the speed of adoption
- 11:15 Our message is “we’re doing it, we’re in the field.” We are very hands on
- 11:15 We chose early on to focus on helping deployment to help drive adoption
- 12:00 “Our first test for partners is: Are you contributing back to the community?”
- 12:44 The community told us “if you are participating then you are going to open source.” Our commits for OpenStack are live and in the open on our github.
- 13:40 Why Github? We’re happy with it.
- 14:20 OpenStack is using Gerrit because they have a gated trunk. They are migrated to Github
- 15:20 APIs have been a big topic for OpenStack
- 16:00 Do you track who is forking and following? Yes. We also have a listserv. We are trying to do a better job managing the Crowbar community. We know we need to do a better job.
- 17:30 OpenStack is defined by its Implementation. That’s “an effective way to move the project forward quickly;” however, we’re getting to a point where people want to use alternate implementations.
- 19:20 Implementation vs Specification is like the SOAP vs REST debate
- 20:05 This is something the community needs to wrestle with
- 21:45 Specification would allow the efforts to scale. The more people consume the API, the more people care about how it operates
- 22:30 “Bugs can become the API”
- 23:10 Asia and Europe are very active. We are seeing a ton of activity overseas.
- 1:00 Essex Conference – what is it, naming conventions
-
23:30 Crowbar
- 24:00 Crowbar arose out of our need to deploy cloud software regardless of customer infrastructure
- 24:45 We would show up and the customer needed all this cloud infrastructure. We created Crowbar because we always needed this
- 26:00 We extended Chef because we had to do the initial bring-up including BIOS and RAID
- 26:45 We added a state machine and an orchestration layer
- 27:45 Updating the system is a huge component. Every month you may be upgrading the infrastructure!
- 28:30 In our lab, we build whole clouds multiple times a day
- 29:45 Crowbar is the “cloud unboxer”
- 30:00 We modularized Crowbar with barclamps. Hadoop and OpenStack are a series of barclamps. Over 5 for each
- 31:00 Barclamps are applied as layers. We are using that as a term to define DevOps
- 31:15 We are using Crowbar to help message that we understand DevOps
- 31:45 Soup vs Sandwich analogy – Images are like soup while DevOps is like a sandwich.
- 32:45 If you don’t want something in a 1000 server deployment, DevOps lets you make a small change. Gives you flexibility.
- 33:45 We added Cloud Foundry
- 34:00 We’ve made it so easy with barclamps that partners are coming to us with ideas for barclamps. It’s like “changing the meat for the sandwich.”
- 34:30 Dreamhost Ceph team created a barclamp and was actually running a majority of the Crowbar demos at the OpenStack conference
- 24:00 Crowbar arose out of our need to deploy cloud software regardless of customer infrastructure
-
35:25 What’s the future for Crowbar?
- 35:30 More aspects of the infrastructure as open source
- 35:45 More Hardware
- 36:00 Multiple operating systems at the same time (XenServer, ESX, etc)
- 36:30 Larger scale
- 36:50 More types of infrastructure: storage & network
- 37:40 Scalr shout out
- 38:00 We know we need to collaborate more with our community
- 38:30 The first step is to download it and try. Read my blog and sign up for the list serve
- 39:00 CROWBAR IS NOT DELL SPECIFIC – we are working with people who want to create support for other vendor’s hardware. This benefits Dell.
- 39:40 We don’t pretend that our customers are single vendor
- 35:30 More aspects of the infrastructure as open source
OpenStack Hardware Sessions Slides & Updates to “Bootstrapping Hyperscale Clouds” October 6, 2011
Posted by Rob H in CloudOps, Greg Althaus, OpenStack Design Summit.1 comment so far
Disclaimer: I work for Dell and we make very fine cloud optimized hardware. This presentation is NOT slanted towards Dell hardware, it simply reflects lessons learned from listening to our customers. Of course, people buying Dell servers, networking and storage products pays my salary and WE LOVE OUR CLOUD CUSTOMERS!
This presentation OpenStack Hardware Oct 2011 was the outline for our revisiting of the hyperscale white paper from last fall. I’m in the process of updating this to reflect lessons learned.
Here are some highlights:
- Fault zones are not as critical as expected – customers are putting in more redundancy instead of striping apps
- 10 GbE is more popular than expected
- Logically segmented redundant networking (teamed) is more popular than physical isolate
- Customers are starting small (AnyScale instead of HyperScale)
Crowbar modularized: latest changes that make clouds even easier to create, update, and maintain September 19, 2011
Posted by Rob H in CloudOps, Crowbar, DevOps, Hadoop, Migration, OpenStack, Opscode, RackSpace.Tags: Barclamp, Cloud Foundry, Crowbar, Github, hadoop, ISO, migration, OpenStack, PowerEdge, rackspace, repo, RHEL, version
8 comments
In the last week, my team at Dell completed a major refactoring of Crowbar that significantly improves our ability to bring in community contributions and field customizations. Today, we merged it into Crowbar’s public repo(s).
From the very first versions, our objective for Crowbar was to create the fastest and most reliable cloud deployments. Along the way, we realized Crowbar’s true potential lay in embracing DevOps as an operational model for maintaining clouds. That meant building up cloud deployments in layers from pieces that we call barclamps (extensions of Chef cookbooks). Our first version, centered on OpenStack Cactus, leveraged barclamps but was still created as a single system. This unified system was a huge step forward in cloud deployments, but did not live up to our CloudOps vision of continuous delivery.
In this version, each Crowbar barclamp is an independent delivery unit that can be integrated before, while or after installing Crowbar.
The core of the change is each barclamp, including the most core ones, are stored in independent code repositories. Putting the code into distinct repos means that each barclamp can have its own life cycle, its own maintainer site and its own dependency tree. This modularization allows customers to manage their Crowbar deployments with a very fine brush: they may choose to customize parts of the system, they could lock components to specific tag and they can bring in barclamps from other vendors.
While the core barclamps are automatically integrated into the Crowbar build using git submodules; other barclamps are installed into the system as needed. This allows you to pull in the suite of OpenStack barclamps at build time or to wait until your Crowbar system is running before installing. Once you install a barclamp, you are able to retrieve an updated barclamp and reapply it to the system.
This feature gives you the ability to 1) choose exactly what you want to include and 2) perform field updates to a live Crowbar system.
Let’s look at some examples:
- The Cloud Foundry barclamp can be sourced Cloud Foundry instead of bundled into the Crowbar repository. This allows the team working on the cloud application to take ownership for their own deployment. As a continuous delivery proponent, I believe strongly that the development team should be responsible for ensuring that their code is deployable (refer to my OpenStack “Deployer API” blue print attempting to codify this).
- DreamHost, maintainers of Ceph Storage, can maintain their own local barclamp repos for OpenStack that are cloned from our community Swift barclamp. This allows them to innovate and customize OpenStack deployments for their business and choose which updates to merge back to the community.
- Rackspace Cloud Builders can work on the most leading edge OpenStack features and maintaining workable deployments on branches. As the code stabilizes, they simply merge in their changes.
- Dell BIOS and RAID barclamps only support the PowerEdge C line today. When we offer PowerEdge R support, you will be able to install or update the barclamps to add that capability. If another hardware vendor creates a barclamp for their hardware then you can install that into your existing system.
I believe that these changes to Crowbar are a huge step forwards on our journey of creating a community supportable Open Operations framework. I hope that you are as excited as I am about these changes.
I encourage you to take the first step by trying out Crowbar and, ultimately, writing your own barclamps.
Post Scripts:
- In addition to the modularization, the updated code includes RHEL as a deployment platform. At present, you must choose to be either RHEL or Ubuntu at build time.
- We have enhanced the network barclamp to describe connections as more abstract connections, called conduits, between nodes. This is a powerful change, but requires some understanding before you start making changes.
- We have only begun testing the change as of 9/12, we expect the system to be fully stabilized by 10/3. If you are not willing to deal with bugs then I recommend building the Crowbar “v1.0″ tag (or using the ISOs from our July launch).
Crowbar near-term features: increasing DevOps mojo and brewing Diablo August 24, 2011
Posted by Rob H in CloudOps, Crowbar, OpenStack, OpenStack Design Summit.Tags: Barclamp, CloudOps, Crowbar, DevOps, Diablo, Feature, OpenStack, RHEL, UI, version
3 comments
We’ve been so busy working on getting RHEL support ready to drop into the Crowbar repos that I have not had time to post about what’s coming next for Crowbar. The RHEL addition has required a substantial amount of work to accommodate different packaging models and capabilities. This change moves Crowbar closer to being able allow nodes’ operating systems (the allocated TFTP Boot Image) to be unique per node.
I will post more forward looking details soon but wanted to prime the pump and invite suggestions from our community.
We are tracking two major features for delivery by the OpenStack October Design Conference
- OpenStack Diablo Barclamps. Expect to see individual barclamps for various components like Keystone, Dashboard, Glace, Nova, Swift, etc)
- Barclamp versioning / connected imports. This feature will enable Crowbar to pull in the latest components for barclamps from remote repositories. I consider this a critical feature for Crowbar’s core DevOps/CloudOps capabilities and to support more community development for barclamps.
We are also working on some UI enhancements
- Merging together the barclamps/proposals/active views into a single view
- Enabling bulk actions for nodes (description, BIOS types, and allocate)
- Allowing users to set node names and showing the names throughout the UI
- More clarity on state of proposal application process (stretch goal)
I am planning to post more about our design ideas as work begins.
If you want to help with Diablo barclamps, these will be worked in the open and we’d be happy to collaborate. We’re also open to suggestion for what’s next.
Crowbar source released, includes OpenStack Cloud install July 26, 2011
Posted by Rob H in Barton George, CloudOps, Joseph George, Open source, OpenStack, Opscode, OSCON.Tags: Apache 2, Barclamp, Chef, Crowbar, Github, hadoop, OpenStack, OpsCode, People/Dell, Wiki
32 comments
I’m delighted to announce (official version) that my team at Dell has opened the Crowbar source under the Apache 2 license. This action is part of the broader Dell OpenStack Cloud Solution which includes OpenStack install packages, Crowbar, reference hardware architectures, and services/consulting to support deployments.
There are two important components to this news:
- Dell is officially offering our OpenStack Solution and helping advance the community’s ability to implement OpenStack quickly and consistently.
- Dell is releasing the Crowbar code (which is included in the solution) as open source.
Both are significant items; however, my focus here is on the Crowbar release.
Crowbar started as a Dell OpenStack installer project and then grew beyond that in scope. Now it can be extended to work with other vendors’ kits and other solutions bits.
We are contributing Crowbar to the community because we believe that everyone benefits by sharing in the operational practices that Crowbar embodies. These are rooted in Opscsode Chef (which Crowbar tightly integrates with) and the cloud & hyper-scale proven DevOps practices that are reflected in our deployment model.
Where to get it?
- Code: Github (Apache 2 license)
- Community: Dell Crowbar listserv,
- Information: wiki on Crowbar Github, Crowbar User’s Guide and RobHirschfeld.com web site.
What’s included?
- A comprehensive set of barclamps to set up an OpenStack cloud.
- Crowbar UI and Remote APIs to make it easy to set up your cloud
- Automated testing scripts for community members doing continuous integration with OpenStack.
- Build scripts so you can create your own Crowbar install ISO
- Switch discovery so you can create Chef Cookbooks that are network aware.
- Open source Chef server that powers much of Crowar’s functionality
What’s not included?
- Non-open source license components (BIOS+RAID config) that we could not distribute under the Apache 2 license. We are working to address this and include them in our release. They are available in the Dell Licensed version of Crowbar.
- Dell Branded Components (skin + overview page). Crowbar has an OpenSource skin with identical functionality.
- Pre-built ISOs with install images (you must download the open source components yourself, we cannot redistribute them to you as a package)
Important notes:
- Crowbar uses Chef Server as its database and relies on cookbooks for node deployments. It is installed (using Chef Solo) automatically as part of the Crowbar install.
- Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as barclamps.
- Each barclamp has its own Chef configuration, UI sub-component, deployment configuration, and documentation.
On the project roadmap:
- Hadoop support
- Additional operating system support (specifically RHEL)
- Barclamp version repository
- Network configuration
- We’d like suggestions! Please comment!
Sites for more information: Joseph George, Barton George (launch day), Dell
Austin CloudCamp 7/20 – Lightening Talk July 19, 2011
Posted by Rob H in Cloud Camp, CloudOps, OpenStack.Tags: ATX, CloudCamp, CloudOps, OpenStack
add a comment
If you’re in Austin on 7/20 then come to the 2011 ATX CloudCamp @ 6pm (Downtown)
In addition to the normal great unconference format, I’ll be giving one of the lightning talks. My topic will be about Cloud Operations for OpenStack.
Here’s a copy of my CloudCamp 07 2011 preso. Unfortunately, the video was not complete so I can’t include it.
The unexpected openness of OpenStack: why it’s important to learn from others’ operations experience. May 2, 2011
Posted by Rob H in Amazon, CloudOps, OpenStack, OpenStack Design Summit.Tags: amazon, Blackbox, CloudOps, community, Crowbar, James Staten, OpenOps, OpenStack, People/Dell, Roman Stanek, transparency
1 comment so far
During the OpenStack Design Conference, Forrester’s James Staten (@Staten7) raved about OpenStack’s transparency compared to AWS. Within the enclave of OpenStack fan boys supports (Dell alone sent >14 people to the summit), his post drew a considerable attention but did little to really further the value proposition.
“Open deployments” are a much more significant value to implementors than transparency from open source code.
For any technology solution, there are significant challenges that will only be understood when the system is under stress. In some cases, these challenges are code defects; however, many will be related to configuration and deployment choices that are site specific. It is correcting these issues that result in design patterns and practices that create a robust infrastructure; consequently, the process of hardening a solution is critical to its ultimate stability and success.
When a solution, like AWS, is deployed and managed by a single entity, it is extremely rare for operational lessons learned and best practices to make it to the larger community. Amazon’s recent post mortem is a welcome exception. This is not a bad thing (Roman Stanek’s contrasting point), it is just the reality of a proprietary cloud. AWS operates as a black box and I don’t believe that Amazon’s operational experience would be relevant to others unless they were also operationally transparent.
While it makes business sense to remain operationally opaque, service providers lose the benefit of external lessons learned when there is no community working in parallel with them.
OpenStack’s community has an opportunity to iterate on CloudOps patterns and practices at a dramatically faster rate than any single provider. This creates distinct value for OpenStack adopters because they can shorten or eliminate their own challenges because other adopters will have the same pains and benefit from the same fixes.
It is critical to understand that the benefit is conferred to both the party sharing the problem (they get advice and support) and the party lending assistance (they avoid the problem). This is distinctly different from proprietary clouds where sharing is likely to cause embarrassment unlikely to create helpful outcomes.
I am not advocating that all OpenStack deployments be the same or follow a prescriptive patterns.
I believe that each installation will be unique in some way; however, there will be enough commonalities and shared code to make sharing worthwhile. This is especially true for adopters who start with tools like Crowbar that leverage community based Chef Recipes and automating scripts. Tools that encourage automation and shared scripts help accelerate the establishment of robust deployment patterns and practices.
Ultimately, the ability to collaborate on cloud operation practice does more to strengthen OpenStack than developers, code reviews or corporate endorsements.
