The Crowbar community has a tradition of “day zero ops” community support for the latest OpenStack release at the summit using our pull-from-source capability. This release we’ve really gone the extra mile by doing it one THREE Linux distros (Ubuntu, RHEL & SLES) in parallel with a significant number of projects and new capabilities included.
I’m especially excited about Crowbar implementation of Havana Docker support which required advanced configuration with Nova and Glance. The community also added Heat and Celiometer in the last release cycle plus High Availability (“Titanium”) deployment work is in active development. Did I mention that Crowbar is rocking OpenStack deployments? No, because it’s redundant to mention that. We’ll upload ISOs of this work for easy access later in the week.
While my team at Dell remains a significant contributor to this work, I’m proud to point out to SUSE Cloud leadership and contributions also (including the new Ceph barclamp & integration). Crowbar has become a true multi-party framework!
Want to learn more? If you’re in Hong Kong, we are hosting a Crowbar Developer Community Meetup on Monday, November 4, 2013, 9:00 AM to 12:00 PM (HKT) in the SkyCity Marriott SkyZone Meeting Room. Dell, dotCloud/Docker, SUSE and others will lead a lively technical session to review and discuss the latest updates, advantages and future plans for the Crowbar Operations Platform. You can expect to see some live code demos, and participate in a review of the results of a recent Crowbar 2 hackathon. Confirm your seat here – space is limited! (I expect that we’ll also stream this event using Google Hangout, watch Twitter #Crowbar for the feed)
My team at Dell has a significant presence at the OpenStack Summit in Hong Kong (details about activities including sponsored parties). Be sure to seek out my fellow OpenStack Board Member Joseph George, Dell OpenStack Product Manager Kamesh Pemmaraju and Enstratius/Dell Multi-Cloud Manager Founder George Reese.
Note: The work referenced in this post is about Crowbar v1. We’ve also reached critical milestones with Crowbar v2 and will begin implementing Havana on that platform shortly.
I believe that we should see many contribution “footprints” for companies in Foundation leadership positions. These footprints do not have to be code in github: there are many visible ways to contribute to OpenStack including internal installs, delivered product, community meetups, open source support around code, service to the community through speaking and sponsoring and, of course, code too.
At this point in the OpenStack evolution, there is so much going on that it is easy to leave footprints because there are so many ways to engage. Footprints are tangible evidence of community leadership and the currency of collaboration. OpenStack thrives because we are committed to working together, being transparent in our actions and providing service to the project beyond our own needs.
I believe OpenStack Foundation’s new gold members will are great additions to our growing community; however, we need to be increasingly deliberate in accepting new Gold members to make sure that they have a history of demonstrating a culture of open source leadership and contribution.
These applications deserve careful consideration for several reasons:
there are a limited number of gold level positions (16 of the 24 are now occupied)
there is no practical way to remove a gold member (but only 8 are elected to the board)
there is a perception (by the applicants) that they gain additional credibility through gold membership
gold and platinum members are the leaders of our community so everyone will models their behavior
It is important to remember that there is no limit or barrier (beyond $) to joining at the corporate sponsors level. So, being a gold member means that companies are seeking a broader leadership role in the project.
Over the next months, Simon Anderson (committee chair, Dreamhost) will be leading me and several other board members in an effort to refine of our Gold member review criteria. I’ll post own list shortly and I’m interested in hearing from you about what type of “footprints” we should be considered in this process.
If registered, you have 8 votes to allocate as you wish. You will get a link via email – you must use that link.
Joseph B George and I are cross-blogging this post because we are jointly seeking your vote(s) for individual member seats on the OpenStack Foundation board. This is key point in the OpenStack journey and we strongly encourage eligible voters to participate no matter who you vote for! As we have said before, success of the Foundation governance process matters just as much as the code because it ensures equal access and limits forking.
We think that OpenStack succeeds because it is collaboratively developed. It is essential that we select board members who have a proven record of community development, a willingness to partner and have demonstrated investment in the project.
Our OpenStack vision favors production operations by being operator, user and ecosystem focused. If elected, we will represent these interests by helping advance deployability, API specifications, open operations and both large and small scale cloud deployments.
Of course, we’re asking for you to consider for both of us; however, if you want to focus on just one then here’s the balance between us. Rob (bio) is a technologist with deep roots in cloud technology, data center operations and open source. Joseph is a business professional with experience new product introduction and enterprise delivery.
Not sure if you can vote? If you registered as an individual member then your name should be on the voting list. In that case, you can vote between 8/20 and 8/24.
I could not be happier with the results Crowbar collaborators and my team at Dell achieved around the 1st Crowbar design summit. We had great discussions and even better participation.
The attendees represented major operating system vendors, configuration management companies, OpenStack hosting companies, OpenStack cloud software providers, OpenStack consultants, OpenStack private cloud users, and (of course) a major infrastructure provider. That’s a very complete cross-section of the cloud community.
I knew from the start that we had too little time and, thankfully, people were tolerant of my need to stop the discussions. In the end, we were able to cover all the planned topics. This was important because all these features are interlocked so discussions were iterative. I was impressed with the level of knowledge at the table and it drove deep discussion. Even so, there are still parts of Crowbar that are confusing (networking, late binding, orchestration, chef coupling) even to collaborators.
In typing up these notes, it becomes even more blindingly obvious that the core features for Crowbar 2 are highly interconnected. That’s no surprise technically; however, it will make the notes harder to follow because of knowledge bootstrapping. You need take time and grok the gestalt and surf the zeitgeist.
Collaboration Invitation: I wanted to remind readers that this summit was just the kick-off for a series of open weekly design (Tuesdays 10am CDT) and coordination (Thursdays 8am CDT) meetings. Everyone is welcome to join in those meetings – information is posted, recorded, folded, spindled and mutilated on the Crowbar 2 wiki page.
These notes are my reflection of the online etherpad notes that were made live during the meeting. I’ve grouped them by design topic.
We are refactoring Crowbar at this time because we have a collection of interconnected features that could not be decoupled
Some items (Database use, Rails3, documentation, process) are not for debate. They are core needs but require little design.
There are 5 key topics for the refactor: online mode, networking flexibility, OpenStack pull from source, heterogeneous/multi operating systems, being CDMB agnostic
Due to time limits, we have to stop discussions and continue them online.
We are hoping to align Crowbar 2 beta and OpenStack Folsom release.
Online / Connected Mode
Online mode is more than simply internet connectivity. It is the foundation of how Crowbar stages dependencies and components for deploy. It’s required for heterogeneous O/S, pull from source and it has dependencies on how we model networking so nodes can access resources.
We are thinking to use caching proxies to stage resources. This would allow isolated production environments and preserves the run everything from ISO without a connection (that is still a key requirement to us).
Suse’s Crowbar fork does not build an ISO, instead it relies on RPM packages for barclamps and their dependencies.
Pulling packages directly from the Internet has proven to be unreliable, this method cannot rely on that alone.
Install From Source
This feature is mainly focused on OpenStack, it could be applied more generally. The principals that we are looking at could be applied to any application were the source code is changing quickly (all of them?!). Hadoop is an obvious second candidate.
We spent some time reviewing the use-cases for this feature. While this appears to be very dev and pre-release focused, there are important applications for production. Specifically, we expect that scale customers will need to run ahead of or slightly adjacent to trunk due to patches or proprietary code. In both cases, it is important that users can deploy from their repository.
We discussed briefly our objective to pull configuration from upstream (not just OpenStack, but potentially any common cookbooks/modules). This topic is central to the CMDB agnostic discussion below.
The overall sentiment is that this could be a very powerful capability if we can manage to make it work. There is a substantial challenge in tracking dependencies – current RPMs and Debs do a good job of this and other configuration steps beyond just the bits. Replicating that functionality is the real obstacle.
CMDB agnostic (decoupling Chef)
This feature is confusing because we are not eliminating the need for a configuration management database (CMDB) tool like Chef, instead we are decoupling Crowbar from the a single CMDB to a pluggable model using an abstraction layer.
It was stressed that Crowbar does orchestration – we do not rely on convergence over multiple passes to get the configuration correct.
We had strong agreement that the modules should not be tightly coupled but did need a consistent way (API? Consistent namespace? Pixie dust?) to share data between each other. Our priority is to maintain loose coupling and follow integration by convention and best practices rather than rigid structures.
The abstraction layer needs to have both import and export functions
Crowbar will use attribute injection so that Cookbooks can leverage Crowbar but will not require Crowbar to operate. Crowbar’s database will provide the links between the nodes instead of having to wedge it into the CMDB.
In 1.x, the networking was the most coupled into Chef. This is a major part of the refactor and modeling for Crowbar’s database.
There are a lot of notes captured about this on the etherpad – I recommend reviewing them
Heterogeneous OS (bare metal provisioning and beyond)
This topic was the most divergent of all our topics because most of the participants were using some variant of their own bare metal provisioning project (check the etherpad for the list).
Since we can’t pack an unlimited set of stuff on the ISO, this feature requires online mode.
Most of these projects do nothing beyond OS provisioning; however, their simplicity is beneficial. Crowbar needs to consider users who just want a stream-lined OS provisioning experience.
It’s OpenStack Summit time again for my team at Dell and there’s deployment in the air. It’s been an amazing journey from the first Austin summit to Folsom today. Since those first heady days, the party has gotten a lot more crowded, founding members have faded away, recruiters became enriched as employees changed email TLDs and buckets of code was delivered.
Throughout, Dell has stayed the course: our focus from day-one has been ensuring OpenStack can be deployed into production in a way that was true to the OpenStack mission of community collaboration and Apache-2-licensed open source.
We’ve delivered on the making OpenStack deployable vision by collaborating broadly on the OpenStack components of the open source Crowbar project. I believe that our vision for sustainable open operations based on DevOps principles is the most complete strategy for production cloud deployments.
We are at the Folsom Summit in force and we’re looking forward to discussions with the OpenStack community. Here are some of the ways to engage with us:
During the summit (M-W), we’ll have our Crowbar OpenStack Essex deployments running. We kicked off Essex development with a world-wide event in early March and we want more people to come and join in.
During the conference (W-F), we’ll be showing off application deployments using enStratus and Chef against our field proven Diablo release.
Thursday 1:00pm, OpenStack Gains Momentum: Customers are Speaking Up by Kamesh Pemmaraju (Dell)
Friday 9:50am, Deploy Apps on OpenStack using Dashboard, Chef and enStratus by Rob Hirschfeld (Dell), Matt Ray (Opscode) and Keith Hudgins (enStratus).
Friday 11:30am, Expanding the Community Panel including Joseph George (Dell)
This fun round trip road trip from Rackspace & Dell HQs in Austin to the summit and home again promises to be an odyssey of inclusion. Dell OpenStack/Crowbar engineer Andi Abes (@a_abes). Follow @RoadstackRV to follow along as they return home and share their thoughts about the summit!
Monday 6pm Mirantis Welcome Party, co-sponsored with Dell, at Sens Restaurant (RSVP)
Tuesday 5pm “Demos & Drinks” Happy Hour, co-hosted by Dell, Mirantis, Morphlabs, Canonical at the Hyatt Regency Hospitality Room off the Atrium
My team has been in the field talking to customers and doing OpenStack deployments. We are proud to talk about it and our approach.
Mostly importantly, we want to collaborate with you on our Essex deployments using Crowbar. Get on our list, download/build crowbar, run the “essex-hack” branch and start banging on the deploy. Let’s work together to make this one rock solid Essex deploy.
I was very impressed by the quality of discussion at the Deployment topic meeting for Austin OpenStack Meetup (#OSATX). Of the 45ish people attending, we had representations for at least 6 different OpenStack deployments (Dell, HP, ATT, Rackspace Internal, Rackspace Cloud Builders, Opscode Chef)! Considering the scope of those deployments (several are aiming at 1000+ nodes), that’s a truly impressive accomplishment for such a young project.
Even with the depth of the discussion (notes below), we did not go into details on how individual OpenStack components are connected together. The image my team at Dell uses is included below. I also recommend reviewing Rackspace’s published reference architecture.
Our deployment discussion was a round table so it is difficult to link statements back to individuals, but I was able to track companies (mostly).
picked Ubuntu & KVM because they were the most vetted. They are also using Chef for deployment.
running Diablo 2, moving to Diablo Final & a flat network model. The network controller is a bottleneck. Their biggest scale issue is RabbitMQ.
is creating their own Nova Volume plugin for their block storage.
At this point, scale limits are due to simultaneous loading rather than total number of nodes.
The Nova node image cache can get corrupted without any notification or way to force a refresh – this defect is being addressed in Essex.
has setup availability zones are completely independent (500 node) systems. Expecting to converge them in the future.
is using the latest Ubuntu. Always stays current.
using Puppet to setup their cloud.
They are expecting to go live on Essex and are keeping their deployment on the Essex trunk. This is causing some extra work but they expect it to pay back by allowing them to get to production on Essex faster.
Deploying on XenServer
“Devs move fast, Ops not so much.” Trying to not get behind.
Rackspace Cloud Builders (RCB) is running major releases being run through an automated test suite. The verified releases are being published to https://github.com/cloudbuilders (note: Crowbar is pulling our OpenStack bits from this repo).
Dell commented that our customers are using Crowbar primarily pilots – they are learning how to use OpenStack
Said they have >10 customer deployments pending
ATT is using OpenSource version of Crowbar
Need for Keystone and Dashboard were considered essential additions to Diablo
KVM is considered the top one for now
Libvirt (which uses KVM) also supports LXE which people found to be interesting
XenServer via XAPI are also popular
No so much activity on ESX & HyperV
We talked about why some hypervisors are more popular – it’s about the node agent architecture of OpenStack.
NetApp via Nova Volume appears to be a popular block storage
Keystone / Dashboard
Customers want both together
Including keystone/dashboard was considered essential in Diablo. It was part of the reason why Diablo Final was delayed.
HP is not using dashboard
Members of the Audience made comments that we need to deprecate the EC2 APIs (because it does not help OpenStack long term to maintain EC2 APIs over its own). [1/5 Note: THIS IS NOT OFFICIAL POLICY, it is a reflection of what was discussed]
HP started on EC2 API but is moving to the OpenStack API
Next meeting is Tuesday 1/10 and sponsored by SUSE (note: Tuesday is just for this January). Topic TBD.
We’ve got sponsors for the next SIX meetups! Thanks for Dell (my employeer), Rackspace, HP, SUSE, Canonical and PuppetLabs for sponsoring.
We discussed topics for the next meetings (see the post image). We’re going to throw it to a vote for guidance.
The OSATX tag is also being used by Occupy San Antonio. Enjoy the cross chatter!