Tomorrow (3/1), numerous sites are gathering around a World Wide Essex Hack Day on 3/1. If you want to participate or even host a hack venue, get on the list and IRC channel (details).
My team at Dell is organizing a community a follow-upOpenStack Essex Install Day next week (3/8) in both Austin and Boston. Just like the Hack Day, the install fest will focus on Essex release code with both online and local presence. Unlike the Hack Day, our focus will be on deployments. For the Dell team, that means working on the Essex deployment for Crowbar. We’re still working on a schedule and partner list so stay tuned. I’m trying to webcast Crowbar & OpenStack training sessions during the install day.
Matt Ray from Opscode presented some of the work with Chef and OpenStack. He talked about the three main chef repos floating around. He called out Anso’s original cookbook set that is the basis for the Crowbar cookbooks (his second set), and his final set is the emerging set of cookbooks in OpenStack proper. The third one is interesting and what he plans to continue working on to make into his public openstack cookbooks. These are an amalgamation of smokestack, RCB, Anso improvements, and his (Crowbar’s).
He then demoed his knife plugin (slideshare) to build openstack virtual servers using the Openstack API. This is nice and works against TryStack.org (previously “Free Cloud”) and RCB’s demo cloud. All of that is on his github repo with instructions how to build and use. Matt and I talked about trying to get that into our Crowbar distro.
There were some questions about flow and choice of OpenStack API versus Amazon EC2 API because there was already an EC2 knife set of plugins.
Ziad Sawalha talks about Keystone
Ziad Sawalha is the PLT (Project Technical Lead) for Keystone. He works for Rackspace out of San Antonio. He drove up for the meeting.
He split his talk into two pieces, Incubation Process and Keystone Overview. He asked who was interested in what and focused his talk more towards overview than incubation.
Some key take-aways:
Keystone comes from Rackspace’s strong, flexible, and scalable API. It started as a known quantity from his perspective.
Community trusted nothing his team produced from an API perspective
Community is python or nothing
His team was ignored until they had a python prototype implementing the API
At this point, comments on API came in.
Churn in API caused problems with implementation and expectations around the close of Diablo.
Because comments were late, changes occurred.
Official implementation lagged and stalled into arriving.
API has been stable since Diablo final, but code is changing. that is good and shows strength of API.
Side note from Greg, Keystone represents to me the power of API over Code. You can have innovation around the implementation as long all the implementations have a fair ground work to plan under which is an API specification. The replacement of Keystone with the Keystone Light code base is an example of this. The only reason this is possible is that the API was sound and documented. (Rob’s post on this)
Ziad spent the rest of his time talking about the work flow of Keystone and the API points. He covered the API points.
Client to Keystone, Keystone to Client for initial auth token
Client to Middleware API for the services to have a front.
Middleware to Keystone to verify and establish identity.
Middleware to Service to pass identity
Not many details other then flow and flexibility. He stressed the API design separated protocol from actions and data at all the layers. This allows for future variations and innovations while maintaining the APIs.
Ziad talked about the state of Essex.
RBAC (aka Role Based Access Control)
Code replacement Keystone Light
Federation was planned but will most likely be pushed to G
Federation is the ability for multiple independent Keystones to operate (bursting use case)
Dependent upon two other federation components (networking and billing/metering)
Just after the OpenStack Essex 3 milestone, Ziad Sawalha of Rackspace announced a major shift in the Keystone code base. I applaud the clarity of Ziad’s email but want to restate my understanding of the changes here rather than simply parrot him.
These changes improve Keystone and OpenStack in several ways.
The Keystone team is keeping the current APIs while swapping their implementation. They recommend switching back to an implementation based on the Rackspace Cloud Builder’s Keystone Light code base. I say switching back because my team at Dell has some experience with the Keystone Light (KSL) code. KSL was used with our first Diablo release work while legacy Keystone (Diablo Keystone?) was being readied for release. Upon reflection, the confusion around Keystone readiness for Diablo may have been an indicator to some disconnects that ultimately contributed to last week’s decision.
This is not an 11th hour rewrite. Keystone Light (now Essex Keystone?) offers
An existing code base that has been proven in real deployments
Stronger identity pluggability, better EC2 compatibility and higher production readiness
An existing testing framework and proven extensibility and flexibility
Plus, the team has committed to ensure a simple migration path
Beyond the code and Keystone, making a change like this takes confidence and guts.
This change is not all sunshine and rainbows. Making a major change midway through the release cycle introduces schedule and delivery risk. Even though not fully graduated to core project status, Keystone is already an essential component in OpenStack. People will certainly raise valid questions about production readiness and code churn within the project. Changes like these are the reality for any major project and doubly so for platforms.
The very fact that this change is visible and discussed by the OpenStack community shows our strength.
Acknowledging and quickly fixing a weakness in the OpenStack code base is exactly the type of behavior that the community needs to be successful and converge towards a great platform. The fact that maintaining the API is a priority shows that OpenStack is moving in the direction of more API based standards. While the Keystone change is not a recommendation for dual implementations (the Diablo Keystone fork will likely die out), it should help set the stage for how the community will handle competing implementations. If nothing else, it is a strong argument for maintaining API tests and compliance.
The Keystone change is a forward looking one. Our Crowbar team will investigate how we will incorporate it. As part of OpenStack, the new Keystone code will (re)surface for the Essex deployment and that code will be part of the Dell OpenStack-Powered Cloud. This work, like the previous, will be done in the open as part of the OpenStack barclamps that we maintain on the Crowbar github.
I was very impressed by the quality of discussion at the Deployment topic meeting for Austin OpenStack Meetup (#OSATX). Of the 45ish people attending, we had representations for at least 6 different OpenStack deployments (Dell, HP, ATT, Rackspace Internal, Rackspace Cloud Builders, Opscode Chef)! Considering the scope of those deployments (several are aiming at 1000+ nodes), that’s a truly impressive accomplishment for such a young project.
Even with the depth of the discussion (notes below), we did not go into details on how individual OpenStack components are connected together. The image my team at Dell uses is included below. I also recommend reviewing Rackspace’s published reference architecture.
Our deployment discussion was a round table so it is difficult to link statements back to individuals, but I was able to track companies (mostly).
picked Ubuntu & KVM because they were the most vetted. They are also using Chef for deployment.
running Diablo 2, moving to Diablo Final & a flat network model. The network controller is a bottleneck. Their biggest scale issue is RabbitMQ.
is creating their own Nova Volume plugin for their block storage.
At this point, scale limits are due to simultaneous loading rather than total number of nodes.
The Nova node image cache can get corrupted without any notification or way to force a refresh – this defect is being addressed in Essex.
has setup availability zones are completely independent (500 node) systems. Expecting to converge them in the future.
is using the latest Ubuntu. Always stays current.
using Puppet to setup their cloud.
They are expecting to go live on Essex and are keeping their deployment on the Essex trunk. This is causing some extra work but they expect it to pay back by allowing them to get to production on Essex faster.
Deploying on XenServer
“Devs move fast, Ops not so much.” Trying to not get behind.
Rackspace Cloud Builders (RCB) is running major releases being run through an automated test suite. The verified releases are being published to https://github.com/cloudbuilders (note: Crowbar is pulling our OpenStack bits from this repo).
Dell commented that our customers are using Crowbar primarily pilots – they are learning how to use OpenStack
Said they have >10 customer deployments pending
ATT is using OpenSource version of Crowbar
Need for Keystone and Dashboard were considered essential additions to Diablo
KVM is considered the top one for now
Libvirt (which uses KVM) also supports LXE which people found to be interesting
XenServer via XAPI are also popular
No so much activity on ESX & HyperV
We talked about why some hypervisors are more popular – it’s about the node agent architecture of OpenStack.
NetApp via Nova Volume appears to be a popular block storage
Keystone / Dashboard
Customers want both together
Including keystone/dashboard was considered essential in Diablo. It was part of the reason why Diablo Final was delayed.
HP is not using dashboard
Members of the Audience made comments that we need to deprecate the EC2 APIs (because it does not help OpenStack long term to maintain EC2 APIs over its own). [1/5 Note: THIS IS NOT OFFICIAL POLICY, it is a reflection of what was discussed]
HP started on EC2 API but is moving to the OpenStack API
Next meeting is Tuesday 1/10 and sponsored by SUSE (note: Tuesday is just for this January). Topic TBD.
We’ve got sponsors for the next SIX meetups! Thanks for Dell (my employeer), Rackspace, HP, SUSE, Canonical and PuppetLabs for sponsoring.
We discussed topics for the next meetings (see the post image). We’re going to throw it to a vote for guidance.
The OSATX tag is also being used by Occupy San Antonio. Enjoy the cross chatter!
I’m glad to acknowledge that I incorrectly reported the OpenStack Quantum project would require licensed components for implementation! I fully stand behind Quantum as being OpenStack’s “killer app” and am happy to post more information about it here.
Side note: My team at Dell is starting to get Crowbar community pings about collaboration on a Quantum barclamp. Yes, we are interested!
This updates comes via Dan Wendlandt from Nicira who pointed out my error in the Seattle Meetup notes (you can read his comment on that post). Rather than summarize his information, I’ll let Dan talk for himself…
Dan’s comments about open source Quantum implementation:
Dan’s comments about Quantum OpenStack project status in the D-E-F release train:
At the end of the Diablo cycle, Quantum applied to become an incubated project, which means it will be incubated for Essex. At the end of the Essex cycle, we plan to apply to be a core project, meaning that if we are accepted, we would be a core project for the F-series release.
Its worth noting, however, that [Dan] knows of many people planning on putting Quantum in production before then, which is the real indicator of a project’s maturity (regardless of whether it is technically “core” or not).
This turned out to be a major open cloud gab fest! In addition to Dell OpenStack leads (Greg and I), we had the Nova Project Technical Lead (PTL, Vish Ishaya, @vish), HP’s Cloud Architect (Alex Howells, @nixgeek), Opscode OpenStack cookbook master (Matt Ray, @mattray). We were joined by several other Chef Summit attendees with OpenStack interest including a pair of engineers from Spain.
We’d planned to demo using Knife-OpenStack against the Crowbar Diablo build. Unfortunately, the knife-openstack is out of date (August 15th?!). We need Keystone support. Anyone up for that?
There’s no way I can recapture everything that was said, but here are some highlights I jotted down the on the way home.
After the miss with Keystone and the Diablo release, solving the project dependency problem is an important problem. Vish talked at length about the ambiguity challenge of Keystone being required and also incubated. He said we were not formal enough around new projects even though we had dependencies on them. Future releases, new projects (specifically, Quantum) will not be allowed to be dependencies.
The focus for Essex is on quality and stability. The plan is for Essex to be a long-term supported (LTS) release tied to the Ubuntu LTS. That’s putting pressure on all the projects to ensure quality, lock features early, and avoid unproven dependencies.
There is a lot of activity around storage and companies are creating volume plug-ins for Nova. Vish said he knew of at least four.
Networking has a lot of activity. Quantum has a lot of activity, but may not emerge as a core project in time for Essex. There was general agreement that Quantum is “the killer app” for OpenStack and will take cloud to the next level. The Quantum Open vSwitch implementaiton is completely open source and free. Some other plugins may require proprietary hardware and/or software, but there is definitely a (very) viable and completely open source option for Quantum networking.
HP has some serious cloud mojo going on. Alex talked about defects they have found and submitted fixes back to core. He also hinted about some interesting storage and networking IP that’s going into their OpenStack deployment. Based on his comments, I don’t expect those to become public so I’m going to limit my observations about them here.
We talked about hypervisors for a while. KVM and XenServer (via XAPI) were the primary topics. We did talk about LXE & OpenVZ as popular approaches too. Vish said that some of the XenServer work is using Xen Storage Manager to manage SAN images.
Vish is seeing a constant rise in committers. It’s hard to judge because some committers appear to be individuals acting on behalf of teams (10 to 20 people).
Based on our last meetup, it appears deployment is a hot topic, so we’ll kick off with that – bring your experiences, opinions, and thoughts! We’ll also open the floor to other OpenStack topics that would be discussed – open technical and business discussions – no commercials please!
We’ll also talk about organizing future OpenStack meet ups! If your company is interested in sponsoring a future meetup, find Joseph George at the meetup and he can work with you on details.
Registration for the OpenStack Conference (which my team at Dell is co-sponsoring), which is now a separate event (from the User Conference the same week) with a separate registration mechanism.
From the OpenStack List:
The Essex Design Summit is an event for the OpenStack developers community (existing and prospective coders) to gather and discuss the upcoming development cycle. It is highly technical in nature and is not targeted to the general OpenStack public, which will find the OpenStack Conference much more enjoyable.
If you think you belong in the former category, please join us. The Summit is free to attend, but with limited attendance, so we require registration.