We had a great discussion about OpenStack, Ops and Crowbar. I appreciate Niki’s insightful questions and an opportunity to share my opinions. I feel that we covered years of material in just 1 hour and I appreciate the opportunity to appear on the podcast.
51:00 We should be encouraging people to use OpenStack for its use cases
51:30 Existential question for OpenStack: are we a suite or product. The community is split here
51:30 In comparing with Amazon, does OpenStack have to implement it or build an ecosystem to compete
53:00 As soon as you make something THE OpenStack project (like Heat) you are sending a message that the alternates are not welcome
54:30 OpenStack ends up in a trap if we pick a single project and make it the way that we are going do something. New implementations are going to surface from WITHIN the projects and we need to ready for that.
55:15 new implementations are coming, we have to be ready for that. We can make ourselves vulnerable to splitting if we do not prepare.
56:00 API vs Implementation? This is something that splits the community. Ultimately we to be an API spec but we are not ready for that. We have a lot of work to do first using the same code base.
56:50 DefCore has taken a balanced approach using our diversity as a strength
57:20 Bylaws did not allow for enough flexibility for what is core
If you are coming to the OpenStack summit in San Diego next week then please find me at the show! I want to hear from you about the Foundation, community, OpenStack deployments, Crowbar and anything else. Oh, and I just ordered a handful of Crowbar stickers if you wanted some CB bling.
Matt Ray from Opscode presented some of the work with Chef and OpenStack. He talked about the three main chef repos floating around. He called out Anso’s original cookbook set that is the basis for the Crowbar cookbooks (his second set), and his final set is the emerging set of cookbooks in OpenStack proper. The third one is interesting and what he plans to continue working on to make into his public openstack cookbooks. These are an amalgamation of smokestack, RCB, Anso improvements, and his (Crowbar’s).
He then demoed his knife plugin (slideshare) to build openstack virtual servers using the Openstack API. This is nice and works against TryStack.org (previously “Free Cloud”) and RCB’s demo cloud. All of that is on his github repo with instructions how to build and use. Matt and I talked about trying to get that into our Crowbar distro.
There were some questions about flow and choice of OpenStack API versus Amazon EC2 API because there was already an EC2 knife set of plugins.
Ziad Sawalha talks about Keystone
Ziad Sawalha is the PLT (Project Technical Lead) for Keystone. He works for Rackspace out of San Antonio. He drove up for the meeting.
He split his talk into two pieces, Incubation Process and Keystone Overview. He asked who was interested in what and focused his talk more towards overview than incubation.
Some key take-aways:
Keystone comes from Rackspace’s strong, flexible, and scalable API. It started as a known quantity from his perspective.
Community trusted nothing his team produced from an API perspective
Community is python or nothing
His team was ignored until they had a python prototype implementing the API
At this point, comments on API came in.
Churn in API caused problems with implementation and expectations around the close of Diablo.
Because comments were late, changes occurred.
Official implementation lagged and stalled into arriving.
API has been stable since Diablo final, but code is changing. that is good and shows strength of API.
Side note from Greg, Keystone represents to me the power of API over Code. You can have innovation around the implementation as long all the implementations have a fair ground work to plan under which is an API specification. The replacement of Keystone with the Keystone Light code base is an example of this. The only reason this is possible is that the API was sound and documented. (Rob’s post on this)
Ziad spent the rest of his time talking about the work flow of Keystone and the API points. He covered the API points.
Client to Keystone, Keystone to Client for initial auth token
Client to Middleware API for the services to have a front.
Middleware to Keystone to verify and establish identity.
Middleware to Service to pass identity
Not many details other then flow and flexibility. He stressed the API design separated protocol from actions and data at all the layers. This allows for future variations and innovations while maintaining the APIs.
Ziad talked about the state of Essex.
RBAC (aka Role Based Access Control)
Code replacement Keystone Light
Federation was planned but will most likely be pushed to G
Federation is the ability for multiple independent Keystones to operate (bursting use case)
Dependent upon two other federation components (networking and billing/metering)
Your’s truly (Rob Hirschfeld) gave the presentation about the OpenStack Foundation. To readers of this blog, it’s obvious that I’m a believer in the OpenStack mission; however, it’s not obvious how creating a foundation helps with that mission and why OpenStack needs its own. As one person at the meetup put it, “Why not? Every major project needs a foundation!”
Governance does not sound sexy compared to writing code and deploying clouds, but it’s very important to the success of the project.
Here are my notes without the poetic elocution I exuded during the meetup…
What: Creating a neutral body to govern OpenStack. Rackspace has been leading OpenStack. This means that they own the copyrights, name and also pay the people who organize the community. They committed (to executives at Dell and others) that they would ultimately setup a standalone body to govern the project before the project was public and endorsed by those early partners. Dell (my employer), Citrix, Accenture and NASA were some of biggest names at the Austin conference launch.
Why: A neutral body is needed because a lot of companies are committing significant time and money to the project. They cannot risk their investments on Rackspace good will alone. This may mean many things. It could be they don’t like Rackspace direction or they feel that Rackspace is not investing enough.
When: Right now and over the next few releases. You should give feedback right now on the OpenStack Foundations mission. The actual foundation will take more time to establish because it requires legal work and funding commitments.
Who: The community – all stakeholders. This is important stuff! While trying to standup a financially independent Foundation, which requires moneys, the little guys are not left out. There is a clear realization and desire to enable independent developers and contributors and small players to have a seat at the table.
How Much: The amounts are unclear, but establishing a foundation will require a significant ongoing investment from highly involved and moneyed parties (Rackspace, Dell, Cisco, HP, Citrix, NTT, startups?, etc). The funding will pay salaries for people dedicated to the community doing the things that I’ll discuss below. Overall, the ROI for those investments must be clear!
The foundation does “governance.” But, what does that mean? Here is a list of vitally important work that the foundation is responsible for.
Branding – Protecting, certifying, and promoting the OpenStack brand is important because it ensures that “OpenStack” has a valuable and predictable meaning to contributors and users. A strong the brand also means a stronger temptation for people to abuse the brand by claiming compatibility, participation and integration.
API – Many would assume that the OpenStack API is the very heart of the project and there is merit to this position. As more and more OpenStack implementations emerge, it is essential that we have a body that can certify which implementations (and even which versions of the implementation!) are valid. This is a substantial value to the community because API integrity ensures project continuity and helps the ecosystem monetize the project. Note: my opinion differs from others here because I think we should favor API over implementation
Community – The OpenStack community is not an accident. It is the function of deliberate actions and choices made by Rackspace and supported by key contributors. That community requires virtual and physical places to coalesce and leaders to organize and manage those meeting places. The excellent conferences, wikis, blogs, media awareness, documentation and meetups are a product of consistent community management.
Arbitration – An open source community is a family and siblings do not always get along. Today, Rackspace must be very careful about balancing their own interests because they are like the oldest sibling playing the parent role – you can get away with it until something serious happens. We need a neutral party so that Rackspace can protect their own interests (alternate spin: because Rackspace protects their own interests at the expense of the community).
Leadership – OpenStack today is a collection of projects with individual leadership. We will increasingly need coordinated leadership as the number of projects and users increases. Centralized leadership is essential because the good of the project as a whole may mean sacrifices within individual projects. It may even mean that some projects chose to leave the OpenStack tent. Stewarding these challenges will require a new level of leadership.
Legal – This is a function of all the above but also something more. From a legal stand point, OpenStack be able to represent itself. There is a significant amount of intellectual property being created. It would be foolish to overlook that this property is valuable and needs adequate legal representation.
I used “vitally important” to describe the above items. Is that an exaggeration? Our goal is collaboration and that requires some infrastructure and rules to make it sustainable. We must have a foundation that encourages innovation (multiple implementations) and collaboration (discourages forking). Innovation and collaboration are the heartbeat of an open source project.
The foundation is vitally important because collaboration by competitors is fragile.
In addition to the core areas above, the foundation needs to handle routine tactical items such as:
Delivering on milestones & releases
Moving new subprojects into OpenStack
Electing and maintaining Project Policy Board
Electing and maintaining Project Technical Leads
Ensuring adherence and extensions to the current bylaws
At the end of the day, OpenStack monetization is the central value for the Foundation.
In order for the OpenStack project, and thus its foundation, to flourish, the contributors, ecosystem, sponsors and users of the project must be able to see a reasonable return (ROI) on their investment. I would love to believe that the foundation is allow about people banding together to solve important problems for the benefit of all; however, it is more realistic to embrace that we can both collaborate and profit simultaneously. Acknowledging the pragmatic self-interested view allows us to create the right incentives and processes as embodied by the OpenStack foundation.
I was very impressed by the quality of discussion at the Deployment topic meeting for Austin OpenStack Meetup (#OSATX). Of the 45ish people attending, we had representations for at least 6 different OpenStack deployments (Dell, HP, ATT, Rackspace Internal, Rackspace Cloud Builders, Opscode Chef)! Considering the scope of those deployments (several are aiming at 1000+ nodes), that’s a truly impressive accomplishment for such a young project.
Even with the depth of the discussion (notes below), we did not go into details on how individual OpenStack components are connected together. The image my team at Dell uses is included below. I also recommend reviewing Rackspace’s published reference architecture.
Our deployment discussion was a round table so it is difficult to link statements back to individuals, but I was able to track companies (mostly).
picked Ubuntu & KVM because they were the most vetted. They are also using Chef for deployment.
running Diablo 2, moving to Diablo Final & a flat network model. The network controller is a bottleneck. Their biggest scale issue is RabbitMQ.
is creating their own Nova Volume plugin for their block storage.
At this point, scale limits are due to simultaneous loading rather than total number of nodes.
The Nova node image cache can get corrupted without any notification or way to force a refresh – this defect is being addressed in Essex.
has setup availability zones are completely independent (500 node) systems. Expecting to converge them in the future.
is using the latest Ubuntu. Always stays current.
using Puppet to setup their cloud.
They are expecting to go live on Essex and are keeping their deployment on the Essex trunk. This is causing some extra work but they expect it to pay back by allowing them to get to production on Essex faster.
Deploying on XenServer
“Devs move fast, Ops not so much.” Trying to not get behind.
Rackspace Cloud Builders (RCB) is running major releases being run through an automated test suite. The verified releases are being published to https://github.com/cloudbuilders (note: Crowbar is pulling our OpenStack bits from this repo).
Dell commented that our customers are using Crowbar primarily pilots – they are learning how to use OpenStack
Said they have >10 customer deployments pending
ATT is using OpenSource version of Crowbar
Need for Keystone and Dashboard were considered essential additions to Diablo
KVM is considered the top one for now
Libvirt (which uses KVM) also supports LXE which people found to be interesting
XenServer via XAPI are also popular
No so much activity on ESX & HyperV
We talked about why some hypervisors are more popular – it’s about the node agent architecture of OpenStack.
NetApp via Nova Volume appears to be a popular block storage
Keystone / Dashboard
Customers want both together
Including keystone/dashboard was considered essential in Diablo. It was part of the reason why Diablo Final was delayed.
HP is not using dashboard
Members of the Audience made comments that we need to deprecate the EC2 APIs (because it does not help OpenStack long term to maintain EC2 APIs over its own). [1/5 Note: THIS IS NOT OFFICIAL POLICY, it is a reflection of what was discussed]
HP started on EC2 API but is moving to the OpenStack API
Next meeting is Tuesday 1/10 and sponsored by SUSE (note: Tuesday is just for this January). Topic TBD.
We’ve got sponsors for the next SIX meetups! Thanks for Dell (my employeer), Rackspace, HP, SUSE, Canonical and PuppetLabs for sponsoring.
We discussed topics for the next meetings (see the post image). We’re going to throw it to a vote for guidance.
The OSATX tag is also being used by Occupy San Antonio. Enjoy the cross chatter!
The Rackspace RA and Crowbar deployment barclamps share the same objective: sharing of best practices for OpenStack operations.
Over the last 12+ months, my team at Dell has had the opportunity to work with many customers on OpenStack deployment designs. While no two of these are identical, they do share many similarities. We are pleased to collaborate with Rackspace and others on capturing these practices as operational code (or “opscode” if you want a reference to the Chef cookbooks that are an intrinsic part of Crowbar’s architecture).
In our customer interactions, we hear clearly that Crowbar must remain flexible and ready to adapt to both customer on-site requirements and evolution within the OpenStack code base. You are also telling us that there is a broader application space for Crowbar and we are listening to that too.
I believe that it will take some time for the community and markets to process today’s Rackspace announcements. Rackspace is showing strong leadership in both sharing information and commercialization around OpenStack. Both of these actions will drive responses from the community members.
Jon Dickinson who is the Project Technical Lead for Swift (Object Storage) was there and presented information on the current Swift offering; It is interesting to note that Swift releases continuously when most of OpenStack releases during the 6 month development cycle like Nova (Compute)
Stephen and Jim Plamondon from Rackspace presented information on the overall community and talked about the announcement yesterday from Internap about their Compute public cloud and the information on the MercadoLibre 600 Node Compute cloud running their business:
“With 58 million users of MercadoLibre.com and growing rapidly, we need to provide our teams instant access to computing resources without heavy administrative layers. With OpenStack, our internal users can instantly provision what they need without having to wait for a system administrator,” said Alejandro Comisario, Infrastructure Senior Engineer, MercadoLibre, the largest online trading platform in Latin America. “With our success running OpenStack Compute in production, we plan to roll OpenStack Diablo out more broadly across the company, and have appreciated the community support in this venture, especially through the OpenStack Forums, where we are also global moderators.”
Discussion on the OpenStack API Issue which is a significant open issue at this time – should OpenStack focus on creating an API specification and then let multiple implementations of that API move forward or build 1 implementation of the API as official OpenStack (see my post for more on this).