I spent Tuesday and Wednesday at DevOpsDaysAustin and continue to be impressed with the enthusiasm and collaborative nature of the DOD events. We also managed to have a very robust and engaged twitter backchannel thanks to an impressive pace set by Gene Kim!
DevOpsDays spends a lot of time talking about culture. I’m a huge believer on the importance of culture as the foundation for the type of fundamental changes that we’re making in the IT industry; however, it’s also a sign that we’re still in the minority if we have to talk about culture evangelism.
Process and DevOps are tightly coupled. It’s very clear that Lean/Agile/Kanban are essential for DevOps success (nice job by Dominica DeGrandis). No one even suggested DevOps+Waterfall as a joke (but Patrick Debois had a picture of a xeroxed butt in his preso which is pretty close).
Still need more Devs people to show up! My feeling is that we’ve got a lot of operators who are engaging with developers and fewer developers who are engaging with operators (the “opsdev” people).
Chef Omnibus installer is very compelling. This approach addresses issues with packaging that were created because we did not have configuration management. Now that we have good tooling we separate the concerns between bits, configuration, services and dependencies. This is one thing to watch and something I expect to see in Crowbar.
The old mantra still holds: If something is hard, do it more often.
Not DevOps, but 3D printing is awesome. This is clearly a game changing technology; however, it takes some effort to get right. Dell brought a Solidoodle 3D printer to the event to try and print OpenStack & Crowbar logos (watch for this in the future).
I’d be interested in hearing what other people found interesting! Please comment here and let me know.
“Double wide” is not a term I’ve commonly applied to servers, but that’s one of the cool things about this new class of servers that Dell, my employer, started shipping today.
My team has been itching for the chance to start cloud and big data reference architectures using this super dense and flexible chassis. You’ll see it included in our next Apache Hadoop release and we’ve already got customers who are making it the foundation of their deployments (Texas Adv Computing Center case study).
If you’re tracking the latest big data & cloud hardware then the Dell PowerEdge C8000 is worth some investigation.
Basically, the Dell C8000 is a chassis that holds a flexible configuration of compute or storage sleds. It’s not a blade frame because the sleds minimize shared infrastructure. In our experience, cloud customers like the dedicated i/o and independence of sleds (as per the Bootstrapping clouds white paper). Those attributes are especially well suited for Hadoop and OpenStack because they support a “flat edges” and scale out design. While i/o independence is valued, we also want shared power infrastructure and density for efficiency reasons. Using a chassis design seems to capture the best of both worlds.
The novelty for the Dell PowerEdge C8000 is that the chassis are scary flexible. You are not locked into a pre-loaded server mix.
There are a plethora of sled choices so that you can mix choices for power, compute density and spindle counts. That includes double-wide sleds positively brimming with drives and expanded GPU processers. Drive density is important for big data configurations that are disk i/o hungry; however, our experience is the customer deployments are highly varied based on the planned workload. There are also significant big data trends towards compute, network, and balanced hardware configurations. Using the C8000 as a foundation is powerful because it can cater to all of these use-case mixes.
This release raises the bar on open Hadoop deployments by making them faster, scalable, more integrated and repeatable.
These barclamps were developed in conjunction with our licensed Dell | Cloudera Solution. The licensed solution is for customers seeking large scale and professionally supported big data solutions. The purpose of the open barclamps (which pull the open source parts from the Cloudera distro) is to help you get started with Hadoop and reduce your learning curve. Our team invested significant testing effort in ensuring that these barclamps work smoothly because they are the foundational layer of our for-pay Hadoop solution.
Included in the Hadoop barclamp suite are Hadoop Map Reduce, Hive, Pig, ZooKeeper and Sqoop running on RHEL 5.7. These barclamps cover the core parts of the Hadoop suite. Like other Crowbar deployments (see OpenStack), the barclamps automatically discover the service configurations and interoperate. One of our team members (call him Scott Jensen) said it very simply “I can deploy a fully an integrated Hadoop cluster in a few hours. That friggin’ rocks!” I just can’t put it more eloquently than that!
I’ll post again when we flip the “open” bit and invite our community to dig in and help us continue to set the standards on open Hadoop deployments.
Jon Dickinson who is the Project Technical Lead for Swift (Object Storage) was there and presented information on the current Swift offering; It is interesting to note that Swift releases continuously when most of OpenStack releases during the 6 month development cycle like Nova (Compute)
Stephen and Jim Plamondon from Rackspace presented information on the overall community and talked about the announcement yesterday from Internap about their Compute public cloud and the information on the MercadoLibre 600 Node Compute cloud running their business:
“With 58 million users of MercadoLibre.com and growing rapidly, we need to provide our teams instant access to computing resources without heavy administrative layers. With OpenStack, our internal users can instantly provision what they need without having to wait for a system administrator,” said Alejandro Comisario, Infrastructure Senior Engineer, MercadoLibre, the largest online trading platform in Latin America. “With our success running OpenStack Compute in production, we plan to roll OpenStack Diablo out more broadly across the company, and have appreciated the community support in this venture, especially through the OpenStack Forums, where we are also global moderators.”
Discussion on the OpenStack API Issue which is a significant open issue at this time – should OpenStack focus on creating an API specification and then let multiple implementations of that API move forward or build 1 implementation of the API as official OpenStack (see my post for more on this).
First, we’ll have our Crowbar demo rack showcasing LIVE MULTI-NODE DIABLO DEPLOYMENTS and some IMPORTANT FEATURE AND COMMUNITY ADDITIONS. No spoilers here – you’ll have to come by. Of course, it’s in the git hub too, but we’ve put a bow on it.
Second, there’s a DEPLOYMENT BLUE PRINT discussion about getting better interlocks between OpenStack development and deployment. We really need to reduce the pain and lag between adding great features and using those features.
Next, we’ve got a limited audience CONCEPT SNEAK PEEK for something from our labs that we think is very interesting and we’d like to get input about. Unfortunately, we’re very limited with space & time for this whisper session so you’ll need to contact OpenStack@Dell.com to request an invitation.
Finally, at the Conference, you can see OUR TEAM IN ACTION:
Thurs 11:30 – Dell Keynote by John Igoe
Thurs 3:30 – Private Cloud Panel w/ Rob Hirschfeld
Wow, this Webinar was an hour of OpenStack insights (see the whole thing). If you don’t have the hour then you can use my time line nodes to jump to what you want to hear.
2:50: Presentation Starts (introductions are over)
3:40: Joseph coins the word “dynormous” for dynamic & large scale clouds
4:40: Customers want to know how they are going to maintain a cloud
4:50: Customers don’t want a 9 month cycle for features, want it faster. DevOps gives us the flexibility to meet our customer needs as quickly as they want to.
7:11: Massive scalability… their (Rackspace & NASA) business is about scale
8:00: Rackspace and NASA started from the beginning to build a community
8:50: We have the data that this has staying power
10:10: We see a lot of companies joining in the community
11:56: Shout out to Opscode Chef
12:40: From bare metal to a fully functioning cloud in under 2 hours. Crowbar allows you to introduce new elements into the environment
13:40: Crowbar leverages our experience with cloud deployments
14:33: Dell was the only provider there from day 1. We have the most experience.
17:27: DevOps Poll
18:40: DevOps is a significant trend that you should consider. Hosters have a lot of operational chops.
19:34: There are a lot of right ways to do cloud. You need to pick what’s best for your business model
20:23: We could get hardware and software, but operational expertise was missing.
21:33: We’re more making the complexity of a cloud go away. We are getting our customers a head start. We are chipping away at the learning curve.
22:05: The cloud is always ready, never finished. Cloud is an ongoing operational environment: DevOps!
23:30: Crowbar bakes a lot of operational experience into the deployment.
25:17: Core tenant of DevOps: there is no single OpenStack image. Cloud is too complex. We build it in layers.
26:26: Before you deploy, you can change the configuration.
27:30: Barclamps are modules that execute a function. We are inviting community participation
28:40: Crowbar process view – Crowbar is a “PXE state machine” is a very simplified description.
29:57: You can go through a tuning cycle where you can get it working, make sure it’s right, flush and reset. That ensures you have an automated system.
30:34: Screen shots with descriptions
33:25: Event the core state machine that runs Crowbar is deployed as a barclamp
35:00: You can download OpenStack and install it yourself from our github. We don’t want to talk about OpenStack, we want to DO OpenStack.
Poll Results (see to the right)
38:00: Online resources
40:00: Question 1: Timeline for RHEL. Answer: RHEL is part of Hadoop, will make it into OpenStack by end of year (or sooner based on market demand)
42:17: Question 2: What led Dell to get involved in OpenStack? Answer: It’s about experience. We like being able to fix and change if we needed. There is a lot of active community
45:30: Question 3: How does a hardware maker play with open source software? Answer: It’s a solution for us. We wanted to make sure that people cloud deploy the software. Adding DevOps takes it to another level.
48:00 Question 4: What elements of Diablo are most exciting? Answer: Keystone (centralized authentication) is a big deal. Networking changes that “bust the top” of the networking hurdles.
50:25: Question 5: Where is OpenStack going long term? Answer: We’re pleasantly surprised about how much it’s picked up. We’ll see more standards in the community. We have high hopes for OpenStack and have invested heavily. We’ll see more as-a-service capabilities to build on a common infrastructure: both open and commercial.
52:47: Question 6: What’s the biggest barrier to operating at scale? Answer: learning how to operate is the biggest hurdle. We took a learning approach to help customers get started. We are hosting a training with Rackspace.
55:00: Question 7: Where does Dell and Rackspace overlap? Answer: We see Rackspace Cloud Builders that the premier experts. Dell Services is involved with all of it. Dell takes the phone call and deals with our customers directly.
Crowbar started as a Dell OpenStack installer project and then grew beyond that in scope. Now it can be extended to work with other vendors’ kits and other solutions bits.
We are contributing Crowbar to the community because we believe that everyone benefits by sharing in the operational practices that Crowbar embodies. These are rooted in Opscsode Chef (which Crowbar tightly integrates with) and the cloud & hyper-scale proven DevOps practices that are reflected in our deployment model.
Build scripts so you can create your own Crowbar install ISO
Switch discovery so you can create Chef Cookbooks that are network aware.
Open source Chef server that powers much of Crowar’s functionality
What’s not included?
Non-open source license components (BIOS+RAID config) that we could not distribute under the Apache 2 license. We are working to address this and include them in our release. They are available in the Dell Licensed version of Crowbar.
Dell Branded Components (skin + overview page). Crowbar has an OpenSource skin with identical functionality.
Pre-built ISOs with install images (you must download the open source components yourself, we cannot redistribute them to you as a package)
Crowbar uses Chef Server as its database and relies on cookbooks for node deployments. It is installed (using Chef Solo) automatically as part of the Crowbar install.
Note: Rackspace rented out the Kung Fu Saloon all day Monday, and are doing various events — from live webinars to a Scoble tweetup to a happy hour and more VIP after hours event.
The demo will be orchestrated by Greg Althaus from my team at Dell. Greg is the primary architect for Crowbar and responsible for some of it’s amazing capabilities including the Chef integrations, network discovery and rockin’ PXE state machine. Dell Cloud Evanglist, Barton George, will also be on hand.