Category Archives: Clouds

Hadoop Crowbar released to open source! (plus AN HOUR of videos!)

Posted on by
3

I’m proud to announce that my team at Dell has open sourced our Apache Hadoop barclamps!  This release follows our Dell | Cloudera Hadoop Solution open source commitment from Hadoop World earlier this month.

As part of this release, we’ve created nearly AN HOUR of video content showing the Hadoop Barclamps in action, installing Crowbar (on CentOS), building Crowbar ISOs in the cloud and specialized developer focused builds.

If you want to talk to the Crowbar team.  We’re attending events in Boston 11/29, Seattle 11/30, and Austin 12/8.

Here are links to the videos:

More Hadoop perspectives from Dell:  Joseph George on what it means and  Barton George‘s backgrounder about barclamps.

Seattle meetup on 11/30 (will bring massive laptops for OpenStack, Hadoop & Crowbar demos)

Posted on by
Reply

After Greg Althaus and I are done attending the sold out Opcode Community Summit (11/29-30), Opscode has offered to let us have an informal meetup at Opscode HQ from 6:30 to 8pm on 11/30.  I’ve proposed this as an official Seattle OpenStack Meetup (waiting on confirmation from @heckj).

We’re not limiting the agenda to OpenStack!  We’ll happily talk about Hadoop, Crowbar, Opscode or any other cloud technology that’s on your mind.  For 90 minutes, we’re offering Cloud Geeking as a Service (CGaaS).

Not in Seattle?  Never fear!  You can hook up with other members of my team at Dell in Boston on 11/29 & Austin 12/8.

Crowbar community support and 111111 sprint plan

Posted on by
Reply

The Dell Crowbar team is working to improve road map transparency. In the last few weeks, the Crowbar community has become more active on our lists, testing builds, and helping with documentation.

We love the engagement and continue to make supporting the list a priority.

Participation in Crowbar, OpenStack and Hadoop has been exceeding our expectations and we’re working to implement more community support and process. Thank you!!!

Our next steps:

  1. I’ve committed to post sprint plans and summary pages (this is the first)
  2. New Crowbar Twitter account
  3. I’m going to setup feature voting on the Crowbar Facebook page (like to vote)
  4. Continue to work the listserv and videos. We need help converting those to documentation on the crowbar wiki.
  5. Formalize collaborator agreements – we’re working with legal on this
  6. Exploring the option of a barclamp certification program and Crowbar support
  7. Moving to a gated trunk model for internal commits to improve quality
  8. Implementing a continuous integration system that includes core and barclamps. This will be part of our open source components.

We are working towards the 1.2 release (Beta 1) . That release is focused on supporting OpenStack but includes enhancements for upgrades, Hadoop, and additional OS support.

Our Sprint 111111 plan.

Source: Crowbar Wiki: [[sprint 111111]]

  • Theme: OpenStack Diablo Final release candidate.
  • Core Work: Refine Deployment for Nova, Glance, Nova Dashboard (horizon), keystone, swift
  • New additions: mySQL barclamp, Nova HA networking, kong
  • Crowbar internals: expose error states for proposals, allow packages to be included with barclamps to make upgrades easier, barclamp group pages
  • Operating system: added CentOS
  • Documentation: we’ve split the user guides into distinct books so Crowbar, OpenStack, and Hadoop each have their own user guide.
  • Pending action: expose the Hadoop barclamps
  • OS note: OpenStack is being tested (at Dell) against Ubuntu 10.10 only. Hadoop was tested against RHEL 5.7 and we expect it to work against CentOS also.

Rackspace unveils OpenStack reference architecture & private cloud offering

Posted on by
1

Yesterday, Rackspace Cloud Builders unveiled both their open reference architecture (RA) and a private cloud offering (on GigaOM) based upon the RA.  The RA (which is well aligned with our Dell OpenStack RA) does a good job laying out the different aspects of an OpenStack deployment.  It also calls for the use of Dell C6100 servers and the open source version of Crowbar.

The Rackspace RA and Crowbar deployment barclamps share the same objective: sharing of best practices for OpenStack operations.

Over the last 12+ months, my team at Dell has had the opportunity to work with many customers on OpenStack deployment designs.  While no two of these are identical, they do share many similarities.  We are pleased to collaborate with Rackspace and others on capturing these practices as operational code (or “opscode” if you want a reference to the Chef cookbooks that are an intrinsic part of Crowbar’s architecture).

In our customer interactions, we hear clearly that Crowbar must remain flexible and ready to adapt to both customer on-site requirements and evolution within the OpenStack code base.  You are also telling us that there is a broader application space for Crowbar and we are listening to that too.

I believe that it will take some time for the community and markets to process today’s Rackspace announcements.  Rackspace is showing strong leadership in both sharing information and commercialization around OpenStack.  Both of these actions will drive responses from the community members.

Dell is open sourcing Crowbar Apache Hadoop barclamps!

Posted on by
6

I’m very excited to announce that my team at Dell will be open sourcing our Apache Hadoop Crowbar barclamps by the end of the month.

This release raises the bar on open Hadoop deployments by making them faster, scalable, more integrated and repeatable.

These barclamps were developed in conjunction with our licensed Dell | Cloudera Solution. The licensed solution is for customers seeking large scale and professionally supported big data solutions. The purpose of the open barclamps (which pull the open source parts from the Cloudera distro) is to help you get started with Hadoop and reduce your learning curve. Our team invested significant testing effort in ensuring that these barclamps work smoothly because they are the foundational layer of our for-pay Hadoop solution.

Included in the Hadoop barclamp suite are Hadoop Map Reduce, Hive, Pig, ZooKeeper and Sqoop running on RHEL 5.7. These barclamps cover the core parts of the Hadoop suite. Like other Crowbar deployments (see OpenStack), the barclamps automatically discover the service configurations and interoperate. One of our team members (call him Scott Jensen) said it very simply “I can deploy a fully an integrated Hadoop cluster in a few hours. That friggin’ rocks!” I just can’t put it more eloquently than that!

I’ll post again when we flip the “open” bit and invite our community to dig in and help us continue to set the standards on open Hadoop deployments.

For more perspectives on this release, check out posts by Barton George (just for devs), Joseph George (About Hadoop) and Aurelian Dumitru

Barton posted these two videos of me talking about the release too:

Hadoop & Crowbar:

Dev’s Only Short:

Talk with Team Crowbar! Online 11/8, Austin 11/15, Boston 11/29 & 11/29 & Seattle 11/30

Posted on by
2

My team at Dell has been getting a great response from our community about Crowbar. Thanks! We’re actively working a rock solid OpenStack deployment that will raise the bar on ease of deploy and drive operational excellence.

We have also heard that we need to improve access to the team; consequently, I’m delighted to announce a long list of places and dates where you can access us online AND in person.

Here’s the list:

Or in a calendar view:

Sun Mon Tuesday Wed Thursday Fri Sat
11/8 Online
Crowbar Chat
11/15 Austin
Cloud User
11/29 Boston
OpenStack Meetup		 11/30 Seattle
Crowbar Drinks TBD
12/6 Boston
Opscode BoaF		 12/8 Austin
OpenStack Meetup

Future watch out, the next generation is IP enabled

Posted on by
3

This morning, I found that my 9 year old daughter had written the IP address of a friend’s server (likely Minecraft) on her arm so that they could collaborate together after school.

This is NOT just proud daughters of geeks: this type of collaborative play is something common to the entire new culture of Internet users.   These “digital natives” use technology at a whole new level.

It’s not their ability to use technology that’s important – it’s the new culture they are creating. 

This new generation is much more team oriented, risk tolerant, connected, and information oriented than previous generations.

I’ve posted about this before (Liquid Leadership) but this picture captures the entire culture clash in a single image.

PS: I wonder what she would do if it was an IPv6 address?  She’s not tall!

The 451 Group Cloudscape report strikes chord misses harmony (DevOps, Hybrid Cloud, Orchestration)

Posted on by
3

It’s impossible to resist posting about this month’s  451 Group Cloudscape report when it calls me out by name as a leading cloud innovator:

… ProTier founders Dave McCrory and Rob Hirschfeld. ProTier [note: now part of Quest] was, indeed, the first VMware ecosystem vendor to be tracked by The 451 Group. In the face of a skeptical world, these entrepreneurs argued that virtualization needed automation in order to realize its full potential, and that the test lab was the low-hanging fruit. Subsequent events have more than vindicated their view (pg. 33).

It’s even better when the report is worth reading and offers insights into forces shaping the industry.  It’s nice to be “more than vindicated” on an amazing journey we started over 10 years ago!

Rather than recite 451’s points (hybrid cloud = automation + orchestration + devops + pixie dust), I’d rather look at the problem different way as a counterpoint.

The problem is “how do we deal with applications that are scattered over multiple data centers?”

I do not think orchestration is the complete answer.  Current orchestration is too focused on moving around virtual machines (aka workloads).

Ultimately, the solution lies in application architecture; however, I feel that is also a misdirection because cloud is redefining what an “application architecture” means.

Applications are a dynamic mix of compute, storage, and connectivity.

We’re entering an age when all of these ingredients will be delivered as elastic services that will be managed by the applications themselves.  The concept of self management is an extension of DevOps principles that fuse application function and deployment. There are missing pieces, but I’m seeing the innovation moving to fill those gaps.

If you want to see the future of cloud applications then look at the network and storage services that are emerging.  They will tell you more about the future than orchestration.

 

Notes from 10/27 OpenStack Austin Meetup (via Stephen Spector)

Posted on by
2

Stephen Spector (now a Dell Services employee!) gave me permission to repost his excellent notes from the first OpenStack Austin (#OSATX) Meetup Group.

Here are his notes:

[Stephen] wanted to update everyone on the Austin OpenStack Meetup last night at the Austin TechRanch sponsored by Joseph and Rob (that’s me!) of the Dell OpenStack team (I think I got that right?). You can find all the tweets from the event at https://twitter.com/#!/search/%23osatx as we created a new hashtag for tweeting during the event, #osatx.

Here are some highlights from the event:

  • About 60 or so attendees with a good amount from Dell (Barton George, Logan McCloud)and Rackspace, Opscode (Matt Ray), Puppet Labs, SUSE talked about their OpenStack commitment (http://t.co/bBnIO7xv), and Ubuntu folks as well
  • Jon Dickinson who is the Project Technical Lead for Swift (Object Storage) was there and presented information on the current Swift offering; It is interesting to note that Swift releases continuously when most of OpenStack releases during the 6 month development cycle like Nova (Compute)
  • Stephen and Jim Plamondon from Rackspace presented information on the overall community and talked about the announcement yesterday from Internap about their Compute public cloud and the information on the MercadoLibre 600 Node Compute cloud running their business:

“With 58 million users of MercadoLibre.com and growing rapidly, we need to provide our teams instant access to computing resources without heavy administrative layers. With OpenStack, our internal users can instantly provision what they need without having to wait for a system administrator,” said Alejandro Comisario, Infrastructure Senior Engineer, MercadoLibre, the largest online trading platform in Latin America. “With our success running OpenStack Compute in production, we plan to roll OpenStack Diablo out more broadly across the company, and have appreciated the community support in this venture, especially through the OpenStack Forums, where we are also global moderators.”

  • Discussion on the OpenStack API Issue which is a significant open issue at this time – should OpenStack focus on creating an API specification and then let multiple implementations of that API move forward or build 1 implementation of the API as official OpenStack (see my post for more on this).
  • Greg Althaus gave a demo of the Nova Dashboard
  • Future Meetings
  • Three organizations have offered to help host (pizza $ and TechRanch space $) but we always need more!  You can offer to sponsor via the meetup site.
  • There will be future OpenStack Austin Meetups so sign up for the group and you’ll be notified automatically.

Pictures…

Continue reading →

OpenStack: Five Challenges & Conference Observations

Posted on by
Reply

I was part of the Dell contingent at the OpenStack design conference earlier in the month.  The conference opened a new chapter for the project because the number of contributing companies reached critical mass.  That means that the core committers are no longer employed by just one or two entities; instead, there are more moneyed interests rubbing elbows and figuring out how to collaborate.

From my perspective (from interview with @Cote ), this changed the tone of the conference from more future looking to pragmatic.

That does not mean that everything is sunshine and rainbows for OpenStack clouds, there are real issues to be resolved.  IMHO, the top issues for OpenStack are:

  1. API implementation vs specification
  2. Building up coverage on continuous integration
  3. Ensuring that we can deploy consistently in multi-node systems
  4. Getting contributions from new members
  5. Figuring out which projects are core, satellite, missing or junk.  [xref 2014 DefCore]

Of these issues, I’ve been reconsidering my position favoring API via Implementation over specification (past position).  This has been a subject of debate on my team at Dell and I like Greg Althaus’ succinct articulation of the problem with implementation driven API: “it is not fair.”  This also ends up being a branding issues for OpenStack because governance needs to figure out which is a “real” OpenStack cloud deployment that can use the brand.  Does it have to be 100% of the source?  What about extensions?  What if it uses the API with an alternate implementation?

Of the other issues, most are related to maturity.  I think #2 needs pressure by and commitment from the larger players (Dell very much included).  Crowbar and the deployment blueprint is our answer #3.  Shouting the “don’t fork it up” chorus from the roof tops addresses contributions while #5 will require some strong governance and inevitably create some hurt feelings.