Tag Archives: Open Source

7 takeaways from DevOps Days Austin

Posted on by
10

Block Tables

I spent Tuesday and Wednesday at DevOpsDays Austin and continue to be impressed with the enthusiasm and collaborative nature of the DOD events.  We also managed to have a very robust and engaged twitter backchannel thanks to an impressive pace set by Gene Kim!

I’ve still got a 5+ post backlog from the OpenStack summit, but wanted to do a quick post while it’s top of mind.

My takeaways from DevOpsDays Austin:

  1. DevOpsDays spends a lot of time talking about culture.  I’m a huge believer on the importance of culture as the foundation for the type of fundamental changes that we’re making in the IT industry; however, it’s also a sign that we’re still in the minority if we have to talk about culture evangelism.
  2. Process and DevOps are tightly coupled.  It’s very clear that Lean/Agile/Kanban are essential for DevOps success (nice job by Dominica DeGrandis).  No one even suggested DevOps+Waterfall as a joke (but Patrick Debois had a picture of a xeroxed butt in his preso which is pretty close).
  3. Still need more Devs people to show up!  My feeling is that we’ve got a lot of operators who are engaging with developers and fewer developers who are engaging with operators (the “opsdev” people).
  4. Chef Omnibus installer is very compelling.  This approach addresses issues with packaging that were created because we did not have configuration management.  Now that we have good tooling we separate the concerns between bits, configuration, services and dependencies.  This is one thing to watch and something I expect to see in Crowbar.
  5. The old mantra still holds: If something is hard, do it more often.
  6. Eli Goldratt’s The Goal is alive again thanks to Gene Kims’s smart new novel, The Phoenix project, about DevOps and IT  (I highly recommend both, start with Kim).
  7. Not DevOps, but 3D printing is awesome.  This is clearly a game changing technology; however, it takes some effort to get right.  Dell brought a Solidoodle 3D printer to the event to try and print OpenStack & Crowbar logos (watch for this in the future).

I’d be interested in hearing what other people found interesting!  Please comment here and let me know.

From orphans to open source, data matters

Posted on by
2

TMF ChildrenMy wife’s day job helps Indian orphans through the Miracle Foundation here in Austin. On the surface, our jobs are very different; however, there is lately more and more intersection in both form and substance. It was not always like that, initially the Miracle Foundation primary engagement had been an emotional appeal: “look, these orphans are sad, they need you. Did we mention that they are orphans?”

Joking aside, there are plenty of kind people who want to help children; however, there are a lots of worthy causes with equally strong appeal. The question is how do you pick which one? Donors/Contributors want one that is both emotionally appealing and effective.

While radically different in human impact, both of raising orphans and building open source rely heavily on personal engagement and passion for success. Just like non-profits, there are many open source projects that want you to invest your time in installing and contributing to their most worthy technology.

About 18 months ago, the Miracle Foundation pivoted their strategy from tending individual children towards cultivating whole orphanages (the “NEST program”, video below). They started tracking things like how much milk and fruit each child ate and if they had been vaccinated. They connected observable data like hemoglobin levels of children to their ability to pay attention in school. They were even aware of additional days girls spent in school just because they got monthly hygiene products.

NEST Spider Graph

Used with Permission, The Miracle Foundation

With this new program, the Miracle Foundation can tell you exactly how much benefit each child will receive from each dollar. These are real results derived from collecting real data, and the results are powerful.

The children the Miracle Foundation nurtures are going from subsistence to flourishing. This is not happening because people care more about these children than before. It is happening because someone is keeping the data and making sure that the support they give gets the results they want. This in turn helps donors (become one) feel confident that their emotional response is delivering tangible improvements. Both are essential to TMF’s mission.

Open source projects have a similar gestalt.

People and companies contributing time and resources to a project want to both believe in the technology and see tangible metrics to validate adoption. Open source transparency makes it easier to find active projects and people are engaged contributors, but it can be harder to determine if the project is having broader impact.

For OpenStack, these tangible metrics began to surface in the last few days. Before the summit, Stephano Muffulli, community manager, launched the OpenStack Activity Board to show commit and quality data for the project. Last Monday, Tim Bell & Ryan Lane presented the results of the first user survey which showed how and what users are adopting for OpenStack.

If you like seeing this type of data driven behavior then vote with your keyboard and become part of an active open source project. For non-profits like the Miracle Foundation, voting is even easier – you just need a credit card to join in their Mothers’ Day campaign. Your mom may not understand anything you add to open source, but she can understand when you help orphans.

Continue reading →

Creating Communities: the intersection between Twitter celebrities and open source

Posted on by
Reply

calvin_leeOne of the unexpected perks of my Chevy SXSW experience was access to some real social media celebrities such as Josh Estrin, Calvin Lee, Kristin Brandt, Doug Mora, Samantha Needham and Jennie Chen.  They are all amazing, fun, wicked smart and NOT INTO CLOUD COMPUTING.

While I already knew Samantha (via Dell) and Jennie (via TechRanch), all of Chevy’s guests brought totally different perspectives to Chevy’s SXSW team ranging from pop culture  and mommies to hypermilers and gearheads.

The common thread is that we are all looking to engage our communities.

We each wanted to find something that would be interesting for our very different audiences to discuss.  That meant using our experiences at SXSW, Chevy and with each other to start a conversation within our communities.  We need good content as a seed but the goal is to drive the interaction.

Josh was the most articulate about this point saying that he measured his success when his followers talked to each other more than to him.   Being able to create content that engages people to do that is a true talent.

Calvin’s focus was more on helping people connect.  He felt successful when he was able to bring people together through his extended network. In those cases and others, the goals and challenges of a social media celebrity were remarkably similar to those helping lead open source projects.

In building communities, you must measure success in member communication and interaction.

If you are intent on being at the center of the universe then your project cannot grow; however, people also need celebrities to bring them together.  The amazing thing about the the people I met at SXSW through Chevy is that they managed to both attract the attention needed to build critical mass and get out of the way so communities could form around them. That’s a skill that we all should practice and foster.

PS: I also heard clearly that “I ate …” tweets are some of their most popular.  Putting on my collaboration hat: if you’re looking to engage a community then food is the most universal and accessible discussion topic.  Perhaps I’ll have to eat crow on that one.  

Crowbar Celebrates 1st Anniversary

Posted on by
1

Nearly a year ago at OSCON 2011, my team at Dell opened sourced “Crowbar, an OpenStack installer.” That first Github commit was a much more limited project than Crowbar today: there was no separation into barclamps, no distinct network configuration, one operating system option and the default passwords were all “openstack.” We simply did not know if our effort would create any interest.

The response to Crowbar has been exciting and humbling. I most appreciate those who looked at Crowbar and saw more than a bare metal installer. They are the ones who recognized that we are trying to solve a bigger problem: it has been too difficult to cope with change in IT operations.

During this year, we have made many changes. Many have been driven by customer, user and partner feedback while others support Dell product delivery needs. Happily, these inputs are well aligned in intent if not always in timing.

  • Introduction of barclamps as modular components
  • Expansion into multiple applications (most notably OpenStack and Apache Hadoop)
  • Multi-Operating System
  • Working in the open (with public commits)
  • Collaborative License Agreements

Dell‘s understanding of open source and open development has made a similar transformation. Crowbar was originally Apache 2 open sourced because we imagined it becoming part of the OpenStack project. While that ambition has faded, the practical benefits of open collaboration have proven to be substantial.

The results from this first year are compelling:

  • For OpenStack Diablo, coordination with the Rackspace Cloud Builder team enabled Crowbar to include the Keystone and Dashboard projects into Dell’s solution
  • For OpenStack Essex, the community focused work we did for the March Essex Hackday are directly linked to our ability to deliver Dell’s OpenStack-Powered Essex solution over two months earlier than originally planned.
  • For Apache Hadoop distributions for 3.x and 4.x with implementation of Cloudera Manager and eco system components.
  • We’ve amassed hundreds of mail subscribers and Github followers
  • Support for multiple releases of RHEL, Centos & Ubuntu including Ubuntu 12.04 while it was still in beta.
  • SuSE does their own port of Crowbar to SuSE with important advances in Crowbar’s install model (from ISO to package).

We stand on the edge of many exciting transformations for Crowbar’s second year. Based on the amount of change from this year, I’m hesitant to make long term predictions. Yet, just within next few months there are significant plans based on Crowbar 2.0 refactor. We have line of site to changes that expand our tool choices, improve networking, add operating systems and become more even production ops capable.

That’s quite a busy year!

Open Source Vision, Strategy & Execution: The Community Garden Analogy

Posted on by
1

I’ve been working on a white paper series about open source culture and projects based on my experience with at Dell with Crowbar and OpenStack.  I’m hoping to show off the first result of that collaboration soon, until then I’m glad to share some ideas that we’ve been throwing around to help explain the fundamental shift that we see taking place in the technology community.

It’s obvious that executing a collaborative open source project is fundamentally different than a either a licensed or an open single contributor project.

However, we have yet to describe the culture, process and success criteria needed to drive a collaborative open source project.

I am not saying that projects are not being successful – there are many great examples (OpenStack, CloudFoundry, Apache). My point is that we I do not have articulated the vision, strategy or execution needed for people grounded in traditional software delivery to understand why these projects are different and how to navigate the differences. Lack of alignment within the lead delivery team can be highly disruptive to the project.

Community gardening is all about people working together to produce something tangible that is bigger than they could accomplish individually. Each participant has an expectation of return – a garden is not a charity, it must produce; however, there is a community drive rewards a garden wide focus.

  • You could just pull weeds from your plot, but doing extra means that the garden as a whole will prosper.
  • Fixing the fence keeps the “MQ” rabbits out of your carrots and your neighbors lettuce
  • The person growing the mint also keeps out the ants and so helps the entire garden.
  • The oddball that wants inverted chinese radishes may also be an expert vermiculturist who nurtures the whole worm population as a side interest
  • While everyone may want tomatoes, they may not want to same variety or amount. You may want a small batch of heirlooms for your salad while someone making pasta sauce wants

For a community garden, like an open source project, the specific objectives of the participants do not have to be identical for the garden to florish. In fact, the very diversity of intent is what makes the garden successful. A single gardener may only plant watermelons and corn, but the community group will likely have a complete crop.

But the analogy does not end with the gardeners. A community garden is also strengthened by the cooks and dinners who enjoy the food because they are the audience.

This post was designed to plant some seeds of understanding.  I know it does not get to the meat of the vision, strategy or execution for open source, that will come in future posts.  Specifically, I’m planning to discuss how OpenStack and Crowbar measure up as they near their respective second and first anniversaries.

OpenStack vs CloudStack? It’s about open innovation.

Posted on by
3

Yesterday, I got a short drive in a “Kick-Ass” Fisker Karma.  As someone who converted a car to electric, it was a great treat to see the amazing quality, polish and sophistication of the Karma.  Especially since, six years ago, I had to build my own EV.  Today there is a diversity of choices ranging from the Nissan, GM, Tesla, Aptera, Fisker and others.  Yet even with all these choices, EVs are far from the main stream.

How does that relate to Cloud *aaS?  It’s all about innovation cycles and adoption.

Cloud platforms (really, IaaS software) have transformed from DIY to vibrant projects in the last few years; however, we still don’t know what the finished products will look like - we are only at the beginning of the innovation cycle.

With yesterday’s Citrix’s “CloudStack joins Apache” announcement painted as a shot against OpenStack, it is tempting to get pulled into a polarized view of the right or wrong way to implement cloud software (NetworkWorld,  JavaWorld, CloudPundit).  I think that feature by feature comparisons miss the real dynamics of the cloud market.

The question is not about features today, it is about forward velocity tomorrow.  There are important areas needing technology development (network, storage, etc) in the cloud infrastructure space.  

So the real story, expressed eloquently by Thierry Carrez, is about open collaboration and the resulting pace of innovation.  That means that I consider all the cloud platforms in the market to be immature because we are still learning the scope of the “cloud” opportunity.

The critical question is how the various cloud projects will maintain growth and adopt innovation.  Like the current generation of EVs, we must both prove value in production and demonstrate our ability to learn and improve.

The Citrix decision to submit CloudStack to the Apache Foundation underscores this point: success of projects is about attracting collaboration and innovation.

From the perspective of building innovation and attracting developers, the tension between OpenStack and CloudStack is very real.

Cloudera Manager Barclamp posted! (part of updated Dell | Cloudera Apache Hadoop Solution)

Posted on by
Reply

My team at Dell has been driving to transparency and openness around Crowbar plus our OpenStack and Hadoop powered solutions.  Specifically, our work for our coming release is maintained in the open on the Dell CloudEdge Github site.  You can see (and participate in!) our development and validation work in advance of our official release.

I’m pleased to note that our Cloudera Manager barclamp has been posted to Github!

This barclamp supersedes  the Hadoop barclamp in the next release of the Dell | Cloudera Apache Hadoop solution.  You can built it in Crowbar using the “cloudera-os-build”  branch for Crowbar.  Do not fear!  The Hadoop barclamp still exists (hadoop-os-build branch).

Both the new and original Hadoop barclamp use the Cloudera Hadoop distribution (aka CDH); however, the new barclamp is able to leverage Cloudera‘s latest management capabilities.  For the Dell solution, Cloudera Manager has always been part of the offering.  The primary difference is that we are improving the level of integration.  I promise to post more about the features of the solution as we get closer to release.

Hadoop Crowbar released to open source! (plus AN HOUR of videos!)

Posted on by
3

I’m proud to announce that my team at Dell has open sourced our Apache Hadoop barclamps!  This release follows our Dell | Cloudera Hadoop Solution open source commitment from Hadoop World earlier this month.

As part of this release, we’ve created nearly AN HOUR of video content showing the Hadoop Barclamps in action, installing Crowbar (on CentOS), building Crowbar ISOs in the cloud and specialized developer focused builds.

If you want to talk to the Crowbar team.  We’re attending events in Boston 11/29, Seattle 11/30, and Austin 12/8.

Here are links to the videos:

More Hadoop perspectives from Dell:  Joseph George on what it means and  Barton George‘s backgrounder about barclamps.

Dell is open sourcing Crowbar Apache Hadoop barclamps!

Posted on by
6

I’m very excited to announce that my team at Dell will be open sourcing our Apache Hadoop Crowbar barclamps by the end of the month.

This release raises the bar on open Hadoop deployments by making them faster, scalable, more integrated and repeatable.

These barclamps were developed in conjunction with our licensed Dell | Cloudera Solution. The licensed solution is for customers seeking large scale and professionally supported big data solutions. The purpose of the open barclamps (which pull the open source parts from the Cloudera distro) is to help you get started with Hadoop and reduce your learning curve. Our team invested significant testing effort in ensuring that these barclamps work smoothly because they are the foundational layer of our for-pay Hadoop solution.

Included in the Hadoop barclamp suite are Hadoop Map Reduce, Hive, Pig, ZooKeeper and Sqoop running on RHEL 5.7. These barclamps cover the core parts of the Hadoop suite. Like other Crowbar deployments (see OpenStack), the barclamps automatically discover the service configurations and interoperate. One of our team members (call him Scott Jensen) said it very simply “I can deploy a fully an integrated Hadoop cluster in a few hours. That friggin’ rocks!” I just can’t put it more eloquently than that!

I’ll post again when we flip the “open” bit and invite our community to dig in and help us continue to set the standards on open Hadoop deployments.

For more perspectives on this release, check out posts by Barton George (just for devs), Joseph George (About Hadoop) and Aurelian Dumitru

Barton posted these two videos of me talking about the release too:

Hadoop & Crowbar:

Dev’s Only Short:

Dell Crowbar Project: Open Source Cloud Deployer expands into the Community

Posted on by
2

Note: Cross posted on Dell Tech Center Blogs.

Background: Crowbar is an open source cloud deployment framework originally developed by Dell to support our OpenStack and Hadoop powered solutions.  Recently, it’s scope has increased to include a DevOps operations model and other deployments for additional cloud applications.

It’s only been a matter of months since we open sourced the Dell Crowbar Project at OSCON in June 2011; however, the progress and response to the project has been over whelming.  Crowbar is transforming into a community tool that is hardware, operating system, and application agnostic.  With that in mind, it’s time for me to provide a recap of Crowbar for those just learning about the project.

Crowbar started out simply as an installer for the “Dell OpenStack™-Powered Cloud Solution” with the objective of deploying a cloud from unboxed servers to a completely functioning system in under four hours.  That meant doing all the BIOS, RAID, Operations services (DNS, NTP, DHCP, etc.), networking, O/S installs and system configuration required creating a complete cloud infrastructure.  It was a big job, but one that we’d been piecing together on earlier cloud installation projects.  A key part of the project involved collaborating with Opscode Chef Server on the many system configuration tasks.  Ultimately, we met and exceeded the target with a complete OpenStack install in less than two hours.

In the process of delivering Crowbar as an installer, we realized that Chef, and tools like it, were part of a larger cloud movement known as DevOps.

The DevOps approach to deployment builds up systems in a layered model rather than using packaged images.  This layered model means that parts of the system are relatively independent and highly flexible.  Users can choose which components of the system they want to deploy and where to place those components.  For example, Crowbar deploys Nagios by default, but users can disable that component in favor of their own monitoring system.  It also allows for new components to identify that Nagios is available and automatically register themselves as clients and setup application specific profiles.  In this way, Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.

We believe that operations that embrace layered deployments are essential for success because they allow our customers to respond to the accelerating pace of change.  We call this model for cloud data centers “CloudOps.”

Based on the flexibility of Crowbar, our team decided to use it as the deployment model for our Apache™ Hadoop™ project (“Dell | Apache Hadoop Solution”).  While a good fit, adding Hadoop required expanding Crowbar in several critical ways.

  1. We had to make major changes in our installation and build processes to accommodate multi-operating system support (RHEL 5.6 and Ubuntu 10.10 as of Oct 2011).
  2. We introduced a modularization concept that we call “barclamps” that package individual layers of the deployment infrastructure.  These barclamps reach from the lowest system levels (IPMI, BIOS, and RAID) to the highest (OpenStack and Hadoop).

Barclamps are a very significant architecture pattern for Crowbar:

  1. They allow other applications to plug into the framework and leverage other barclamps in the solution.  For example, VMware created a Cloud Foundry barclamp and Dream Host has created a Ceph barclamp.  Both barclamps are examples of applications that can leverage Crowbar for a repeatable and predictable cloud deployment.
  2. They are independent modules with their own life cycle.  Each one has its own code repository and can be imported into a live system after initial deployment.  This allows customers to expand and manage their system after initial deployment.
  3. They have many components such as Chef Cookbooks, custom UI for configuration, dependency graphs, and even localization support.
  4. They offer services that other barclamps can consume.  The Network barclamp delivers many essential services for bootstrapping clouds including IP allocation, NIC teaming, and node VLAN configuration.
  5. They can provide extensible logic to evaluate a system and make deployment recommendations.  So far, no barclamps have implemented more than the most basic proposals; however, they have the potential for much richer analysis.

Making these changes was a substantial investment by Dell, but it greatly expands the community’s ability to participate in Crowbar development.  We believe these changes were essential to our team’s core values of open and collaborative development.

Most recently, our team moved Crowbar development into the open.  This change was reflected in our work on OpenStack Diablo (+ Keystone and Dashboard) with contributions by Opscode and Rackspace Cloud Builders.  Rather than work internally and push updates at milestones, we are now coding directly from the Crowbar repositories on Github.  It is important to note that for licensing reasons, Dell has not open sourced the optional BIOS and RAID barclamps.  This level of openness better positions us to collaborate with the crowbar community.

For a young project, we’re very proud of the progress that we’ve made with Crowbar.  We are starting a new chapter that brings new challenges such as expanding community involvement, roadmap transparency, and growing Dell support capabilities.  You will also begin to see optional barclamps that interact with proprietary and licensed hardware and software.  All of these changes are part of growing Crowbar in framework that can support a vibrant and rich ecosystem.

We are doing everything we can to make it easy to become part of the Crowbar community.  Please join our mailing list, download the open source code or ISO, create a barclamp, and make your voice heard.  Since Dell is funding the core development on this project, contacting your Dell salesperson and telling them how much you appreciate our efforts goes a long way too.