Mayflies and Dinosaurs (extending Puppies and Cattle)

Dont Be FragileJosh McKenty and I were discussing the common misconception of the “Puppies and Cattle” analogy. His position is not anti-puppy! He believes puppies are sometimes unavoidable and should be isolated into portable containers (VMs) so they can be shuffled around seamlessly. His more provocative point is that we want our underlying infrastructure to be cattle so it remains highly elastic and flexible. More cattle means a more resilient system. To me, this is a fundamental CloudOps design objective.

We realized that the perfect cloud infrastructure would structurally discourage the creation of puppies.

Imagine a cloud in which servers were automatically decommissioned after a week of use. In a sort of anti-SLA, any VM running for more than 168 hours would be (gracefully) terminated. This would force a constant churn of resources within the infrastructure that enables true cattle-like management. This cloud would be able to very gracefully rebalance load and handle disruptive management operations because the workloads are designed for the churn.

We called these servers mayflies due to their limited life span.

While this approach requires a high degree of automation, the most successful cloud operators I have met are effectively building workloads with this requirement. If we require application workloads to be elastic and fault-resilient then we have a much higher degree of flexibility with the underlying infrastructure. I’ve seen this in practice with several OpenStack clouds: operators with helped applications deploy using automation were able to decommission “old” clouds much more gracefully. They effectively turned their entire cloud into a cow. Sadly, the ones without that investment puppified™ the ops infrastructure and created a much more brittle environment.

The opposite of a mayfly is the dinosaur: a server that is so brittle and locked that the slightest disturbance wipes out everything it touches.

Dinosaurs are puppies grown into a T-Rex with rows of massive razor sharp teeth and tiny manicured hands. These are systems that are so unique and historical that there’s no way to recreate them if there’s a failure. The original maintainers exit happy hour was celebrated by people who were laid-off two CEOs ago. The impact of dinosaurs goes beyond their operational risk; they are typically impossible to extend or maintain and, consequently, ossify other server around them. This type of server drains elasticity from your ops team.

Puppies do not grow up to become dogs, they become dinosaurs.

It’s a classic lean adage to do hard things more frequently. Perhaps it’s time to start creating mayflies in your ops infrastructure.

Doing is Doing – my 10 open source principles

2013-07-14_17-28-21_468Open source projects’ greatest asset is their culture and FOSS practitioners need to deliberately build and expand it. To me, culture is not soft or vague.  Culture is something specific and actionable that we need to define and hold people accountable for.

I have simple principles that guide me in working in open source.   At their root, they are all simply “focus on the shared work.”

I usually sum them up as “Doing is Doing.”  While that’s an excellent test to see if you’re making the right choices, I suspect many will not find that tautology sufficiently actionable.

The 10 principles I try to model in open source leadership:

  1. Leadership includes service: connecting, education, documentation and testing
  2. Promotion is a two-edged sword – leaders needs to take extra steps to limit self-promotion or we miss hearing the community voice.
  3. Collaboration must be modeled by the leaders with other leaders.
  4. Vision must be articulated, but shared in a way that leaves room for new ideas and tactical changes.
  5. Announcements should be based on available capability not intention. In open source, there is less need for promises and forward-looking statements because your actions are transparent.
  6. Activity (starting from code and beyond) should be visible (Github = social coding) – it’s the essence of collaboration.
  7. Testing is essential because it allows other people to join with reduced risk.
  8. Docs are essential because it reduces friction for users to adopt.
  9. Upstreaming (unlike Forking) is a team sport so be prepared for some give-and-take.
  10. It’s not just about code, open source is about solving shared problems together.  When we focus on the shared goals (“the doing”) then the collaboration comes naturally.

Kicking off discussion about OpenStack Core

What's in?I’ve been leading an effort with Alan Clark to define “what is OpenStack core” for the Foundation Board.  Now that I am sitting here at OSCON and celebrating OpenStack’s third birthday, I think it’s a great time to bring the general community into the discussion.

There is significant history behind this topic.  According to Foundation governance, the Technical Committee (TC) defines which incubated projects are integrated and the Board of Directors (I am one) determines which of the integrated projects are labeled as core.

When it comes to the core label, the stakes are high.

Defining core is a convoluted topic.  To make it digestible, I’m breaking it down into multiple blog posts over a series of weeks:

  1. Why do we care about core?
  2. Decomposing the problem (“the spider”)
  3. Insights from decomposition (healthy tensions in OpenStack)
  4. 12 Positions: A Common Framework (I recommend ready the list of 10 instead)
  5. Community Feedback at OSCON 
  6. “What is Core” the visualization
  7. Where I think this is going, OpenStack’s Test Driven Core
  8. Core Positions Refined: the 10 positions behind the core visualization (above).
  9. Videos (most >90 mins).  The online meetups are easier to follow.
    1. 9/5 OpenStack Core Meetup in San Antonio
    2. 9/22 OpenStack Core Meetup in NYC
    3. online: 10/16 Online Meetup (daytime) and 10/22 Online Meetup (evening)
  10. Thinking about how to Implement OpenStack Core Definition

Too much reading?  At OSCON, Rafael Knuth shot a video of me talking about “what is core.”

my lean & open source reading list – recommendations welcome!

Cube Seat

I think it’s worth pulling together a list of essential books that I think should be required reading for people on Lean & open source teams (like mine):

  • Basis for the team values that we practice: The Five Dysfunctions of a Team: A Leadership Fable Patrick Lencioni (amazon)
  • This is a foundational classic for team building:  Peopleware: Productive Projects and Teams (Second Edition) Tom DeMarco (amazon)
  • This novel is good primer for lean and devops The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win Gene Kim and George Spafford & Kevin Behr (amazon)
  • Business Focus on Lean: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses Eric Ries (amazon)
  • Foundational (and easy) reading about Lean: The Goal: A Process of Ongoing Improvement Eliyahu M. Goldratt (amazon)
  • One of my favorites on Lean / Agile: Implementing Lean Software Development: From Concept to Cash Mary Poppendieck (amazon)
  • Should be required reading for open source (as close to “Open Source for Dummies as you can get): The Cathedral & the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary Eric S. Raymond (amazon)
  • Culture Change Liquid Leadership: From Woodstock to Wikipedia–Multigenerational Management Ideas That Are Changing the Way We Run Things Brad Szollose (amazon)
  • More Team Building – this one is INTERACTIVE! http://www.strengthsfinder.com/home.aspx

There are some notable additions, but I think this is enough for now.  I’m always looking for recommendations!  Please post your favorites in the comments!

OpenStack leaders learning by humility, doing and being good partners

With the next OpenStack Board meeting on Thursday (5/30/13 agenda) and Mark McLoughlin’s notes crossing my desk, I was reminded of still open discussion topics around OpenStack leadership.  Reminder: except for executive sessions, OpenStack Board Meetings are open (check agenda for details).

2013-03-11_20-01-50_458

Many of the people and companies involved in OpenStack are new to open source projects. Before OpenStack, I had no direct experience building a community like we’ve built together with OpenStack or I’ve been leading with Crowbar. There is no Collaborative Open Source Communities for Dummies book (I looked).

I am not holding myself, OpenStack or Crowbar up as shining examples of open source perfection. Just the opposite, we’ve had to learn the hard way about what works and what fails. I attribute our successes to humility to accept feedback and willingness to ask for help.

But being successful in the small (like during OpenStack Cactus) is different than where we are heading.  In the small, everyone was an open source enthusiast and eager collaborator.  In the large, we should be asking the question “how will we teach people to join and build an open source community?”

The answer is that collaboration must be modeled by the OpenStack leadership.

At the Summit, I was talking with fellow board director Sean Roberts (Yahoo!) and I think he made this point very simply:

“Being in open source is a partnership. If you don’t bring something to the partnership then you’re a user not a partner. We love users but we need to acknowledge the difference.” (Sean Roberts, OpenStack Director)

OpenStack will succeed by building a large base of users; consequently, we need our leaders to be partners in the community.

7 takeaways from DevOps Days Austin

Block Tables

I spent Tuesday and Wednesday at DevOpsDays Austin and continue to be impressed with the enthusiasm and collaborative nature of the DOD events.  We also managed to have a very robust and engaged twitter backchannel thanks to an impressive pace set by Gene Kim!

I’ve still got a 5+ post backlog from the OpenStack summit, but wanted to do a quick post while it’s top of mind.

My takeaways from DevOpsDays Austin:

  1. DevOpsDays spends a lot of time talking about culture.  I’m a huge believer on the importance of culture as the foundation for the type of fundamental changes that we’re making in the IT industry; however, it’s also a sign that we’re still in the minority if we have to talk about culture evangelism.
  2. Process and DevOps are tightly coupled.  It’s very clear that Lean/Agile/Kanban are essential for DevOps success (nice job by Dominica DeGrandis).  No one even suggested DevOps+Waterfall as a joke (but Patrick Debois had a picture of a xeroxed butt in his preso which is pretty close).
  3. Still need more Devs people to show up!  My feeling is that we’ve got a lot of operators who are engaging with developers and fewer developers who are engaging with operators (the “opsdev” people).
  4. Chef Omnibus installer is very compelling.  This approach addresses issues with packaging that were created because we did not have configuration management.  Now that we have good tooling we separate the concerns between bits, configuration, services and dependencies.  This is one thing to watch and something I expect to see in Crowbar.
  5. The old mantra still holds: If something is hard, do it more often.
  6. Eli Goldratt’s The Goal is alive again thanks to Gene Kims’s smart new novel, The Phoenix project, about DevOps and IT  (I highly recommend both, start with Kim).
  7. Not DevOps, but 3D printing is awesome.  This is clearly a game changing technology; however, it takes some effort to get right.  Dell brought a Solidoodle 3D printer to the event to try and print OpenStack & Crowbar logos (watch for this in the future).

I’d be interested in hearing what other people found interesting!  Please comment here and let me know.

From orphans to open source, data matters

TMF ChildrenMy wife’s day job helps Indian orphans through the Miracle Foundation here in Austin. On the surface, our jobs are very different; however, there is lately more and more intersection in both form and substance. It was not always like that, initially the Miracle Foundation primary engagement had been an emotional appeal: “look, these orphans are sad, they need you. Did we mention that they are orphans?”

Joking aside, there are plenty of kind people who want to help children; however, there are a lots of worthy causes with equally strong appeal. The question is how do you pick which one? Donors/Contributors want one that is both emotionally appealing and effective.

While radically different in human impact, both of raising orphans and building open source rely heavily on personal engagement and passion for success. Just like non-profits, there are many open source projects that want you to invest your time in installing and contributing to their most worthy technology.

About 18 months ago, the Miracle Foundation pivoted their strategy from tending individual children towards cultivating whole orphanages (the “NEST program”, video below). They started tracking things like how much milk and fruit each child ate and if they had been vaccinated. They connected observable data like hemoglobin levels of children to their ability to pay attention in school. They were even aware of additional days girls spent in school just because they got monthly hygiene products.

NEST Spider Graph

Used with Permission, The Miracle Foundation

With this new program, the Miracle Foundation can tell you exactly how much benefit each child will receive from each dollar. These are real results derived from collecting real data, and the results are powerful.

The children the Miracle Foundation nurtures are going from subsistence to flourishing. This is not happening because people care more about these children than before. It is happening because someone is keeping the data and making sure that the support they give gets the results they want. This in turn helps donors (become one) feel confident that their emotional response is delivering tangible improvements. Both are essential to TMF’s mission.

Open source projects have a similar gestalt.

People and companies contributing time and resources to a project want to both believe in the technology and see tangible metrics to validate adoption. Open source transparency makes it easier to find active projects and people are engaged contributors, but it can be harder to determine if the project is having broader impact.

For OpenStack, these tangible metrics began to surface in the last few days. Before the summit, Stephano Muffulli, community manager, launched the OpenStack Activity Board to show commit and quality data for the project. Last Monday, Tim Bell & Ryan Lane presented the results of the first user survey which showed how and what users are adopting for OpenStack.

If you like seeing this type of data driven behavior then vote with your keyboard and become part of an active open source project. For non-profits like the Miracle Foundation, voting is even easier – you just need a credit card to join in their Mothers’ Day campaign. Your mom may not understand anything you add to open source, but she can understand when you help orphans.

Continue reading

As OpenStack enters rapids with Grizzly, watch for strong currents, hidden rocks & eddies.

White Water

Play Boating From Wikipedia

I enjoy kayaking white water rapids – they are exhilarating and demanding. The water accelerates around obstacles and shows its power. You cannot simply ride the current; you must navigate your way around obstacles, stay clear of eddies that pull you back and watch for hidden rocks. The secret to success is to read the current and make small adjustments as you are carried along – resistance is futile.

After the summit, I see the OpenStack with the Grizzly release like water entering the rapids. The quality and capability of the code base continues to improve while the number of players with offerings in the ecosystem is also increasing rapidly. Until now, there was plenty of room to play together; however, as scope, activity and velocity increase there will more inter-vendor interactions.

As a member of the OpenStack board, I have tremendous enthusiasm for what the OpenStack community has accomplished. There have been some really positive accounts of the summit including CSC “OpenStack gains maturity…“, Silicon Angle “OpenStack has reached a Flash Point”, Randy Bias’ “OpenStack is THE Stack”, Wayne Walls “Hallway Track” and much more on the Planet OpenStack aggregator.

In fact, we’ve created such a love fest for OpenStack that I fear we are drinking our own kool aide.

I have a responsibility to be transparent and honest about challenges facing the us because it’s the Foundation’s job to guide us forward. My positions result from many conversations that I had throughout the week of the Summit. They are also the result of my first hand experiences along with my 14 years of cloud experience.

Over the next posts, I’ll explore a number of these topics with the goal of helping navigate a path through the potential turbulence. The simple fact is the OpenStack is growing quickly and that creates challenges:

  1. A growing number of new developers are joining. Since our work surface area is expanding, it’s both easier than ever to participate and harder to navigate where to begin. We need to get ahead of the design cycles.
  2. A growing number of non-devs are participating and bringing important contributions and experience. We must include them in the OpenStack meritocracy because they speak for the quality and usability of the project.
  3. A growing number of companies (many “name brands”) who are still trying to figure out how to participate and collaborate in open source projects. Lack of experience increases the risk of divergence (forking) and market confusion.
  4. A growing number of products based on OpenStack also increases forking risk as OpenStack contributors feel compelled to differentiate.
  5. A growing number of core components (compute+block+network+…) that are required to have base functionality.
  6. A growing number of incubated projects that continue to stress innovation and pace of change that challenges the very question of “what is OpenStack?”
  7. A growing number of deployed sites offering OpenStack clouds but the community lacks a way to verify (or really discuss) compatibility between the sites.

This list is a cause for celebration not a cause for alarm – every item is a challenges based on our success. The community and Foundation are already working to address the risks.

While some of us enjoy the chaos and excitement of rapids, other can take comfort from the fact that they are always followed by calm waters. Don’t worry – we’ll navigate through this together.

“Stack Shop” cover of Macklemore’s Thrift Shop

Sometimes a meme glitters too strongly for me to resist getting pulled in… that happened to great effect that just before the OpenStack Havana summit. When my code-addled mind kept swapping “poppin’ tags” for “OpenStack” on the radio edit, I stopped fighting and rewrote the Thrift Shop lyrics for OpenStack (see below the split).

With a lot of help from summit attendees (many of them are OpenStack celebrities, CEOs, VPs and members OpenStack Foundation board), I was able to create a freaking awesome cover of Macklemore’s second hand confection (NSFW).

Frankly, I don’t know everyone in the video (what, what?)!

But here’s a list of those that I do know.  I’m happy to update so the victims actors get credit.  Singers (in order):

Rob Hirschfeld (me) & Monty Taylor, Peter Poulliot, Judd Maltin, Forrest Norrod, Josh Kleinpeter, Tristan Goode, Dan Bode, Jay Pipes, Prabhakar Gopalan, Peter Chadwick, Simon Andersen, Vish Ishaya, Wayne Walls, Alex Freedland, Niki Acosta, Ops Track Monday Session 1, Ben Cherian, Eric Windisch, Brandon Draeger, Joseph B George,  Mark Collier, Joseph Heck, Tim Bell,  Chris Kemp, Kyle McDonald & Joshua McKenty,

Continue reading

What foo is “contribution” to open source? Mik Kersten & Tasktop @ SXSW

Nested

How do we really know who influences most in a software project?  We can easily track code commits, but there are more bits to the project than the commits.

I had the good fortune to attend Mik Kersten’s Code Graph presentation at SXSW last week. Mik started the Eclipse Mylyn project and went on to found Tasktop. Both are built on the very intriguing concepts that software development production (aka work) is organized around tasks.

His premise is that organizing around tasks provides a more manageable and actionable view of a project than a more traditional application life-cycle management (ALM) approaches.  I’m a sucker for any presentation about lean development process that includes references to both DevOps and industrial engineering (I have an MS in IE), but Mik surprised me by taking his code graph concept to a whole ‘nutha level.

The software value chain is much deeper than just the people who write code. Mik’s approach included managers, testers and operators in the interaction graphs for his projects.

By including all of the ALM artifacts in the analysis, you get a much richer picture of the influencers for a project.

For example, the development manager may never show up as a code committer; however, they are hugely influential in which work gets prioritized. If your graph includes who is touching the work assignments and stories then the manager’s influence jumps out immediately. That knowledge would completely change how and who you may interact with a team. It effectively brings a shadow contributor into the light.

The same is true for QA members who are running tests and opening defects and operators who are building deployment scripts. Ideally, it should include users who exercise different parts of the applications capabilities.

Mik’s graphs clearly showed the influence impact of managers because they touched all of the story cards for the project.  The people who own the story cards are the most potent influencers in a project, yet they are invisible in code repositories!

I would love to see an impact graph for a software project that equally reflected the wide range of contributions that people make to its life-cycle.  This type of information helps rebalance the power in a project.

Industrial engineering legend W.E. Demming‘s advice is to look at production as a system.  Finding ways to show everyone’s contributions is an important step towards bringing lean processes fully into software manufacturing.