Continuous Release combats disruptions of “Free Fall” development

Since I posted the “Free Fall” development post, I’ve been thinking a bit about the pros and cons of this type of off-release development.

The OpenStack Swift project does not do free fall because they are on a constant “ship ready” state for the project and only loosely flow the broader OpenStack release track.  My team at Dell also has minimal free fall development because we have a more frequent release clock and choose to have the team focus together through dev/integrate/harden cycles as much as possible.

From a Lean/Agile/CI perspective, I would work to avoid hidden development where possible.  New features are introduced by split test (they are in the code, but not active for most users) so that the all changes in incremental.  That means that refactoring, rearchitecture and new capabilities appear less disruptively.  While it may this approach appears to take more effort in the short term; my experience is that it accelerates delivery because we are less likely to over develop code.

Unfortunately, free fall development has the opposite effect.  Having code that appears in big blocks is contrary to best practices in my opinion.  Further, it rewards groups that work asynchronously and

While I think that OpenStack benefits from free fall work, I think that it is ultimately counter-productive.

Ballistic Release Cycles: Tracking the Trajectory of OpenStack Milestones

I’ve been watching a pattern emerge on the semiannual OpenStack release cycles for a while now. There is a hidden but crucial development phase that accelerates projects faster than many observers realize. In fact, I believe that substantial work is happening outside of the “normal” design cycle during what I call “free fall” development.

Understanding when the cool, innovative stuff happens is essential to getting (and giving) the most from OpenStack.

The published release cycle looms like a 6 stage ballistic trajectory. Launching at the design summit, the release features change and progress the most in the first 3 milestones. At the apogee of the release, maximum velocity is reached just as we start having to decide which features are complete enough to include in the release. Since many are not ready, we have to jettison (really, defer) partial work to ensure that we can land the release on schedule.

I think of the period where we lose potential features as free fall because thing can go in any direction. The release literally reverses course: instead of expanding, it is contracting. This process is very healthy for OpenStack. It favors code stability and “long” hardening times. For operators, this means that the code stops changing early enough that we have more time to test and operationalize the release.

But what happens to the jettisoned work? In free fall, objects in motion stay in motion. The code does not just disappear! It continues on its original upward trajectory.

The developers who invested time in the code do not simply take a 3 month sabbatical, nor do they stop their work and start testing the code that was kept. No, after the short in/out sorting pause, the free fall work continues onward with rockets blasting. The challenge is that it is now getting outside of the orbit of the release plan and beyond the radar of many people who are tracking the release.

The consequence of this ongoing development is that developers (and the features they are working on) show up at the summit with 3 extra months of work completed. It also means that OpenStack starts each release cycle with a bucket of operationally ready code. Wow, that’s a huge advantage for the project in terms of delivered work, feature velocity and innovation. Even better, it means that the design summit can focus on practical discussions of real prototypes and functional features.

Unfortunately, this free fall work has hidden costs:

  • It is relatively hidden because it is outside of the normal release cycle.
  • It makes true design discussions less productive because the implemented code is more likely to make the next release cycle
  • Integration for the work is postponed because it continues before branching
  • Teams that are busy hardening a core feature can be left out of work on the next iteration of the same feature
  • Forking can make it hard to capture bugs caught during hardening

I think OpenStack greatly benefits from free fall development; consequently, I think we need to acknowledge and embrace it to reduce its costs. A more explicit mid-release design synchronization when or before we fork may help make this hidden work more transparent.

Agile takes discipline: having a strategy means saying “no” more than saying “yes”

With the Crowbar release behind us, it’s time for my team at Dell to do some Capital “P” Planning. Planning for us includes both tactical (next release) and strategic (the releases beyond the one after next), but each type of planning looks very different. I’m going to call it “roadmapping” because planning means something specific and tactical in Agile.

I love roadmapping but I’m a pain to roadmap with because I’m a ruthless prioritizer.

When I sit down for roadmapping, I always do it from a 1 to N list without ties. That means that when marketing asks for a new feature (double the foo on the bar!) we put it on the list relative to other work that needs to get done. If you add something at the top then something else will fall off the bottom. Effectively, we’re using the list to say no to a lot of great ideas. This is essential because “the great is the enemy of the good (Voltaire).” It’s hard, but that’s the cold reality of delivering product.

The most important part of strategy is figuring out what to push down to make room for the precious few yes items.

Successful roadmapping is negotiating the splitting of big ideas into smaller ones. Decomposition is a circular process because one compromise may require another, but one change may force a cascading assumption fault. If you get too emotionally committed to one feature or subset then you’re going to slow down the process. It’s vital to approach roadmapping in free fall.

As always, my advice is to not mix meeting objectives. If you need more strategy then you’ve got to make time for it.

Interested in more…stay tuned for Agile Tao: balancing tactics & strategy

Black Hat Feedback Essential For Cloud Success

It’s been too long since my “Agile in the Cloud” blog focused on the agile process side of the equation.  Since my theme for 2011 will be “Doing is Doing” (meaning, attending meetings and building powerpoint is NOT doing); it’s vital to embrace processes that are action and delivery focused.  Personally, I tend towards the Lean side (Poppendieck & Ries) of the Agile process spectrum.

Let’s start with something so obvious that most teams don’t bother with it:

The secret to improving your performance is to regularly work to improve your performance.

There is no further magic, no secret sauce, no superhero team member, and no silver process bullet that substitutes for working together to improve.  A team that commits to improving will ultimately transform into a high performing team.  Hmmm…that seems like a desirable result.  So why is it so uncommon?

Unfortunately, many teams skimp on improvement because:

  1. All that talking and listening and improving takes time and attention.
  2. Post mortem or “lessons learned” meetings that never make any difference because they happen after the work is done.  (well, duh).
  3. No one has a methodology that makes the meetings productive.
  4. Lists with more than 3 items on them have too many items to be productive for improvement activities because of reader’s limited attention spans.  Doh!

So #1 and #2 just take some resolve to resolve.  My (atypical) team at Dell spends 1 to 3+ hours every biweekly sprint talking about process improvement (aka retrospective).  That’s nearly half of our planning day and a pretty significant investment; however, it’s the one part of the meeting that no one is willing to shorten.  It just that damn productive, important, and enlightening.

If you’re committed to taking the time to improve then finding a methodology (#3) is a piece of cake by comparison.  I highly recommend De Bono’s Six Thinking Hats the framework for retrospectives.  Michael Klobe taught me this process and I’ve used it successfully on many teams.  Here are some guidelines to retrospectives (we call them “Hats Meetings”) using De Bono’s hats:

  1. Write everything down.  All our planning meetings are documented.  This is ESSENTIAL because we review our past notes.
  2. Start with “Yellow” hats from all team members.  Yellow hats are accolades, complements, thanks, generally positive news, and anything that the team can celebrate.  Yellow hats get the meeting opened on a positive note.
  3. Go until your team runs out of yellows.
  4. Write down and number the “Black” hats.   Black hats are things that impacted team performance.  Ideally, they are concrete and specific examples of a problem and a cost.  Black hats are NOT solutions or suggestions hidden as complaints.  If you’re frustrated that people are late to meeting then a good hat would be “people being late to meeting is costing us 10 minutes of productivity because we have to restart the meeting 3 times.”  Hats like “The vendor messed up” are not as helpful as “The vendor’s late delivery of the futz dinger caused us to work over the weekend.”  Good black hats are a learned skill and are essential to success.
  5. Do NOT record every detail when writing down a black hat.  You just need to capture the ending understanding of the person who brought up the item (not of the team).  I’ve seen many hats that would have been a huge HR problem if we wrote down the starting idea but the ending hat was perfectly ok.
  6. Go until your team runs out of blacks.  This may take a long time if you’re just starting out!
  7. Come up with a “Green” hat for each Black hat.  Green hats are ways that the team or individuals will solve the problem addressed by the Black hat.  Solutions should be practical and immediate.  The team must discuss the solution (free fall) – it cannot be imposed by one member.  In many cases, solutions will go in steps: “everyone will try to show up on time” would be a good hat for my example above.  If that does not work, then future hats may involve “the late person pays $1/minute late.”  The point is that the team agrees on how they will improve.
  8. Don’t worry about White (data), Blue (new ideas), or Red (frustration, issues beyond your control) hats for now.  They are not critical for retrospective meetings.  Sometimes our team has a black hat that is really red.  If you’re arguing over a hat, then it may be red.  Stop fighting and move on.
  9. At your next meeting, review your past Green hats.  This is where accountability and improvement happen.

Please don’t expect that these meetings will be easy – they are messy and potentially emotional.  Since we’re talking about teams of people, we have to expose and resolve issues.  Investing the time upfront allows for continuous improvement and prevents building up larger issues.

Hats provides a durable structure for getting issues on the table in a productive way.

Free Fall

Or elimination of pre-discussion bias

Learning Free Fall was the hardest of all the Agile comeuppance meted out to me.  Of course, it was not called free fall by the team teaching me about it.  Instead, it was the sinking feeling that I was always arguing the wrong side of every issue.  I was supposed to be the hot architect but my ideas kept getting shot down by the team.  They weren’t wrong per se, but they were not as good as the team could figure out during a discussion.

Meetings went like this:

Rob: “Hey guys, I’ve got this great idea that all of our SQL queries should leverage the zeitgeist feature because it will improve performance 25%.”

Team: “Um, yeah but zeitgeist requires us to perform green joins on the red data.”

Rob: “That’s a good point.  I thought that most of the data was yellow.”

Team: “That’s true.  What about meme queries?  Their like zeitgeist but work for yellow and red data.”

Rob: “I really thought that zeitgeist was the way to go.  I think we can make that work if we limit the red data.”

Team: “Zeitgeist was a good suggestion, but we’ll go with meme.”

While my ideas helped the team, I could not get over the fact they did not go with my original idea.  Even worse, I would keep fighting for my idea after the team had obviously moved past it.  It slowed down decisions, made meeting more contentious, and caused our optometrist bills to skyrocket from excessive eye rolling.

All that changed when I learned how to trust my team and free fall during meetings.  Free falling (queue Tom Petty) is decoupling the problem from your idea of how it should be solved.  If you bring the problem to your team without a solution then you’ll get everyone’s brain working on ideas for the solution.  When I stopped being emotionally committed to the solution, I was much more ready to listen to other ideas.  More importantly, it was obvious to everyone else that I was more willing to listen so they were more willing to contribute and less interested in arguing.

Let me repeat that:

When I started the discussion without having picked a solution, it was obvious to everyone that their ideas would be heard.  We had more ideas, better discussion, and better results.  Generally, the team result was much better than my original idea and (bonus!) the team was always more willing to implement them after a free fall discussion.

Free fall takes some practice.  Find a buddy to help you stay on course by kicking you under the table when you’re reaching for your parachute.  Trust me – it’s like EPO for your team (and it’s legal!).