Continuous Release combats disruptions of “Free Fall” development

Since I posted the “Free Fall” development post, I’ve been thinking a bit about the pros and cons of this type of off-release development.

The OpenStack Swift project does not do free fall because they are on a constant “ship ready” state for the project and only loosely flow the broader OpenStack release track.  My team at Dell also has minimal free fall development because we have a more frequent release clock and choose to have the team focus together through dev/integrate/harden cycles as much as possible.

From a Lean/Agile/CI perspective, I would work to avoid hidden development where possible.  New features are introduced by split test (they are in the code, but not active for most users) so that the all changes in incremental.  That means that refactoring, rearchitecture and new capabilities appear less disruptively.  While it may this approach appears to take more effort in the short term; my experience is that it accelerates delivery because we are less likely to over develop code.

Unfortunately, free fall development has the opposite effect.  Having code that appears in big blocks is contrary to best practices in my opinion.  Further, it rewards groups that work asynchronously and

While I think that OpenStack benefits from free fall work, I think that it is ultimately counter-productive.

The Go-Fasterer OpenStack Cloud Strategy

Dell’s OpenStack strategy (besides being interesting by itself) brings together Agile and Lean approaches and serves as a good illustration of the difference between the two approaches.

Before I can start the illustration, I need to explain the strategy clearly enough that the discussion makes sense.   Of course, my group is selling these systems so the strategy starts a sales pitch.  Bear with me, this is a long post and I promise we’ll get to the process parts as fast as possible.

Dell’s OpenStack strategy is to enter the market with the smallest possible working cloud infrastructure practical.  We have focused maniacally on eliminating all barriers and delays for customers’ evaluation processes.  Our targets are early adopters who want to invest in a real, hands-on OpenStack evaluation and understand they will have to work to figure out OpenStack.   White gloves, silver spoons and expensive licensed applications are not included in this offering.

We are delivering a cloud foundation kit: 7u hardware setup (6 nodes+switch), white paper, installer, and a dollop of consulting services.  It is a very small foot print system with very little integration.  The most notable deliverable is our target of going from boxes to working cloud in less than 4 hours (I was calling this “nuts to soup before lunch” but marketing didn’t bite).

Enough background?  Let’s talk about business process!

From this point on, our product offering is just an example.   You should imagine your product or service in these descriptions.  You should think about the internal reconfiguration required needed to bring your product or service to market in the way I am describing.

There are two critical elements in the go-fasterer strategy:

  1. a very limited “lean” product and
  2. a very fast “agile” installation process.

The offering challenges the de facto definition of solutions as being complete packages bursting with features, prescriptive processes, licensed companion products and armies of consultants.  While Dell will eventually have a solution that meets (or exceeds) these criteria; our team did not think we should wait until we had all those components before we begin engaging customers.

Our first offering is not for everyone by design.  It is highly targeted to early adopters who have specific needs (desire to move quickly) that outweigh all other feature requirements.  They are willing to invest in a less complete product because to core alone solves an important problem.

The concept of stripping back your product to the very core is the essence of Lean process.  Along this line of thinking, maintaining ship readiness is the primary mantra – if you can’t sell your product then your entire company’s existence is at risk.  I like the way the Poppendieck ‘s describe it:  you should consider product features as perishable inventory.  If we were selling fruit salad and you had bananas and apples but no cherries then it makes sense to sell apple/banana medley while you work on the cherries.

Whittling back a product to the truly smallest possible feature set is very threatening and difficult.  It forces teams to take risks and guesses that leave you with a product that many customers will reject.  Let me repeat that: you’re objective is to create a product that many customers will reject.  You must do this because it:

  1. gets into the market much faster for some customers (earning $ is wonderfully clarifying)
  2. learn immediately what’s missing (fewer future guesses)
  3. learn immediately what’s important to customers (less risk)
  4. builds credibility that you are delivering something (you’re building relationships)

Ironically, while lean approaches exist to reduce risk and guesswork; they will feel very risky and like gambling to organizations used to traditional processes.   This is not surprising because our objective is to go faster so initially we will be uncomfortable that we have enough information to make decisions.

The best cure for lack of information is not more analysis!  The cure is interacting with customers.

Lean says that you need product if you want to interact meaningfully with customers.  This is because customers (even those who are not buying right away) will take you more seriously if you’ve got a product.  Talking about products that you are going to release is like talking about the person you wanted to take to prom but never asked.

To achieve product early, you need to find the true minimum product set.  This is not the smallest comfortable set.  It is the set that is so small, so uncomfortable, so stripped down that it seems to barely do anything at all.

In our case, we considered it sufficient if the current OpenStack release could be reliably and quickly installed on Dell hardware.  We believe there are early adopter customers who want to evaluate OpenStack right away and their primary concern starting their pilot and marketing towards eventually deployment.

Mixing Agile into Lean is needed to make the “skinny down” discipline practical and repeatable.

Agile brings in a few critical disciplines to enable Lean:

  1. Prioritized roadmaps help keep teams focused on what’s needed first but don’t lose sight of longer term plans.
  2. Predictable pace of delivery allows committed interactions with customers that give timelines for fixing issues or adding capabilities.
  3. Working out of order keeps the great from being the enemy of the good so that we delay field testing while we solve imagined problems.
  4. Focus on quality / automation / repeatability reduces paying for technical debt internally and time firefighting careless defects when a product is “in the wild” with customers.
  5. Insistence on installable “ship ready” product ensures that product gets into the field whenever the right customer is found.  Note: this does not mean any customer.  Selling to the wrong customer can be deadly too, but that’s a different topic.
  6. Feedback driven iterations ensures that Lean engagements with customers are interactive and inform development.

These disciplines are important for any organization but vital when you go Lean.  To take your product early and aggressively to market, you must have confidence that you can continue to deliver after your customers get a taste of the product.

You cannot succeed with Lean if you cannot quickly evolve your initial offering.

The enabling compromise with Lean is that you will keep the train running with incremental improvements:  Lean fails if you engage customers early then disappear back into a long delivery cycle.  That means committing to an Agile product delivery cycle if you want Lean (note: the reverse not true)

I think of Lean and Agile as two sides of the same results driven coin: Lean faces towards the customer and market while Agile faces internally to engineering.

Please let me know how your team is trying to accelerate product delivery.

Note: of course, you’re also welcome to contact me if you’re interested in being an early adopter for our OpenStack foundation kit.

Very Costly Accounting and why I value ship ready code

Or be careful what you measure because you’ll get it.

One of Agile’s most effective, least understood and seldom realized disciplines is that teams should “maintain ship-readiness” of their product at all times.  Explaining the real value of this discipline in simple terms eluded me for years until I was talking to an accountant on a ski lift.

Before I talk about snow and mountain air inspired insights, let me explain what ship-ready means.  It means that you should always be ready to deliver the output from your last iteration to your most valuable customer.   For example, when my company, BEware, finished our last iteration we could have delivered it to our top customer, Fin & Key, without losing our much needed beauty sleep.   Because we maintain ship-readiness, we worked to ensure that they could upgrade, have complete documentation, and high quality without spending extra cycles on release hardening.

The version that we’d ship to Fin & Key would probably not have any new features enabled (see tippy towers & Eric Ries on split testing) , but it would have fixed defects and the incremental code for those new features baked in.  While we may decide to limit shipments to fixed times for marketing reasons, that must not keep us from always being ready to ship.

Meanwhile, back at 8,200 feet, my accountant friend was enrolled in Cost Accounting 101.  To fulfill my mission as an Agile Subversive, I suggested reading  “The Goal” by Eli Goldratt which comes out very strongly against the evils of CA.  Goldratt’s logic is simple – if you want people to sub-optimize and ignore the overall system productivity then you’d assign costs to each sub-component of your system.  The result in manufacturing is that people will always try to keep the most expensive machine 100% utilized even if that causes lots of problems elsewhere and drives up costs all over the factory.

Cost Accounting’s process of measuring on a per cost basis (instead of a system basis) causes everyone to minimize the cost at their area rather than collaborate to make the system more efficient.  I’ve worked in many environments where management tried to optimize expensive developer time by off loading documentation, quality, and support to different teams.  The result was a much less effective system where defects were fixed late (if ever), test automation was never built, documentation was incomplete, and customer field issues lingered like the smell of stale malt beverage in a frat house.

No one wanted these behaviors, yet they were endemic because the company optimized developer time instead of working product.

Agile maintains ship readiness because it becomes Engineering’s primary measurement.  Making sellable product the top priority focuses team on systems and collaboration.  It may not be as “efficient” to have a highly paid developer running tests; however, it does real economic harm if the developer continues to write untested code ahead of your ability to verify it.  Even more significantly, a developer who plays a larger part in test, documentation, and supports is much more invested in fixing problems early.

If your company wants ship product then find a metric that gets you to that goal.  For Agile teams, that metric is percent of iterations delivering ship ready product.  My condolences if your team’s top values are milestones completed, bugs fixed, or hours worked.