Note: This is part 1 of a 3 part series. I have been working on this series for nearly six months in an attempt to make this subtle but extremely expensive problem understandable. Rather than continue to polish the posts, I will post series for your enjoyment. I hope that it is enlightening, humorous or (ideally) both. Comments are welcome!
I’ve been struggling to explain a subtle process fail that occurs every day at my company (Dell) and also at every company I’ve ever worked with or for. I call this demon “Process Interlock” and it is the invisible bane of projects big and small. It manifests by forcing well-meaning product managers and engineering directors to make trade-offs that they know are wrong because of schedule commitments. It means that product quality consistently drops to the bottom of the list in favor of getting in that one promised feature. It shows up when customers get products late because of prospect who decided not to buy demanded a feature a year ago. These are the symptoms of the process interlock dilemma.
Process Interlock occurs when another team depends on your team for a future feature.
That sounds pretty innocuous right? It makes sense that other teams, customers and partners should be able to ask you about your roadmap and then build your delivery schedule into their plans. That is the perfectly logical request that happens inside my group every single day. Unfortunately, that exact commitment is what creates the problem because it locks your team’s velocity into the future and eliminates agility.
Note: I was reading chapter 11 in Eric Ries’ Lean Startup as was surprised to find him making very similar arguments but from a different perspective.
To hopefully help explain, I’m inventing a hypothetical project from the car division of the G.Mordler company. GM plans to add time travel as an option for their 2016 product line. They believe that there is a big market in minivan’s that can solve the proverbial “are we there yet problem” by simply skipping over the boring part of the trip. The trans-dimensional mommy mobile (or Trans Ma’am) will be part of a refresh of their 2014 model. The addition of a time circuit and power generator developed two internal divisions, Alpha and Omega, support a critical marketing event for the company so timing is important.
Let’s examine four outcomes of how these two divisions turn their assumed schedules into rigidly locked conundrum.
Scenario 0: Ideal Case.
Alpha makes the fusion power supply and Omega is making the time circuits. Based on experimental data, Omega’s design calls for 3.14 Gigawatts to operate their time capacitor; however, Alpha’s available design is limited to 0.73 Gigawatts. Alpha expects to reach 3.5 Gigawatts in 9 months when their supplier releases an updated nitrogen cooled super conductor. Based on that commitment, Omega has enough information to make an informed decision about their timeline. Since Alpha commits to deliver in 12 months (9 for the new part + 3 for development), Omega expects to deliver a working time circuit in 20 months (12 for the supply + 8 for development). In this example, there are 3 levels of Process Interlock: Alpha interlocks with the supplier and then Omega interlocks with Alpha. From a PERT schedule perspective, the world is now under control! It’s a brand new day and the birds are singing…
Scenario 1: Meet Schedule w/ Added Cost
Unfortunately, we now have a highly interlocked schedule. In the best case scenario (the one where we meet the schedule), Alpha has just signed up to meet an aggressive delivery timeframe. They have to put heavy pressure on the supplier to deliver their part which causes the supplier to increase the price for the cooler component. When their product manager identifies available alternative markets (such as power generating pet waste incineration), they are not able to purse the opportunities because they cannot risk the schedule impact of redirecting engineers. Meanwhile, Omega understands that a critical part is missing for 12 months and decides to reduce staffing while waiting for the needed part. In the process, they lose a key engineer who could have optimized the manufacturing process to half the production defect rate. Overall, the project meets schedule but at added cost, reduced quality and missed opportunities. This happened because the interlocks eliminated flexibility in the schedule for upstream and downstream participants. GM meets the launch window for the Trans Ma’am but high costs for the upgrade limit sales.
Scenario 2: Meet Schedule w/ Lost Features
A more likely “on schedule” alternative is that Alpha’s supplier cuts some corners to meet the aggressive deadline; consequently, power generation for Alpha is not reliable. This issue is not revealed by load testing in Alpha’s labs or short time travel testing by Omega. Instead, the faulty generators fail in integration field testing accidentally sending a DOT test driver home during rush hour traffic. Fixing the problem requires a redesign of the power plant. The new design does not fit into space allowed by the Trans Ma’am design team causing the entire program, while delivered “on time,” to be considered a failure and not shipped. GM misses the launch window for the Trans Ma’am.
In the most likely scenario the project is late. The schedule for Alpha slips because supplier requires an extra three months to meet the Alpha’s specs. In a common turn of fate, the supplier’s specs would be sufficient for Alpha to proceed; however, Alpha’s risk manager bumped up the cooling requirements by 20% in order to ensure they had wiggle room in their own design. Because of the supplier contract requiring delivery per spec, the supplier could not ship a workable but contractually unacceptable product. Since the part is delayed, Alpha has to slip the schedule to Omega. Compounding the problem, Alpha’s manager is optimistic that it will work out and does not alert Omega until 2 weeks before the deadline. Omega, who has been testing their circuits using liquid sodium cooled nuclear fission power plants, attempts to make up the schedule delay by imposing 20 hour Mountain Dew fueled work days. The aggressive schedule results in quality issues for the time circuits so that they can only be used during Mountain-time rebroadcasts of Seinfeld. After an unsuccessful bid to purchase the Denver cable TV station KDEV, GM misses the launch window for the Trans Ma’am.
I realize these examples are complicated, but I hope they humorously illuminate the problem.
In part 2, I’ll show an alternate approach for GM that addresses the process interlock.
Of course, for this example, the entire project plan is a moot point since we’re talking about time machines! I’m offering two likely endings for the scenarios above:
The Pragmatists’ Ending: Once the project is finally complete, the manager simply drives the car back to the beginning of the project. Over white Russian martinis and sushi, her future self explains how the painful delivery schedule cost her the best years of her life causing her to quit. Her replacement cannot maintain funding for the project so it is eventually scraped by G.Mordler six months before the working pieces can be assembled.
The Realists’ Ending: Once the project is finally complete, the manager simply drives the car back to the beginning of the project. Over lemonade vodka tonics and tapas, her future self provides a USB stick with the critical design data needed to complete the project on time and budget. When she examines the data, the resulting time paradox creates a rift in the Einstein-Jacob space-time fabric thus ending the universe.
Pingback: How Good beats Great and avoids Process Interlock failure « Rob Hirschfeld's Blog
Pingback: Four alternatives to Process Interlock « Rob Hirschfeld's Blog
Pingback: Quick turn OpenStack Essex on Crowbar (BOOM, now we’re at v1.4!) « Rob Hirschfeld's Blog
Pingback: Nextcast #14 Transcription on OpenStack & Crowbar > “we can’t hand out trophies to everyone” | Rob Hirschfeld