DevOps for Non-Profits?! The Miracle Foundation does IRL Puppies v. Cattle

In what’s become an annual tradition, I’m taking a post to think about the intersection of Cloud and Non-profits using my better-half’s employer, The Miracle Foundation, as my inspiration (and to help support their Mothers’ Day campaign).

TMF girl with puppyTheir deceptively simple sounding mission is to nurture children – they’ve just added some minor wrinkles like the children are orphans, in economically challenged areas generally tucked away in remote areas of India half way around the world from their Austin HQ.  That does nothing to dampen their tenacious drive to ensure that these children have the benefits of food, health care, housing, education and, most critically, nurturing caregivers.

How does that relate to the Puppies & Cattle analogy?

Like any scalable operation, they need to create highly repeatable processes to deliver their service.    The Miracle Foundation service, environments where house mothers nurture children, is by its very nature a “puppy” since each child must be treated uniquely; however, everything leading up to the point of delivery must be “cattle-like” to they can scale the care they give.  For example, unique lesson plan is good while a unique chart of accounts is not.

Last year, I talked about how the Miracle Foundation was using quantitative measures to evaluate quality of care.  They’ve used these metrics very effectively in their operations to identify places where they must standardize (like accounting practices, health care regimens and dietary requirements) and high touch places where they cannot (selecting and promoting homes out of incubation).  Exactly like cloud deployments, success means finding places where variation creates complexity (cattle) and ones where it increases value (puppies).

I’ve been impressed to see how the Miracle Foundation identified the need for standardized house-mother training curriculum as part of this analysis.  Their years of experience across a breath of orphanages has shown that giving clear guidance and setting standards for the people in direct contact with the children nets tremendous results; however, just making sure this training is delivered means building up a lot of other process and standardization.

If you think your job of building DevOps scripts and practice is hard then you need to step away from the keyboard for a while.  This organization, and other non-profits like it, are taking on similar challenges with real people across distances that are more than just a few router hops from your desktop.  I’m inspired by how they take on these challenges and fascinated at how much commonality there is between my work and theirs.

If you’re interested in their mission, please visit them for more details.

Mayflies and Dinosaurs (extending Puppies and Cattle)

Dont Be FragileJosh McKenty and I were discussing the common misconception of the “Puppies and Cattle” analogy. His position is not anti-puppy! He believes puppies are sometimes unavoidable and should be isolated into portable containers (VMs) so they can be shuffled around seamlessly. His more provocative point is that we want our underlying infrastructure to be cattle so it remains highly elastic and flexible. More cattle means a more resilient system. To me, this is a fundamental CloudOps design objective.

We realized that the perfect cloud infrastructure would structurally discourage the creation of puppies.

Imagine a cloud in which servers were automatically decommissioned after a week of use. In a sort of anti-SLA, any VM running for more than 168 hours would be (gracefully) terminated. This would force a constant churn of resources within the infrastructure that enables true cattle-like management. This cloud would be able to very gracefully rebalance load and handle disruptive management operations because the workloads are designed for the churn.

We called these servers mayflies due to their limited life span.

While this approach requires a high degree of automation, the most successful cloud operators I have met are effectively building workloads with this requirement. If we require application workloads to be elastic and fault-resilient then we have a much higher degree of flexibility with the underlying infrastructure. I’ve seen this in practice with several OpenStack clouds: operators with helped applications deploy using automation were able to decommission “old” clouds much more gracefully. They effectively turned their entire cloud into a cow. Sadly, the ones without that investment puppified™ the ops infrastructure and created a much more brittle environment.

The opposite of a mayfly is the dinosaur: a server that is so brittle and locked that the slightest disturbance wipes out everything it touches.

Dinosaurs are puppies grown into a T-Rex with rows of massive razor sharp teeth and tiny manicured hands. These are systems that are so unique and historical that there’s no way to recreate them if there’s a failure. The original maintainers exit happy hour was celebrated by people who were laid-off two CEOs ago. The impact of dinosaurs goes beyond their operational risk; they are typically impossible to extend or maintain and, consequently, ossify other server around them. This type of server drains elasticity from your ops team.

Puppies do not grow up to become dogs, they become dinosaurs.

It’s a classic lean adage to do hard things more frequently. Perhaps it’s time to start creating mayflies in your ops infrastructure.