Apply, Rinse, Repeat! How do I get that DevOps conditioner out of my hair?

I’ve been trying to explain the pain Tao of physical ops in a way that’s accessible to people without scale ops experience.   It comes down to a yin-yang of two elements: exploding complexity and iterative learning.

Science = Explosions!Exploding complexity is pretty easy to grasp when we stack up the number of control elements inside a single server (OS RAID, 2 SSD cache levels, 20 disk JBOD, and UEFI oh dear), the networks that server is connected to, the multi-layer applications installed on the servers, and the change rate of those applications.  Multiply that times 100s of servers and we’ve got a problem of unbounded scope even before I throw in SDN overlays.

But that’s not the real challenge!  The bigger problem is that it’s impossible to design for all those parameters in advance.

When my team started doing scale installs 5 years ago, we assumed we could ship a preconfigured system.  After a year of trying, we accepted the reality that it’s impossible to plan out a scale deployment; instead, we had to embrace a change tolerant approach that I’ve started calling “Apply, Rinse, Repeat.”

Using Crowbar to embrace the in-field nature of design, we discovered a recurring pattern of installs: we always performed at least three full cycle installs to get to ready state during every deployment.

  1. The first cycle was completely generic to provide a working baseline and validate the physical environment.
  2. The second cycle attempted to integrate to the operational environment and helped identify gaps and needed changes.
  3. The third cycle could usually interconnect with the environment and generally exposed new requirements in the external environment
  4. The subsequent cycles represented additional tuning, patches or redesigns that could only be realized after load was applied to the system in situ.

Every time we tried to shortcut the Apply-Rinse-Repeat cycle, it actually made the total installation longer!  Ultimately, we accepted that the only defense was to focus on reducing A-R-R cycle time so that we could spend more time learning before the next cycle started.

Three critical ingredients for digital age relationships. [Collaborate Series 8/8]

Translation: Are you ready to apply these lessons?

This post is the final post in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

End of LineDuring this blog series, we’ve explored how important culture is in the work place.  The high tech areas are especially sensitive because they disproportionately embrace the millennial culture which often causes conflicts.

Our world has changed, driven by technology, new thinking, and new methodologies yet we may be using 20th century management techniques on 21st century customers and workers. There is an old business axiom that states, “If you can’t measure it, you can’t manage it.”  Yet how much of our process, interaction, successes, and failures never wind up on a spreadsheet, yet impact it?

Customers don’t leave bad companies; they leave companies that miss the mark when it comes to customer engagement. To better serve our customers we need to understand and adapt to the psychology of a new customer … one who has been trained to work as a Digital Native.

What would that look like? Tech people who interact with patience, collaboration, deep knowledge, and an openness to input, adapting to a customer’s needs in real-time. Wouldn’t that create a relationship that is second to none and unbreakable? Wouldn’t that be a leg up on the competition?

By understanding that new business culture has been influenced by the gaming experience, we have a deeper understanding of what is important to our customer base. And like a video game, if you cling to hierarchy, you lose. If you get caught up in linear time management, you lose. If you cling to bottlenecks and tradition you lose.

Three key takeaways: speed, adaptation, and collaboration

Those three words sum up today’s business environment. By now, you should not be surprised that those drivers are skills honed in video games.

We’ve explored the radically different ways that Digital Natives approach business opportunities. As the emerging leaders of the technological world, we must shift our operations to be more open, collaborative, iterative, and experience based.

Rob challenges you to get involved in his and other collaborative open source projects. Brad challenges you to try new leadership styles that engage with the Cloud Generation. Together, we challenge our entire industry to embrace a new paradigm that redefines how we interact and innovate. We may as well embrace it because it is the paradigm that we’ve already trained the rising generation or workers to intuitively understand.

What’s next?

Brad and Rob collaborated on this series with the idea of extending the concepts beyond a discussion of the “digital divide” and really looking at how culture impacts business leadership.  Lately, we’ve witnessed that the digital divide is not about your birthday alone.  We’ve seen that age alone does not drive the all cultural differences we’ve described here.  Our next posts will reflect the foundations for different ways that we’ve seen people respond to each other with a focus on answering “can digital age workers deliver?”

Like the conclusion?  Reading the rest of the series! 1: Intro > 2: ToC > 3: Video Reality > 4: Authority > 5: On The Game Training > 6: Win by Failing > 7: Go Digital Native > 8: Three Takeaways

 

Cloud Culture: Becoming L33T – Five ways to “go digital native” [Collaborative Series 7/8]

Subtitle: Five keys to earn Digital Natives’ trust

This post is #7 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

WARNING: These are not universal rules! These are two cultures. What gets high scores for Digital Natives is likely to get you sacked with Digital Immigrants.

How do Digital Natives do business?

You've gotta deal with itYou don’t sell! You collaborate with them to help solve their problems. They’ll discredit everything say if you “go all marketing on them” and try to “sell them.”

Here are five ways that you can build a two-way collaborative relationship instead of one-way selling. These tips aren’t speculation: Brad has proven these ideas work in real-world business situations.

Interested in Digital Native Culture?  We recommend reading (more books):

1) Share, don’t tell.

Remember the cultural response in Rob’s presentation discussed in the introduction to this paper? The shift took place because Rob wanted to share his expertise instead of selling the awesomeness of his employeer. This is what changed the dynamic.

In a selling situation, the sales pitch doesn’t address our client’s needs. It addresses what we want to tell them and what we think they need. It is a one-way conversation. And if someone has a choice between saying “yes” or “no” in a sales meeting, a client can always have the choice to say “no.”

Sharing draws our customers in so we can hear their problems and solve them. We can also get a barometer on what they know versus what they need. When Rob is presenting to a customer, he’s qualifying the customer too. Solutions are not one size fits all and Digital Natives respect you more for admitting this to them.

Digital Native business is about going for a long-term solution-driven approach instead of just positioning a product. If you’ve collaborated with customers and they agree you’ve got a solution for them then it’s much easier to close the sale. And over the long term, it’s a more lucrative way to do business.

2) Eliminate bottlenecks.

Ten years ago, IT departments were the bottleneck to getting products into the market. If customers resisted, it could take years to get them to like something new. Today, Apple introduces new products every six month with a massive adoption rate because Digital Natives don’t wait for permission from an authority.

The IT buyer has made that sales cycle much more dynamic because our new buyers are Digital Natives. Where Digital Immigrants stayed entrenched in a process or technology, Digital Natives are more willing to try something unproven. Amazon’s EC2 public cloud presented a huge challenge to the authority of IT departments because developers were simply bypassing internal controls. Digital Natives have been trained to look for out-of-the-box solutions to problems.

Time-to-market has become the critical measure for success.

We now have IT end-user buyers who adopt and move faster through the decision process than ever before! We interfere with their decision process if we still treating new buyers as if they can’t keep up and we have to educate them.

Today’s Digital Workers are smart, self-starters who more than understand technology; they live it. Their intuitive nature toward technology and the capacity to use it without much effort has become a cultural skill set. Also they can look up, absorb, and comprehend products without much effort. They did their homework before we walked in the door.

Digital Natives are impatient. They want to skip over what they know and get to real purpose and collaboration. You add bottlenecks when you force them back into a traditional decision process that avoids risk; instead, they are looking to business partners to help them iterate and accelerate.

 How did this apply to the Crowbar project?

Crowbar addresses a generation’s impatience to be up and running in record time. But there is more to it than that: we engage with customers differently too. Our open source collaboration and design flexibility mean that we can dialog with customers and partners to figure out the real wants and needs in record time.

3) Let go of linear.

Digital Natives do not want to be walked through detailed linear presentations. They do want the information but leave out the hand holding. The best strategy is to prepare to be a well-trained digital commando—plan a direction, be confident, be ready to respond, and be willing to admit knowledge gaps. It’s a strategy without a strategy.

Ask questions at the beginning of a meeting—this becomes a knowledge base “smell test.” Listening to what our clients know and don’t know gets us to the heart and purpose of why we are there. Take notes. Stay open to curve balls, tough questions, and—dare we say it—the client telling us we are off base. You should not be surprised at how much they know.

For open source projects at Dell (Rob’s Employeer), customers have often downloaded and installed the product before they have talked to the sales team. Rob has had to stop being surprised when they are better informed about our offerings than our well trained internal teams. Digital Natives love collecting information and getting started independently. This completely violates the normal linear sales process; instead, customers enter more engaged and ready if you can be flexible enough to meet them where they already are.

4) Be attentively interactive.

No one likes to sit in one meeting after another. Why are meetings boring? Meetings should be engaging and collaborative; unfortunately, most meetings are simply one-way presentations or status updates. When Digital Natives interrupt a presentation, it may mean they are not getting what they want but it also means they are paying attention.

Aren’t instant messaging, texting, and tweeting attention-stealing distractions?

Don’t confuse IMing, texting, emailing, and tweeting as lack of attention or engagement.

Digital Natives use these “back channels” to speed up knowledge sharing while eliminating the face-to-face meeting inertia of centralized communication.

Of course, sometimes we do check out and stop paying attention.

Time and attention are valuable commodities!

With all the distractions and multi-tasking for speed and connectivity, giving someone undivided attention is about respect, and paying attention is not passive! When we ask questions, it shows that we’re engaged and paying attention. When we compile all the answers from those questions, our intention leads us to solutions. Solving our client’s problems is about getting to the heart of the matter and becomes the driving force behind every action and solution.

Don’t be afraid to stray from the agenda—our attention is the agenda.

5) Stay open to happy accidents.

In Brad’s book, Liquid Leadership, the chapter titled “Have Laptop. Will Travel” points out how Digital Natives have been trained in virtualized work habits because they are more effective.

Our customers are looking for innovative solutions to their problems and may find them in places that we do not expect. It is our job to stay awake and open to solution serendipity. Let’s take this statement out of our vocabulary: “That’s not how we do it.” Let’s try a new approach: “That isn’t traditionally how we would do it, but let us see if it could improve things.”

McDonald’s uses numbers for their combo meals to make sure ordering is predictable and takes no more than 30 seconds. It sounds simple, but changes come from listening to customers’ habits. We need to stop judging and start adapting. Imagine a company that adapts to the needs of its customers?

Sales guru Jeffery Gitomer pays $100 in cash to any one of his employees who makes a mistake. This mistake is analyzed to figure out if it is worthy of application or to be discarded. He doesn’t pay $100 if they make the same mistake twice. Mistakes are where we can discover breakthrough ideas, products, and methods.

Making these kinds of leaps requires that we first let go of rigid rules and opinions and make it OK to make a few mistakes … as long as we look at them through a lens of possibility. Digital Natives have spent 10,000 hours playing learning to make mistakes, take risks, and reach mastery.

Keep Reading! Next post is Three Takeaways (previous Win by Failing)

 

 

Cloud Culture: Level up – You win the game by failing successfully [Collaborative Series 6/8]

Translation: Learn by playing, fail fast, and embrace risk.

This post is #6 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

It's good to failDigital Natives have been trained to learn the rules of the game by just leaping in and trying. They seek out mentors, learn the politics at each level, and fail as many times as possible in order to learn how NOT to do something. Think about it this way: You gain more experience when you try and fail quickly then carefully planning every step of your journey. As long as you are willing to make adjustments to your plans, experience always trumps prediction.

Just like in life and business, games no longer come with an instruction manual.

In Wii Sports, users learn the basic in-game and figure out the subtlety of the game as they level up. Tom Bissel, in Extra Lives: Why Video Games Matter, explains that the in-game learning model is core to the evolution of video games. Game design involves interactive learning through the game experience; consequently, we’ve trained Digital Natives that success comes from overcoming failure.

Early failure is the expected process for mastery.

You don’t believe that games lead to better decision making in real life? In a January 2010 article, WIRED magazine reported that observations of the new generation of football players showed they had adapted tactics learned in Madden NFL to the field. It is not just the number of virtual downs played; these players have gained a strategic field-level perspective on the game that was before limited only to coaches. Their experience playing video games has shattered the on-field hierarchy.

For your amusement…Here is a video about L33T versus N00B culture From College Humor “L33Ts don’t date N00Bs.”  Youtu.be/JVfVqfIN8_c

Digital Natives embrace iterations and risk as a normal part of the life.

Risk is also a trait we see in entrepreneurial startups. Changing the way we did things before requires you to push the boundaries, try something new, and consistently discard what doesn’t work. In Lean Startup Lessons Learned, Eric Ries built his entire business model around the try-learn-adjust process. He’s shown that iterations don’t just work, they consistently out innovate the competition.

The entire reason Dell grew from a dorm to a multinational company is due to this type of fast-paced, customer-driven interactive learning. You are either creating something revolutionary or you will be quickly phased out of the Information Age. No one stays at the top just because he or she is cash rich anymore. Today’s Information Age company needs to be willing to reinvent itself consistently … and systematically.

Why do you think larger corporations that embrace entrepreneurship within their walls seem to survive through the worst of times and prosper like crazy during the good times?

Gamer have learned that Risk that has purpose will earn you rewards.

To improve flow, we must view OpenStack community as a Software Factory

This post was sparked by a conversation at OpenStack Atlanta between OpenStack Foundation board members Todd Moore (IBM) and Rob Hirschfeld (Dell/Community).  We share a background in industrial and software process and felt that sharing lean manufacturing translates directly to helping face OpenStack challenges.

While OpenStack has done an amazing job of growing contributors, scale has caused our code flow processes to be bottlenecked at the review stage.  This blocks flow throughout the entire system and presents a significant risk to both stability and feature addition.  Flow failures can ultimately lead to vendor forking.

Fundamentally, Todd and I felt that OpenStack needs to address system flows to build an integrated product.  The post expands on the “hidden influencers” issue and adds an additional challenge because improving flow requires that the community influences better understands the need to optimize work inter-project in a more systematic way.

Let’s start by visualizing the “OpenStack Factory”

Factory Floor

Factory Floor from Alpha Industries Wikipedia page

Imagine all of OpenStack’s 1000s of developers working together in a single giant start-up warehouse.  Each project in its own floor area with appropriate fooz tables, break areas and coffee bars.  It’s easy to visualize clusters of intent developers talking around tables or coding in dark corners while PTLs and TC members dash between groups coordinating work.

Expand the visualization so that we can actually see the code flowing between teams as little colored boxes.  Giving project has a unique color allows us to quickly see dependencies between teams.  Some features are piled up waiting for review inside teams while others are waiting on pallets between projects waiting on needed cross features have not completed.  At release time, we’d be able to see PTLs sorting through stacks of completed boxes to pick which ones were ready to ship.

Watching a factory floor from above is a humbling experience and a key feature of systems thinking enlightenment in both The Phoenix Project and The Goal.  It’s very easy to be caught up in a single project (local optimization) and miss the broader system implications of local choices.

There is a large body of work about Lean Process for Manufacturing

You’ve already visualized OpenStack code creation as a manufacturing floor: it’s a small step to accept that we can use the same proven processes for software and physical manufacturing.

As features move between teams (work centers), it becomes obvious that we’ve created a very highly interlocked sequence of component steps needed to deliver product; unfortunately, we have minimal coordination between the owners of the work centers.  If a feature is needs a critical resource (think programmer) to progress then we rely on the resource to allocate time to the work.  Since that person’s manager may not agree to the priority, we have a conflict between system flow and individual optimization.

That conflict destroys flow in the system.

The number #1 lesson from lean manufacturing is that putting individual optimization over system optimization reduces throughput.  Since our product and people managers are often competitors, we need to work doubly hard to address system concerns.  Worse yet our inventory of work in process and the interdependencies between projects is harder to discern.  Unlike the manufacturing floor, our developers and project leads cannot look down upon it and see the physical work as it progresses from station to station in one single holistic view.  The bottlenecks that throttle the OpenStack workflow are harder to see but we can find them, as can be demonstrated later in this post.

Until we can engage the resource owners in balancing system flow, OpenStack’s throughput will decline as we add resources.  This same principle is at play in the famous aphorism: adding developers makes a late project later.

Is there a solution?

There are lessons from Lean Manufacturing that can be applied

  1. Make quality a priority (expand tests from function to integration)
  2. Ensure integration from station to station (prioritize working together over features)
  3. Make sure that owners of work are coordinating (expose hidden influencers)
  4. Find and mange from the bottleneck (classic Lean says find the bottleneck and improve that)
  5. Create and monitor a system view
  6. Have everyone value finished product, not workstation output

Added Subscript: I highly recommend reading Daniel Berrange’s email about this.

Cloud Culture: Online Games, the real job training for Digital Natives [Collaborative Series 5/8]

Translation: Why do Digital Natives value collaboration over authority?

Kids Today

This post is #5 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

Before we start, we already know that some of you are cynical about what we are suggesting—Video games? Are you serious? But we’re not talking about Ms. Pac-Man. We are talking about deeply complex, rich storytelling, and task-driven games that rely on multiple missions, worldwide player communities, working together on a singular mission.

Leaders in the Cloud Generation not just know this environment, they excel in it.

The next generation of technology decision makers is made up of self-selected masters of the games. They enjoy the flow of learning and solving problems; however, they don’t expect to solve them alone or a single way. Today’s games are not about getting blocks to fall into lines; they are complex and nuanced. Winning is not about reflexes and reaction times; winning is about being adaptive and resourceful.

In these environments, it can look like chaos. Digital workspaces and processes are not random; they are leveraging new-generation skills. In the book Different, Youngme Moon explains how innovations looks crazy when they are first revealed. How is the work getting done? What is the goal here? These are called “results only work environments,” and studies have shown they increase productivity significantly.

Digital Natives reject top-down hierarchy.

These college educated self-starters are not rebels; they just understand that success is about process and dealing with complexity. They don’t need someone to spoon feed them instructions.

Studies at MIT and The London School of Economics have revealed that when high-end results are needed, giving people self-direction, the ability to master complex tasks, and the ability to serve a larger mission outside of themselves will garnish groundbreaking results.

Gaming does not create mind-addled Mountain Dew-addicted unhygienic drone workers. Digital Natives raised on video games are smart, computer savvy, educated, and, believe it or not, resourceful independent thinkers.

Thomas Edison said:

“I didn’t fail 3,000 times. I found 3,000 ways how not to create a light bulb.”

Being comfortable with making mistakes thousands of times ’til mastery sounds counter-intuitive until you realize that is how some of the greatest breakthroughs in science and physics were discovered.  Thomas Edison made 3,000 failed iterations in creating the light bulb.

Level up: You win the game by failing successfully.

Translation: Learn by playing, fail fast, and embrace risk.

Digital Natives have been trained to learn the rules of the game by just leaping in and trying. They seek out mentors, learn the politics at each level, and fail as many times as possible in order to learn how NOT to do something. Think about it this way: You gain more experience when you try and fail quickly then carefully planning every step of your journey. As long as you are willing to make adjustments to your plans, experience always trumps prediction.Just like in life and business, games no longer come with an instruction manual.

In Wii Sports, users learn the basic in-game and figure out the subtlety of the game as they level up. Tom Bissel, in Extra Lives: Why Video Games Matter, explains that the in-game learning model is core to the evolution of video games. Game design involves interactive learning through the game experience; consequently, we’ve trained Digital Natives that success comes from overcoming failure.

Keep Reading! Win by Failing (previous Authority)

VMware Integrated OpenStack (VIO) is smart move, it’s like using a Volvo to tow your ski boat

I’m impressed with VMware’s VIO (beta) play and believe it will have a meaningful positive impact in the OpenStack ecosystem.  In the short-term, it paradoxically both helps enterprises stay on VMware and accelerates adoption of OpenStack.  The long term benefit to VMware is less clear.

From VWVortex

Sure, you can use a Volvo to tow a boat

Why do I think it’s good tactics?  Let’s explore an analogy….

My kids think owning a boat will be super fun with images of ski parties and lazy days drifting at anchor with PG13 umbrella drinks; however, I’ve got concerns about maintenance, cost and how much we’d really use it.  The problem is not the boat: it’s all of the stuff that goes along with ownership.  In addition to the boat, I’d need a trailer, a new car to pull the boat and driveway upgrades for parking.  Looking at that, the boat’s the easiest part of the story.

The smart move for me is to rent a boat and trailer for a few months to test my kids interest.  In that case, I’m going to be towing the boat using my Volvo instead of going “all in” and buying that new Ferd 15000 (you know you want it).  As a compromise, I’ll install a hitch in my trusty sedan and use it gently to tow the boat.  It’s not ideal and causes extra wear to the transmission but it’s a very low risk way to explore the boat owning life style.

Enterprise IT already has the Volvo (VMware vCenter) and likely sees calls for OpenStack as the illusion of cool ski parties without regard for the realities of owning the boat.  Pulling the boat for a while (using OpenStack on VMware) makes a lot of sense to these users.  If the boat gets used then they will buy the truck and accessories (move off VMware).  Until then, their still learning about the open source boating life style.

Putting open source concerns aside.  This helps VMware lead the OpenStack play for enterprises but may ultimately backfire if they have not setup their long game to keep the customers.