About Rob H

A Baltimore transplant to Austin, Rob drives an electric car, the RAVolt, and thinks about ways of building software for the clouds using Agile processes. He works for Dell and works software to enable hyperscale clouds.

To improve flow, we must view OpenStack community as a Software Factory

This post was sparked by a conversation at OpenStack Atlanta between OpenStack Foundation board members Todd Moore (IBM) and Rob Hirschfeld (Dell/Community).  We share a background in industrial and software process and felt that sharing lean manufacturing translates directly to helping face OpenStack challenges.

While OpenStack has done an amazing job of growing contributors, scale has caused our code flow processes to be bottlenecked at the review stage.  This blocks flow throughout the entire system and presents a significant risk to both stability and feature addition.  Flow failures can ultimately lead to vendor forking.

Fundamentally, Todd and I felt that OpenStack needs to address system flows to build an integrated product.  The post expands on the “hidden influencers” issue and adds an additional challenge because improving flow requires that the community influences better understands the need to optimize work inter-project in a more systematic way.

Let’s start by visualizing the “OpenStack Factory”

Factory Floor

Factory Floor from Alpha Industries Wikipedia page

Imagine all of OpenStack’s 1000s of developers working together in a single giant start-up warehouse.  Each project in its own floor area with appropriate fooz tables, break areas and coffee bars.  It’s easy to visualize clusters of intent developers talking around tables or coding in dark corners while PTLs and TC members dash between groups coordinating work.

Expand the visualization so that we can actually see the code flowing between teams as little colored boxes.  Giving project has a unique color allows us to quickly see dependencies between teams.  Some features are piled up waiting for review inside teams while others are waiting on pallets between projects waiting on needed cross features have not completed.  At release time, we’d be able to see PTLs sorting through stacks of completed boxes to pick which ones were ready to ship.

Watching a factory floor from above is a humbling experience and a key feature of systems thinking enlightenment in both The Phoenix Project and The Goal.  It’s very easy to be caught up in a single project (local optimization) and miss the broader system implications of local choices.

There is a large body of work about Lean Process for Manufacturing

You’ve already visualized OpenStack code creation as a manufacturing floor: it’s a small step to accept that we can use the same proven processes for software and physical manufacturing.

As features move between teams (work centers), it becomes obvious that we’ve created a very highly interlocked sequence of component steps needed to deliver product; unfortunately, we have minimal coordination between the owners of the work centers.  If a feature is needs a critical resource (think programmer) to progress then we rely on the resource to allocate time to the work.  Since that person’s manager may not agree to the priority, we have a conflict between system flow and individual optimization.

That conflict destroys flow in the system.

The number #1 lesson from lean manufacturing is that putting individual optimization over system optimization reduces throughput.  Since our product and people managers are often competitors, we need to work doubly hard to address system concerns.  Worse yet our inventory of work in process and the interdependencies between projects is harder to discern.  Unlike the manufacturing floor, our developers and project leads cannot look down upon it and see the physical work as it progresses from station to station in one single holistic view.  The bottlenecks that throttle the OpenStack workflow are harder to see but we can find them, as can be demonstrated later in this post.

Until we can engage the resource owners in balancing system flow, OpenStack’s throughput will decline as we add resources.  This same principle is at play in the famous aphorism: adding developers makes a late project later.

Is there a solution?

There are lessons from Lean Manufacturing that can be applied

  1. Make quality a priority (expand tests from function to integration)
  2. Ensure integration from station to station (prioritize working together over features)
  3. Make sure that owners of work are coordinating (expose hidden influencers)
  4. Find and mange from the bottleneck (classic Lean says find the bottleneck and improve that)
  5. Create and monitor a system view
  6. Have everyone value finished product, not workstation output

Added Subscript: I highly recommend reading Daniel Berrange’s email about this.

Cloud Culture: Online Games, the real job training for Digital Natives [Collaborative Series 5/8]

Translation: Why do Digital Natives value collaboration over authority?

Kids Today

This post is #5 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

Before we start, we already know that some of you are cynical about what we are suggesting—Video games? Are you serious? But we’re not talking about Ms. Pac-Man. We are talking about deeply complex, rich storytelling, and task-driven games that rely on multiple missions, worldwide player communities, working together on a singular mission.

Leaders in the Cloud Generation not just know this environment, they excel in it.

The next generation of technology decision makers is made up of self-selected masters of the games. They enjoy the flow of learning and solving problems; however, they don’t expect to solve them alone or a single way. Today’s games are not about getting blocks to fall into lines; they are complex and nuanced. Winning is not about reflexes and reaction times; winning is about being adaptive and resourceful.

In these environments, it can look like chaos. Digital workspaces and processes are not random; they are leveraging new-generation skills. In the book Different, Youngme Moon explains how innovations looks crazy when they are first revealed. How is the work getting done? What is the goal here? These are called “results only work environments,” and studies have shown they increase productivity significantly.

Digital Natives reject top-down hierarchy.

These college educated self-starters are not rebels; they just understand that success is about process and dealing with complexity. They don’t need someone to spoon feed them instructions.

Studies at MIT and The London School of Economics have revealed that when high-end results are needed, giving people self-direction, the ability to master complex tasks, and the ability to serve a larger mission outside of themselves will garnish groundbreaking results.

Gaming does not create mind-addled Mountain Dew-addicted unhygienic drone workers. Digital Natives raised on video games are smart, computer savvy, educated, and, believe it or not, resourceful independent thinkers.

Thomas Edison said:

“I didn’t fail 3,000 times. I found 3,000 ways how not to create a light bulb.”

Being comfortable with making mistakes thousands of times ’til mastery sounds counter-intuitive until you realize that is how some of the greatest breakthroughs in science and physics were discovered.  Thomas Edison made 3,000 failed iterations in creating the light bulb.

Level up: You win the game by failing successfully.

Translation: Learn by playing, fail fast, and embrace risk.

Digital Natives have been trained to learn the rules of the game by just leaping in and trying. They seek out mentors, learn the politics at each level, and fail as many times as possible in order to learn how NOT to do something. Think about it this way: You gain more experience when you try and fail quickly then carefully planning every step of your journey. As long as you are willing to make adjustments to your plans, experience always trumps prediction.Just like in life and business, games no longer come with an instruction manual.

In Wii Sports, users learn the basic in-game and figure out the subtlety of the game as they level up. Tom Bissel, in Extra Lives: Why Video Games Matter, explains that the in-game learning model is core to the evolution of video games. Game design involves interactive learning through the game experience; consequently, we’ve trained Digital Natives that success comes from overcoming failure.

VMware Integrated OpenStack (VIO) is smart move, it’s like using a Volvo to tow your ski boat

I’m impressed with VMware’s VIO (beta) play and believe it will have a meaningful positive impact in the OpenStack ecosystem.  In the short-term, it paradoxically both helps enterprises stay on VMware and accelerates adoption of OpenStack.  The long term benefit to VMware is less clear.

From VWVortex

Sure, you can use a Volvo to tow a boat

Why do I think it’s good tactics?  Let’s explore an analogy….

My kids think owning a boat will be super fun with images of ski parties and lazy days drifting at anchor with PG13 umbrella drinks; however, I’ve got concerns about maintenance, cost and how much we’d really use it.  The problem is not the boat: it’s all of the stuff that goes along with ownership.  In addition to the boat, I’d need a trailer, a new car to pull the boat and driveway upgrades for parking.  Looking at that, the boat’s the easiest part of the story.

The smart move for me is to rent a boat and trailer for a few months to test my kids interest.  In that case, I’m going to be towing the boat using my Volvo instead of going “all in” and buying that new Ferd 15000 (you know you want it).  As a compromise, I’ll install a hitch in my trusty sedan and use it gently to tow the boat.  It’s not ideal and causes extra wear to the transmission but it’s a very low risk way to explore the boat owning life style.

Enterprise IT already has the Volvo (VMware vCenter) and likely sees calls for OpenStack as the illusion of cool ski parties without regard for the realities of owning the boat.  Pulling the boat for a while (using OpenStack on VMware) makes a lot of sense to these users.  If the boat gets used then they will buy the truck and accessories (move off VMware).  Until then, their still learning about the open source boating life style.

Putting open source concerns aside.  This helps VMware lead the OpenStack play for enterprises but may ultimately backfire if they have not setup their long game to keep the customers.

Cloud Culture: No spacesuits, Authority comes from doing, not altitude [Collaborative Series 4/8]

Subtitle: Why flattening org charts boosts your credibility

This post is #4 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

Unlike other generations, Digital Natives believe that expertise comes directly from doing, not from position or education. This is not hubris; it’s a reflection both their computer experience and dramatic improvements in technology usability.

AstronautIf you follow Joel Spolsky’s blog, “Joel on Software,” you know about a term he uses when describing information architects obsessed with the abstract and not the details; Architecture Astronauts—so high up above the problem that they might as well be in space. “They’re astronauts because they are above the oxygen level, I don’t know how they’re breathing.”

For example, a Digital Native is much better positioned to fly a military attack drone than a Digital Immigrant. According to New Scientist, March 27, 2008, the military is using game controllers for drones and robots because they are “far more intuitive.” Beyond the fact that the interfaces are intuitive to them, Digital Natives have likely logged hundreds of hours flying simulated jets under trying battle conditions. Finally, they rightly expect that they can access all the operational parameters and technical notes about the plane with a Google search.

Our new workforce is ready to perform like none other in history.

Being able to perform is just the tip of the iceberg; having the right information is the more critical asset. A Digital Native knows information (and technology) is very fast moving and fluid. It also comes from all directions … after all it’s The Information Age. This is a radical paradigm shift. Harvard Researcher David Weinberger highlights in his book Too Big to Know that people are not looking up difficult technical problems in a book or even relying on their own experiences; they query their social networks and discover multiple valid solutions. The diversity of their sources is important to them, and an established hierarchy limits their visibility; inversely, they see leaders who build strict organizational hierarchies as cutting off their access to information and diversity.

Today’s thought worker is on the front lines of the technological revolution. They see all the newness, data, and interaction with a peer-to-peer network. Remember all that code on the screen in the movie The Matrix? You get the picture.

To a Digital Native, the vice presidents of most organizations are business astronauts floating too high above the world to see what’s really going on but feeling like they have perfect clarity. Who really knows the truth? Mission Control or Major Tom? This is especially true with the acceleration of business that we are experiencing. While the Astronaut in Chief is busy ordering the VPs to move the mountains out of the way, the engineers at ground control have already collaborated on a solution to leverage an existing coal mine and sell coal as a byproduct.

The business hierarchy of yesterday worked for a specific reason: workers needed to just follow rules, keep their mouth shut, and obey. Input, no matter how small, was seen as intrusive and insubordinate … and could get one fired. Henry Ford wanted an obedient worker to mass manufacture goods. The digital age requires a smarter worker because, in today’s world, we make very sophisticated stuff that does not conform to simple rules. Responsibility, troubleshooting, and decision-making has moved to the frontlines. This requires open-source style communication.

Do not confuse the Astronaut problem as a lack of respect for authority.

Digital Natives respect informational authority, not positional. For Digital Natives, authority is flexible. They have experience forming and dissolving teams to accomplish a mission. The mission leader is the one with the right knowledge and skills for the situation, not the most senior or highest scoring. In Liquid Leadership, Brad explains that Digital Natives are not expecting managers to solve team problems; they are looking to their leadership to help build, manage, and empower their teams to do it themselves.

So why not encourage more collaboration with a singular mission in mind: develop a better end product? In a world that is expanding at such mercurial speed, a great idea can come from anywhere! Even from a customer! So why not remember to include customers in the process?

Who is Leroy Jenkins?

This viral video is about a spectacular team failure from one individual (Leroy Jenkins) who goes rogue during a team massively multi-player game.  This is a Digital Natives’ version of the ant and grasshopper parable: “Don’t pull a Leroy Jenkins on us—we need to plan this out.”  Youtu.be/LkCNJRfSZBU

Think about it like this: Working as a team is like joining a quest.

If comparing work to a game scenario sounds counterintuitive then let’s reframe the situation. We may have the same destination and goals, but we are from very different backgrounds. Some of us speak different languages, have different needs and wants. Some went to MIT, some to community college. Some came through Internet startups, others through competitors. Big, little, educated, and smart. Intense and humble. Outgoing and introverted.  Diversity of perspective creates stronger teams.

This also means that leadership roles rotate according to each mission.

This is the culture of the gaming universe. Missions and quests are equivalent to workplace tasks accomplished and point to benchmarks achieved. Each member excepts to earn a place through tasks and points. This is where Digital Natives’ experience becomes advantage. They expect to advance in experience and skills. When you adapt the workplace to these expectations the Digital Natives thrive.

Leaders need to come down to earth and remove the spacesuit.

A leader at the top needs to stay connected to that information and disruption. Start by removing your helmet. Breathe the same oxygen as the rest of us and give us solutions that can be used here on planet earth.

On Gamification

Jeff Attwood, founder of the community-based FAQ site Stack Overflow, has been very articulate about using game design to influence how he builds communities around sharing knowledge. We recommend reading his post about “Building Social Software for the Anti-Social” on his blog, CodingHorror.com.

OpenStack DefCore Process Flow: Community Feedback Cycles for Core [6 points + chart]

If you’ve been following my DefCore posts, then you already know that DefCore is an OpenStack Foundation Board managed process “that sets base requirements by defining 1) capabilities, 2) code and 3) must-pass tests for all OpenStack™ products. This definition uses community resources and involvement to drive interoperability by creating the minimum standards for products labeled OpenStack™.”

In this post, I’m going to be very specific about what we think “community resources and involvement” entails.

The draft process flow chart was provided to the Board at our OSCON meeting without additional review.  It below boils down to a few key points:

  1. We are using the documents in the Gerrit review process to ensure that we work within the community processes.
  2. Going forward, we want to rely on the technical leadership to create, cluster and describe capabilities.  DefCore bootstrapped this process for Havana.  Further, Capabilities are defined by tests in Tempest so test coverage gaps (like Keystone v2) translate into Core gaps.
  3. We are investing in data driven and community involved feedback (via Refstack) to engage the largest possible base for core decisions.
  4. There is a “safety valve” for vendors to deal with test scenarios that are difficult to recreate in the field.
  5. The Board is responsible for approving the final artifacts based on the recommendations.  By having a transparent process, community input is expected in advance of that approval.
  6. The process is time sensitive.  There’s a need for the Board to produce Core definition in a timely way after each release and then feed that into the next one.  Ideally, the definitions will be approved at the Board meeting immediately following the release.

DefCore Process Draft

Process shows how the key components: designated sections and capabilities start from the previous release’s version and the DefCore committee manages the update process.  Community input is a vital part of the cycle.  This is especially true for identifying actual use of the capabilities through the Refstack data collection site.

  • Blue is for Board activities
  • Yellow is or user/vendor community activities
  • Green is for technical community activities
  • White is for process artifacts

This process is very much in draft form and any input or discussion is welcome!  I expect DefCore to take up formal review of the process in October.

Cloud Culture: Reality has become a video game [Collaborative Series 3/8]

This post is #3 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

DO VIDEO GAMES REALLY MATTER THAT MUCH TO DIGITAL NATIVES?

Yes. Video games are the formative computer user experience (a.k.a. UX) for nearly everyone born since 1977. Genealogists call these people Gen X, Gen Y, or Millennials, but we use the more general term “Digital Natives” because they were born into a world surrounded by interactive digital technology starting from their toys and learning devices.

Malcolm Gladwell explains, in his book Outliers, that it takes 10,000 hours of practice to develop a core skill. In this case, video games have trained all generations since 1977 in a whole new way of thinking. It’s not worth debating if this is a common and ubiquitous experience; instead, we’re going to discuss the impact of this cultural tsunami.

Before we dive into impacts, it is critical for you to suspend your attitude about video games as a frivolous diversion. Brad explores this topic in Liquid Leadership, and Jane McGonnagle, in Reality is Broken, spends significant time exploring the incredibly valuable real world skills that Digital Natives hone playing games. When they are “gaming,” they are doing things that adults would classify as serious work:

  • Designing buildings and creating machines that work within their environment
  • Hosting communities and enforcing discipline within the group
  • Recruiting talent to collaborate on shared projects
  • Writing programs that improve their productivity
  • Solving challenging mental and physical problems under demanding time pressures
  • Learning to persevere through multiple trials and iterative learning
  • Memorizing complex sequences, facts, resource constraints, and situational rules.

Why focus on video gamers?

Because this series is about doing business with Digital Natives and video games are a core developmental experience.

The impact of Cloud Culture on technology has profound implications and is fertile ground for future collaboration between Rob and Brad.  However, we both felt that the challenge of selling to gamers crystallized the culture clash in a very practical and financially meaningful sense.  Culture can be a “soft” topic, but we’re putting a hard edge on it by bringing it home to business impacts.

Digital Natives play on a global scale and interact with each other in ways that Digital Immigrants cannot imagine. Brad tells it best with this story about his nephew:

Years ago, in a hurry to leave the house, we called out to our video game playing nephew to join us for dinner.

“Sebastian, we’re ready.” I was trying to be as gentle as possible without sounding Draconian. That was the parenting methods of my father’s generation. Structure. Discipline. Hierarchy. Fear. Instead, I wanted to be the Cool Uncle.

“I can’t,” he exclaimed as wooden drum sticks pounded out their high-pitched rhythm on the all too familiar color-coded plastic sensors of a Rock Band drum kit.

“What do you mean you can’t? Just stop the song, save your data, and let’s go.”

“You don’t understand. I’m in the middle of a song.” Tom Sawyer by RUSH to be exact. He was tackling Neil Peart. Not an easy task. I was impressed.

“What do you mean I don’t understand? Shut it off.” By now my impatience was noticeable. Wow, I lasted 10 seconds longer than my father if he had been in this same scenario. Progress I guess.

And then my 17-year-old nephew hit me with some cold hard facts without even knowing it… “You don’t understand… the guitar player is some guy in France, and the bass player is this girl in Japan.”

In my mind the aneurism that was forming just blew… “What did he just say?”

And there it was, sitting in my living room—a citizen of the digital age. He was connected to the world as if this was normal. Trained in virtualization, connected and involved in a world I was not even aware of!

My wife and I just looked at each other. This was the beginning of the work I do today. To get businesses to realize the world of the Digital Worker is a completely different world. This is a generation prepared to work in The Cloud Culture of the future.

A Quote from Liquid Leadership, Page 94, How Technology Influences Behavior…

In an article in the Atlantic magazine, writer Nicholas Carr (author of The Shallows: What the Internet Is Doing to Our Brains) cites sociologist Daniel Bell as claiming the following: “Whenever we begin to use ‘intellectual technologies’ such as computers (or video games)—tools that extend our mental rather than our physical capacities—we inevitably begin to take on the qualities of those technologies.

In other words, the technology we use changes our behavior!

There’s another important consideration about gamers and Digital Natives. As we stated in post 1, our focus for this series is not the average gamer; we are seeking the next generation of IT decision makers. Those people will be the true digital enthusiasts who have devoted even more energy to mastering the culture of gaming and understand intuitively how to win in the cloud.

“All your base belongs to us.”

Translation: If you’re not a gamer, can you work with Digital Natives?

Our goal for this series is to provide you with actionable insights that do not require rewriting how you work. We do not expect you to get a World of Warcraft subscription and try to catch up. If you already are one then we’ll help you cope with your Digital Immigrant coworkers.

In the next posts, we will explain four key culture differences between Digital Immigrants and Digital Natives. For each, we explore the basis for this belief and discuss how to facilitate Digital Natives decision-making processes.

Cloud Culture Series TL;DR? Generation Cloud Cheat sheet [Collaborative Series 2/8]

SUBTITLE: Your series is TOO LONG, I DID NOT READ It!

This post is #2 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

Your attention is valuable to us! In this section, you will find the contents of this entire blog series distilled down into a flow chart and one-page table.  Our plan is to release one post each Wednesday at 1 pm ET.

Graphical table of contents

flow chartThe following flow chart is provided for readers who are looking to maximize the efficiency of their reading experience.

If you are unfamiliar with flow charts, simply enter at the top left oval. Diamonds are questions for you to choose between answers on the departing arrows. The curved bottom boxes are posts in the series.

Culture conflict table (the Red versus Blue game map)

Our fundamental challenge is that the cultures of Digital Immigrants and Natives are diametrically opposed.  The Culture Conflict Table, below, maps out the key concepts that we explore in depth during this blog series.

Digital Immigrants (N00Bs) Digital Natives (L33Ts)
Foundation: Each culture has different expectations in partners
  Obey Rules

They want us to prove we are worthy to achieve “trusted advisor” status.

They are seeking partners who fit within their existing business practices.

Test Boundaries

They want us to prove that we are innovative and flexible.

They are seeking partners who bring new ideas that improve their business.

  1. Organizational Hierarchy see No Spacesuits (Post 4)
  Permission Driven

Organizational Hierarchy is efficient

Feel important talking high in the org

Higher ranks can make commitments

Bosses make decisions (slowly)

Peer-to-Peer Driven

Organizational Hierarchy is limiting

Feel productive talking lower in the org

Lower ranks are more collaborative

Teams make decisions (quickly)

  1. Communication Patterns see MMOG as Job Training (Post 5)
  Formalized & Structured

Waits for Permission

Bounded & Linear

Requirements Focused

Questions are interruptions

Casual & Interrupting

Does NOT KNOW they need permission

Open Ended

Discovered & Listening

Questions show engagement

  1. Risks and Rewards see Level Up (Post 6)
  Obeys Rules

Avoid Risk—mistakes get you fired!

Wait and see

Fear of “looking foolish”

Breaks Rules

Embrace Risk—mistakes speed learning

Iterate to succeed

Risks get you “in the game”

  1. Building your Expertise see Becoming L33T (Post 7)
Knowledge is Concentrated

Expertise is hard to get (Diploma)

Keeps secrets (keys to success)

Quantitate—you can measure it

Knowledge is Distributed and Shared

Expertise is easy to get (Google)

Likes sharing to earn respect

Qualitative—trusts intuition

Hopefully, this condensed version got you thinking.  In the next post, we start to break this information down.