Cloud Culture: Online Games, the real job training for Digital Natives [Collaborative Series 5/8]

Translation: Why do Digital Natives value collaboration over authority?

Kids Today

This post is #5 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

Before we start, we already know that some of you are cynical about what we are suggesting—Video games? Are you serious? But we’re not talking about Ms. Pac-Man. We are talking about deeply complex, rich storytelling, and task-driven games that rely on multiple missions, worldwide player communities, working together on a singular mission.

Leaders in the Cloud Generation not just know this environment, they excel in it.

The next generation of technology decision makers is made up of self-selected masters of the games. They enjoy the flow of learning and solving problems; however, they don’t expect to solve them alone or a single way. Today’s games are not about getting blocks to fall into lines; they are complex and nuanced. Winning is not about reflexes and reaction times; winning is about being adaptive and resourceful.

In these environments, it can look like chaos. Digital workspaces and processes are not random; they are leveraging new-generation skills. In the book Different, Youngme Moon explains how innovations looks crazy when they are first revealed. How is the work getting done? What is the goal here? These are called “results only work environments,” and studies have shown they increase productivity significantly.

Digital Natives reject top-down hierarchy.

These college educated self-starters are not rebels; they just understand that success is about process and dealing with complexity. They don’t need someone to spoon feed them instructions.

Studies at MIT and The London School of Economics have revealed that when high-end results are needed, giving people self-direction, the ability to master complex tasks, and the ability to serve a larger mission outside of themselves will garnish groundbreaking results.

Gaming does not create mind-addled Mountain Dew-addicted unhygienic drone workers. Digital Natives raised on video games are smart, computer savvy, educated, and, believe it or not, resourceful independent thinkers.

Thomas Edison said:

“I didn’t fail 3,000 times. I found 3,000 ways how not to create a light bulb.”

Being comfortable with making mistakes thousands of times ’til mastery sounds counter-intuitive until you realize that is how some of the greatest breakthroughs in science and physics were discovered.  Thomas Edison made 3,000 failed iterations in creating the light bulb.

Level up: You win the game by failing successfully.

Translation: Learn by playing, fail fast, and embrace risk.

Digital Natives have been trained to learn the rules of the game by just leaping in and trying. They seek out mentors, learn the politics at each level, and fail as many times as possible in order to learn how NOT to do something. Think about it this way: You gain more experience when you try and fail quickly then carefully planning every step of your journey. As long as you are willing to make adjustments to your plans, experience always trumps prediction.Just like in life and business, games no longer come with an instruction manual.

In Wii Sports, users learn the basic in-game and figure out the subtlety of the game as they level up. Tom Bissel, in Extra Lives: Why Video Games Matter, explains that the in-game learning model is core to the evolution of video games. Game design involves interactive learning through the game experience; consequently, we’ve trained Digital Natives that success comes from overcoming failure.

Cloud Culture: No spacesuits, Authority comes from doing, not altitude [Collaborative Series 4/8]

Subtitle: Why flattening org charts boosts your credibility

This post is #4 in an collaborative eight part series by Brad Szollose and I about how culture shapes technology.

Unlike other generations, Digital Natives believe that expertise comes directly from doing, not from position or education. This is not hubris; it’s a reflection both their computer experience and dramatic improvements in technology usability.

AstronautIf you follow Joel Spolsky’s blog, “Joel on Software,” you know about a term he uses when describing information architects obsessed with the abstract and not the details; Architecture Astronauts—so high up above the problem that they might as well be in space. “They’re astronauts because they are above the oxygen level, I don’t know how they’re breathing.”

For example, a Digital Native is much better positioned to fly a military attack drone than a Digital Immigrant. According to New Scientist, March 27, 2008, the military is using game controllers for drones and robots because they are “far more intuitive.” Beyond the fact that the interfaces are intuitive to them, Digital Natives have likely logged hundreds of hours flying simulated jets under trying battle conditions. Finally, they rightly expect that they can access all the operational parameters and technical notes about the plane with a Google search.

Our new workforce is ready to perform like none other in history.

Being able to perform is just the tip of the iceberg; having the right information is the more critical asset. A Digital Native knows information (and technology) is very fast moving and fluid. It also comes from all directions … after all it’s The Information Age. This is a radical paradigm shift. Harvard Researcher David Weinberger highlights in his book Too Big to Know that people are not looking up difficult technical problems in a book or even relying on their own experiences; they query their social networks and discover multiple valid solutions. The diversity of their sources is important to them, and an established hierarchy limits their visibility; inversely, they see leaders who build strict organizational hierarchies as cutting off their access to information and diversity.

Today’s thought worker is on the front lines of the technological revolution. They see all the newness, data, and interaction with a peer-to-peer network. Remember all that code on the screen in the movie The Matrix? You get the picture.

To a Digital Native, the vice presidents of most organizations are business astronauts floating too high above the world to see what’s really going on but feeling like they have perfect clarity. Who really knows the truth? Mission Control or Major Tom? This is especially true with the acceleration of business that we are experiencing. While the Astronaut in Chief is busy ordering the VPs to move the mountains out of the way, the engineers at ground control have already collaborated on a solution to leverage an existing coal mine and sell coal as a byproduct.

The business hierarchy of yesterday worked for a specific reason: workers needed to just follow rules, keep their mouth shut, and obey. Input, no matter how small, was seen as intrusive and insubordinate … and could get one fired. Henry Ford wanted an obedient worker to mass manufacture goods. The digital age requires a smarter worker because, in today’s world, we make very sophisticated stuff that does not conform to simple rules. Responsibility, troubleshooting, and decision-making has moved to the frontlines. This requires open-source style communication.

Do not confuse the Astronaut problem as a lack of respect for authority.

Digital Natives respect informational authority, not positional. For Digital Natives, authority is flexible. They have experience forming and dissolving teams to accomplish a mission. The mission leader is the one with the right knowledge and skills for the situation, not the most senior or highest scoring. In Liquid Leadership, Brad explains that Digital Natives are not expecting managers to solve team problems; they are looking to their leadership to help build, manage, and empower their teams to do it themselves.

So why not encourage more collaboration with a singular mission in mind: develop a better end product? In a world that is expanding at such mercurial speed, a great idea can come from anywhere! Even from a customer! So why not remember to include customers in the process?

Who is Leroy Jenkins?

This viral video is about a spectacular team failure from one individual (Leroy Jenkins) who goes rogue during a team massively multi-player game.  This is a Digital Natives’ version of the ant and grasshopper parable: “Don’t pull a Leroy Jenkins on us—we need to plan this out.”

Think about it like this: Working as a team is like joining a quest.

If comparing work to a game scenario sounds counterintuitive then let’s reframe the situation. We may have the same destination and goals, but we are from very different backgrounds. Some of us speak different languages, have different needs and wants. Some went to MIT, some to community college. Some came through Internet startups, others through competitors. Big, little, educated, and smart. Intense and humble. Outgoing and introverted.  Diversity of perspective creates stronger teams.

This also means that leadership roles rotate according to each mission.

This is the culture of the gaming universe. Missions and quests are equivalent to workplace tasks accomplished and point to benchmarks achieved. Each member excepts to earn a place through tasks and points. This is where Digital Natives’ experience becomes advantage. They expect to advance in experience and skills. When you adapt the workplace to these expectations the Digital Natives thrive.

Leaders need to come down to earth and remove the spacesuit.

A leader at the top needs to stay connected to that information and disruption. Start by removing your helmet. Breathe the same oxygen as the rest of us and give us solutions that can be used here on planet earth.

On Gamification

Jeff Attwood, founder of the community-based FAQ site Stack Overflow, has been very articulate about using game design to influence how he builds communities around sharing knowledge. We recommend reading his post about “Building Social Software for the Anti-Social” on his blog,

a Ready State analogy: “roughed in” brings it Home for non-ops-nerds

I’ve been seeing great acceptance on the concept of ops Ready State.  Technologists from both ops and dev immediately understand the need to “draw a line in the sand” between system prep and installation.  We also admit that getting physical infrastructure to Ready State is largely taken for granted; however, it often takes multiple attempts to get it right and even small application changes can require a full system rebuild.

Since even small changes can redefine the ready state requirements, changing Ready State can feel like being told to tear down your house so you remodel the kitchen.

Foundation RawA friend asked me to explain “Ready State” in non-technical terms.  So far, the best analogy that I’ve found is when a house is “Roughed In.”  It’s helpful if you’ve ever been part of house construction but may not be universally accessible so I’ll explain.

Foundation PouredGetting to Rough In means that all of the basic infrastructure of the house is in place but nothing is finished.  The foundation is poured, the plumbing lines are placed, the electrical mains are ready, the roof on and the walls are up.  The house is being built according to architectural plans and major decisions like how many rooms there are and the function of the rooms (bathroom, kitchen, great room, etc).  For Ready State, that’s like having the servers racked and setup with Disk, BIOS, and network configured.

Framed OutWhile we’ve built a lot, rough in is a relatively early milestone in construction.  Even major items like type of roof, siding and windows can still be changed.  Speaking of windows, this is like installing an operating system in Ready State.  We want to consider this as a distinct milestone because there’s still room to make changes.  Once the roof and exteriors are added, it becomes much more disruptive and expensive to make.

Roughed InOnce the house is roughed in, the finishing work begins.  Almost nothing from roughed in will be visible to the people living in the house.  Like a Ready State setup, the users interact with what gets laid on top of the infrastructure.  For homes it’s the walls, counters, fixtures and following.  For operators, its applications like Hadoop, OpenStack or CloudFoundry.

Taking this analogy back to where we started, what if we could make rebuilding an entire house take just a day?!  In construction, that’s simply not practical; however, we’re getting to a place in Ops where automation makes it possible to reconstruct the infrastructure configuration much faster.

While we can’t re-pour the foundation (aka swap out physical gear) instantly, we should be able to build up from there to ready state in a much more repeatable way.

OpenCrowbar Design Principles: Emergent services [Series 5 of 6]

This is part 5 of 6 in a series discussing the principles behind the “ready state” and other concepts implemented in OpenCrowbar.  The content is reposted from the OpenCrowbar docs repo.

Emergent services

We see data center operations as a duel between conflicting priorities. On one hand, the environment is constantly changing and systems must adapt quickly to these changes. On the other hand, users of the infrastructure expect it to provide stable and consistent services for consumption. We’ve described that as “always ready, never finished.”

Our solution to this duality to expect that the infrastructure Crowbar builds is decomposed into well-defined service layers that can be (re)assembled dynamically. Rather than require any component of the system to be in a ready state, Crowbar design principles assume that we can automate the construction of every level of the infrastructure from bios to network and application. Consequently, we can hold off (re)making decisions at the bottom levels until we’ve figured out that we’re doing at the top.

Effectively, we allow the overall infrastructure services configuration to evolve or emerge based on the desired end use. These concepts are built on computer science principles that we have appropriated for Ops use; since we also subscribe to Opscode “infrastructure as code”, we believe that these terms are fitting in a DevOps environment. In the next pages, we’ll explore the principles behind this approach including concepts around simulated annealing, late binding, attribute injection and emergent design.

Emergent (aka iterative or evolutionary) design challenges the traditional assumption that all factors must be known before starting

  • Dependency graph – multidimensional relationship
  • High degree of reuse via abstraction and isolation of service boundaries.
  • Increasing complexity of deployments means more dependencies
  • Increasing revision rates of dependencies but with higher stability of APIs

OpenCrowbar Design Principles: Late Binding [Series 3 of 6]

This is part 3 of 6 in a series discussing the principles behind the “ready state” and other concepts implemented in OpenCrowbar.  The content is reposted from the OpenCrowbar docs repo.

2013-09-13_18-56-39_197Ops Late Binding

In terms of computer science languages, late binding describes a class of 4th generation languages that do not require programmers to know all the details of the information they will store until the data is actually stored. Historically, computers required very exact and prescriptive data models, but later generation languages embraced a more flexible binding.

Ops is fluid and situational.

Many DevOps tooling leverages eventual consistency to create stable deployments. This iterative approach assumes that repeated attempts of executing the same idempotent scripts do deliver this result; however, they are do not deliver predictable upgrades in situations where there are circular dependencies to resolve.

It’s not realistic to predict the exact configuration of a system in advance –

  • the operational requirements recursively impact how the infrastructure is configured
  • ops environments must be highly dynamic
  • resilience requires configurations to be change tolerant

Even more complex upgrade where the steps cannot be determined in advanced because the specifics of the deployment direct the upgrade.

Late Binding is a  foundational topic for Crowbar that we’ve been talking about since mid-2012.  I believe that it’s an essential operational consideration to handle resiliency and upgrades.  We’ve baked it deeply into OpenCrowbar design.

Continue Reading > post 4

OpenCrowbar Design Principles: The Ops Challenge [Series 2 of 6]

This is part 2 of 6 in a series discussing the principles behind the “ready state” and other concepts implemented in OpenCrowbar.  The content is reposted from the OpenCrowbar docs repo.

The operations challenge

A deployment framework is key to solving the problems of deploying, configuring, and scaling open source clusters for cloud computing.

2012-09-21_13-51-00_331Deploying an open source cloud can be a complex undertaking. Manual processes, can take days or even weeks working to get a cloud fully operational. Even then, a cloud is never static, in the real world cloud solutions are constantly on an upgrade or improvement path. There is continuous need to deploy new servers, add management capabilities, and track the upstream releases, while keeping the cloud running, and providing reliable services to end users. Service continuity requirements dictate a need for automation and orchestration. There is no other way to reduce the cost while improving the uptime reliability of a cloud.

These were among the challenges that drove the development of the OpenCrowbar software framework from it’s roots as an OpenStack installer into a much broader orchestration tool. Because of this evolution, OpenCrowbar has a number of architectural features to address these challenges:

  • Abstraction Around OrchestrationOpenCrowbar is designed to simplify the operations of large scale cloud infrastructure by providing a higher level abstraction on top of existing configuration management and orchestration tools based on a layered deployment model.
  • Web ArchitectureOpenCrowbar is implemented as a web application server, with a full user interface and a predictable and consistent REST API.
  • Platform Agnostic ImplementationOpenCrowbar is designed to be platform and operating system agnostic. It supports discovery and provisioning from a bare metal state, including hardware configuration, updating and configuring BIOS and BMC boards, and operating system installation. Multiple operating systems and heterogeneous operating systems are supported. OpenCrowbar enables use of time-honored tools, industry standard tools, and any form of scriptable facility to perform its state transition operations.
  • Modular ArchitectureOpenCrowbar is designed around modular plug-ins called Barclamps. Barclamps allow for extensibility and customization while encapsulating layers of deployment in manageable units.
  • State Transition Management EngineThe core of OpenCrowbar is based on a state machine (we call it the Annealer) that tracks nodes, roles, and their relationships in groups called deployments. The state machine is responsible for analyzing dependencies and scheduling state transition operations (transitions).
  • Data modelOpenCrowbar uses a dedicated database to track system state and data. As discovery and deployment progresses, system data is collected and made available to other components in the system. Individual components can access and update this data, reducing dependencies through a combination of deferred binding and runtime attribute injection.
  • Network AbstractionOpenCrowbar is designed to support a flexible network abstraction, where physical interfaces, BMC’s, VLANS, binding, teaming, and other low level features are mapped to logical conduits, which can be referenced by other components. Networking configurations can be created dynamically to adapt to changing infrastructure.

Continue Reading > post 3

Networking in Cloud Environments, SDN, NFV, and why it matters [part 1 of 2]

scott_jensen2Scott Jensen is an Engineering Director and colleague of mine from Dell with deep networking and operations experience.  He had first hand experience deploying OpenStack and Hadoop and has a critical role in defining Dell’s Reference Architectures in those areas.  When I saw this writeup about cloud networking, I asked if it would be OK to share it with you.

Guest Post 1 of 2 by Scott Jensen:

Having a basis in enterprise data center networking, Cloud computing I have many conversations with customers implementing a cloud infrastructure.  Their design the networking infrastructure can and should be different from a classic network configuration and many do not understand why.  Either due to a lack of knowledge in networking or due to a lack of understanding as to why cloud computing is different from virtualization.  Once you have an understanding of both of these areas you can begin to see why emerging technologies such as SDN (Software Defined Networking) and NFV (Network Function Virtualization) begin to address some of the issues that Cloud Computing can cause with your network.

Networking is all about traffic flows.  In order to properly design your infrastructure you need to understand where traffic is originating, where it is going and how much traffic will be following a specific route and at what times.

There are many differences between Cloud Computing and virtualization.  In many cases people I will talk to think of Cloud as virtualization in a different environment.  Of course this will work just fine however it does not take advantage of the goodness that a Cloud infrastructure can bring.  Some of the major differences between Virtualization and Cloud Computing have profound effects on how the network is utilized.  This all has to do with the application.  That is really what it is all about anyway.  Rob Hirschfeld has a great post on the difference between Pets and Cattle which describes this well.

Pets and Cattle as a workload evolution

In typical virtualized infrastructures, the applications have a fairly common pattern.  Many people describe these as Pets and are managed largely the same as a physical system.  They have a name, they are one of a kind, they are cared for, and when the die it can be traumatic (I know I have been there).

  • They run on large stateful VMs
  • They have a lifecycle which is typically very long such as years
  • The applications themselves are not designed to tolerate failures.  Other technologies are brought in to ensure uptime.
  • The application is scaled up when demands increase.  This is done by adding more memory or CPU to the VM.

Cloud applications are different.  Some people describe them as cattle and they are treated like cattle in many ways.  They do not necessarily have a name and if one dies it is sad but not a really big deal.  We should probably figure out what killed it but life goes on.

  • They run on smaller stateless VMs
  • They have a lifecycle measured in hours or months.  Sometimes even less than an hour.
  • The application is designed to expect failures
  • The application scales out by increasing the number of instances which is running when the demand increases.

In his follow-up post next week, Scott discusses how this impacts the network and how SDN and NFV promises to help.