DevOps approaches to upgrade: Cube Visualization

Posted on April 3, 2013 by Rob H

I’m working on my OpenStack summit talk about DevOps upgrade patterns and got to a point where there are three major vectors to consider:

Step Size (shown as X axis): do we make upgrades in small frequent steps or queue up changes into larger bundles? Larger steps mean that there are more changes to be accommodated simultaneously.
Change Leader (shown as Y axis): do we upgrade the server or the client first? Regardless of the choice, the followers should be able to handle multiple protocol versions if we are going to have any hope of a reasonable upgrade.
Safeness (shown as Z axis): do the changes preserve the data and productivity of the entity being upgraded? It is simpler to assume to we simply add new components and remove old components; this approach carries significant risks or redundancy requirements.

I’m strongly biased towards continuous deployment because I think it reduces risk and increases agility; however, I laying out all the vertices of the upgrade cube help to visualize where the costs and risks are being added into the traditional upgrade models.

Breaking down each vertex:

Continuous Deploy – core infrastructure is updated on a regular (usually daily or faster) basis
Protocol Driven – like changing to HTML5, the clients are tolerant to multiple protocols and changes take a long time to roll out
Staged Upgrade – tightly coordinate migration between major versions over a short period of time in which all of the components in the system step from one version to the next together.
Rolling Upgrade – system operates a small band of versions simultaneously where the components with the oldest versions are in process of being removed and their capacity replaced with new nodes using the latest versions.
Parallel Operation – two server systems operate and clients choose when to migrate to the latest version.
Protocol Stepping – rollout of clients that support multiple versions and then upgrade the server infrastructure only after all clients have achieved can support both versions.
Forced Client Migration – change the server infrastructure and then force the clients to upgrade before they can reconnect.
Big Bang – you have to shut down all components of the system to upgrade it

This type of visualization helps me identify costs and options. It’s not likely to get much time in the final presentation so I’m hoping to hear in advance if it resonates with others.

PS: like this visualization? check out my “magic 8 cube” for cloud hosting options.

What foo is “contribution” to open source? Mik Kersten & Tasktop @ SXSW

Posted on March 20, 2013 by Rob H

How do we really know who influences most in a software project? We can easily track code commits, but there are more bits to the project than the commits.

I had the good fortune to attend Mik Kersten’s Code Graph presentation at SXSW last week. Mik started the Eclipse Mylyn project and went on to found Tasktop. Both are built on the very intriguing concepts that software development production (aka work) is organized around tasks.

His premise is that organizing around tasks provides a more manageable and actionable view of a project than a more traditional application life-cycle management (ALM) approaches. I’m a sucker for any presentation about lean development process that includes references to both DevOps and industrial engineering (I have an MS in IE), but Mik surprised me by taking his code graph concept to a whole ‘nutha level.

The software value chain is much deeper than just the people who write code. Mik’s approach included managers, testers and operators in the interaction graphs for his projects.

By including all of the ALM artifacts in the analysis, you get a much richer picture of the influencers for a project.

For example, the development manager may never show up as a code committer; however, they are hugely influential in which work gets prioritized. If your graph includes who is touching the work assignments and stories then the manager’s influence jumps out immediately. That knowledge would completely change how and who you may interact with a team. It effectively brings a shadow contributor into the light.

The same is true for QA members who are running tests and opening defects and operators who are building deployment scripts. Ideally, it should include users who exercise different parts of the applications capabilities.

Mik’s graphs clearly showed the influence impact of managers because they touched all of the story cards for the project. The people who own the story cards are the most potent influencers in a project, yet they are invisible in code repositories!

I would love to see an impact graph for a software project that equally reflected the wide range of contributions that people make to its life-cycle. This type of information helps rebalance the power in a project.

Industrial engineering legend W.E. Demming‘s advice is to look at production as a system. Finding ways to show everyone’s contributions is an important step towards bringing lean processes fully into software manufacturing.

Creating Communities: the intersection between Twitter celebrities and open source

Posted on March 15, 2013 by Rob H

One of the unexpected perks of my Chevy SXSW experience was access to some real social media celebrities such as Josh Estrin, Calvin Lee, Kristin Brandt, Doug Mora, Samantha Needham and Jennie Chen. They are all amazing, fun, wicked smart and NOT INTO CLOUD COMPUTING.

While I already knew Samantha (via Dell) and Jennie (via TechRanch), all of Chevy’s guests brought totally different perspectives to Chevy’s SXSW team ranging from pop culture and mommies to hypermilers and gearheads.

The common thread is that we are all looking to engage our communities.

We each wanted to find something that would be interesting for our very different audiences to discuss. That meant using our experiences at SXSW, Chevy and with each other to start a conversation within our communities. We need good content as a seed but the goal is to drive the interaction.

Josh was the most articulate about this point saying that he measured his success when his followers talked to each other more than to him. Being able to create content that engages people to do that is a true talent.

Calvin’s focus was more on helping people connect. He felt successful when he was able to bring people together through his extended network. In those cases and others, the goals and challenges of a social media celebrity were remarkably similar to those helping lead open source projects.

In building communities, you must measure success in member communication and interaction.

If you are intent on being at the center of the universe then your project cannot grow; however, people also need celebrities to bring them together. The amazing thing about the the people I met at SXSW through Chevy is that they managed to both attract the attention needed to build critical mass and get out of the way so communities could form around them. That’s a skill that we all should practice and foster.

PS: I also heard clearly that “I ate …” tweets are some of their most popular. Putting on my collaboration hat: if you’re looking to engage a community then food is the most universal and accessible discussion topic. Perhaps I’ll have to eat crow on that one.

SXSW Volt hard to give up – this is the EV that my family wants me to buy

Posted on March 15, 2013 by Rob H

Today Chevy took my SXSW loaner Volt back to the dealership in the cloud. While I was already inclined toward the Volt, I was much more impressed that I expected. Frankly, the fact that the Volt and my home-conversion RAVolt both have a 30 mile range made the Volt seem like a pretend EV on the surface. Yet, the Volt proved it was a full EV and more.

The Volt is a very solid electric car with the all torque and feel I was expecting.

Since the Volt was delivered without a charge, my first day in the car relied on the gas hybrid feature. Even on gas, I was getting >40 MPG. While I would have preferred to get the Volt fully charged, it was an important lesson for me to see it perform as a gas fueled car first.

Once I got it charged, I was able to drive nearly all my trips electric only. That included my 30+ mile commute. The magic of the Volt is that I never worried about running out of charge. As an EV driver who has had close scrapes, that is a truly liberating experience.

I enjoyed playing with the Volt’s drive system. I managed to find an accurate power input/output gauge on the dash instrumentation (the center console view is mainly a pretty animation for passengers). I was able to extend my electric range using feedback from the more accurate gauge. In addition, the built-in tutorial explained that I could use the PRNDL “low gear” in traffic. Low gear activated behaviors in the Volt that made it more efficient AND EASIER to drive in traffic than conventional gas cars. I was also intrigued by how efficiently the Volt used its gas generator – it used some smarts so the generator ran as little of possible. Of course, I also enjoyed the acceleration of the electric motor 🙂

In addition to the power train, I found the fit and finish of the Volt to be very satisfying. The electronics were effective, interior comfortable and handling responsive. While it did not pretend to be a luxury car, the Volt does not feel like a econobox either.

One note: if you are considering a Volt then plan on a 220 charger. Relying on charging from household power is simply not practical. Using 220, you can charge quickly at night when power is cheapest/cleanest to generate.

Helping redefine “what is a car” on Chevy tours at SXSWi

Posted on March 13, 2013 by Rob H

I’m at SWSW as a guest of Chevy and enjoying the benefits of behind the scenes tours and access. On Friday, the Volt team toured me and a collection of bloggers and journalists through the Pecan Street project and GM’s customer interaction center.

At one point, Colin Rowan compared the relatively long cell phone adoption (they first appeared in 1973) to the likely ramp of electric cars and green homes. Doug Moran from GearDiary pointed out the weakness of this comparison. Cell phones are inexpensive with short life-cycles while cars are expensive durables.

However, Cell phones stopped being phones around 2007 when iPhone adoption exploded. Smart phones are not phones – they are mobile platforms. In fact, they are lousy phones when compared to cell phones.

Comparing electric cars to gas cars is more like comparing smart phones to dumb phones!

While both electric and gas cars can be used for transportation, electric cars have the potential to become energy transportation platforms.

You cannot use the energy stored in your car’s gas tank for anything but moving the car around. Further, you can only get more gas from a very small set of vendors.

Electric cars are fundamentally different – the energy stored in the car’s batteries can be apply to nearly any application you want from transportation to lighting to computing to heating and refrigeration. Further, you can get more energy for your car from nearly any source (local solar and wind or grid power). For a hybrid like the Volt, the options are even broader because it includes gas to electricity generation.

From this perspective, electric cars are an energy mobility platform.

We need to accept that we are living in a world with unreliable power distribution due to weather, peak demand and/or carbon tax. In this type of situation, cars with batteries are as fundamentally different from gas cars as smart phones are to rotary POTS phone.

PS: For more extra credit reading, check out the Vehicle to Grid concept

SXSWi bound thanks to GM & Chevy Volt

Posted on March 8, 2013 by Rob H

I’ll be blogging from SXSWi over the next few days as a guest of the Chevy Volt team AND they’ve given me a Volt to drive for the week (disclaimer: total cash value of $1150). I’ve been in Austin for over 10 years and find it ironic that my electric car past gets me to event instead of my cloud or software work. Either way, I’m delighted to attend.

Yes, I have electric car construction experience…

About 6 years ago before the first days of $4 gas, I took on the entrepreneurial/science project of converting an electric car (a 96 RAV4) to run on batteries. The result proved clearly that there was no sustainable business in converting gas cars to electric because the mechanics are different enough to require purpose-built design.

My conversion project, the RAVolt, is still in daily use with over 3,000 electric miles. I never intended it to be a show car – my goal was to be time and cash effective so it uses the most time-tested components: lead acid batteries, 18 hp elevator motor and a forklift DC control system. With a 30 minute range, it has limited utility.

And now, I find myself in the market for a new car AND being given a Volt for the week. The Volt was on my short list to consider (I drive a Honda Fit now) along with the Tesla S, Leaf and Smart. This topic deserves it’s own post.

For the next few days, I’m going to wallow in the SXSWi nerdfest and enjoy experimenting with a production class electric car. I will, of course, be sharing my experiences and observations about both here and on twitter. Both are subjects of long-standing interest.

Disclaimer: Chevy has provided me with a SXSWi pass and Volt use ($1150 value). They have asked for nothing in return and I have made no commitments for favorable comments. Further, my employer, Dell, is aware of this arrangement with Chevy. My opinions are my own.

double Block Head with OpenStack+Equallogic & Crowbar+Ceph

Posted on December 14, 2012 by Rob H

Whew….Yesterday, Dell announced TWO OpenStack block storage capabilities (Equallogic & Ceph) for our OpenStack Essex Solution (I’m on the Dell OpenStack/Crowbar team) and community edition. The addition of block storage effectively fills the “persistent storage” gap in the solution. I’m quadrupally excited because we now have:

both open source (Ceph) and enterprise (Equallogic) choices
both Nova drivers’ code is in the open at part of our open source Crowbar work

Frankly, I’ve been having trouble sitting on the news until Dell World because both features have been available in Github before the announcement (EQLX and Ceph-Barclamp). Such is the emerging intersection of corporate marketing and open source.

As you may expect, we are delivering them through Crowbar; however, we’ve already had customers pickup the EQLX code and apply it without Crowbar.

The Equallogic+Nova Connector

If you are using Crowbar 1.5 (Essex 2) then you already have the code! Of course, you still need to have the admin information for your SAN – we did not automate the configuration of the storage system, but the Nova Volume integration.

We have it under a split test so you need to do the following to enable the configuration options:

Install OpenStack as normal
Create the Nova proposal
Enter “Raw” Attribute Mode
Change the “volume_type” to “eqlx”
Save
The Equallogic options should be available in the custom attribute editor! (of course, you can edit in raw mode too)

Want Docs? Got them! Check out these > EQLX Driver Install Addendum

Usage note: the integration uses SSH sessions. It has been performance tested but not been tested at scale.

The Ceph+Nova Connector

The Ceph capability includes a Ceph barclamp! That means that all the work to setup and configure Ceph is done automatically done by Crowbar. Even better, their Nova barclamp (Ceph provides it from their site) will automatically find the Ceph proposal and link the components together!

Ceph has provided excellent directions and videos to support this install.

My Dilemma with Folsom – why I want to jump to G

Posted on December 6, 2012 by Rob H

These views are my own. Based on 1×1 discussions I’ve had in the OpenStack community, I am not alone.

If you’ve read my blog then you know I am a vocal and active supporter of OpenStack who is seeking re-election to the OpenStack Board. I’m personally and professionally committed to the project’s success. And, I’m confident that OpenStack’s collaborative community approach is out innovating other clouds.

A vibrant project requires that we reflect honestly: we have an equal measure of challenges: shadow free fall Dev, API vs implementation, forking risk and others. As someone helping users deploy OpenStack today, I find my self straddling between a solid release (Essex) and a innovative one (Grizzly). Frankly, I’m finding it very difficult to focus on Folsom.

Grizzly excites me and clearly I’m not alone. Based on pace of development, I believe we saw a significant developer migration during feature freeze free fall.

In Grizzly, both Cinder and Quantum will have progressed to a point where they are ready for mainstream consumption. That means that OpenStack will have achieved the cloud API trifecta of compute-store-network.

Cinder will get beyond the “replace Nova Volume” feature set and expands the list of connectors.
Quantum will get to parity with Nova Network, addresses overlapping VM IPs and goes beyond L2 with L3 feature enablement like load balancing aaS.
We are having a real dialog about upgrades while the code is still in progress
And new projects like Celio and Heat are poised to address real use problems in billing and application formation.

Everything I hear about Folsom deployment is positive with stable code and significant improvements; however, we’re too late to really influence operability at the code level because the Folsom release is done. This is not a new dilemma. As operators, we seem to be forever chasing the tail of the release.

The perpetual cycle of implementing deployment after release is futile, exhausting and demoralizing because we finish just in time for the spotlight to shift to the next release.

I don’t want to slow the pace of releases. In Agile/Lean, we believe that if something is hard then we do should it more. Instead, I am looking at Grizzly and seeing an opportunity to break the cycle. I am looking at Folsom and thinking that most people will be OK with Essex for a little longer.

Maybe I’m a dreamer, but if we can close the deployment time gap then we accelerate adoption, innovation and happy hour. If that means jilting Folsom at the release altar to elope with Grizzly then I can live with that.

Impressions from OpenStack Foundation board meeting 10/15

Posted on October 16, 2012 by Rob H

I spent the first day of the OpenStack summit doing my service as a Foundation board member. While you need to wait for a summary by Jonathan Bryce, the Foundation Executive Director, and/or the official minutes to get the details of our actions, I think that it’s reasonable to share my impressions.

In some ways, the chemistry of the board is as important as our actions at this point. We are still working thorough start up issues and, more importantly, learning how to work together. The board is not only big, 24 members, it is also has many companies that often compete (power of we).

Here are my top 5 impressions:

Transparency. The bylaws are crafted to ensure a openness and our meetings are broadcast to the maximum extent possible. Even with that background, there is a consistent self-check and discussion about increasing transparency so that the community is included.
Humor. It’s a great sign of progress that we are laughing together because it shows that we trust and respect each other.
Frustration. Recognition that we have important decisions to make and a degree of impatience to make them. Boards are subtle: we spend a lot of time setting up the right structures that allow us to make hard decisions quickly. Also, grumbling that we’d overlapped the board meeting and the summit.
Consensus. This something I committed to help build. I feel that our actions reflect both healthy discussion as individuals and desire to work together as a board.
Leadership. Looking back from our first meeting, it is clear that Jonathan and Alan have fully assumed their roles. I’m also seeing how the Foundation team (Mark, Lauren, Stephano) are smoothly supporting operations even though we’re really just weeks from the Foundation launch.

I hope that gives you some insights into the board meeting and that more of you can join the broadcast for the next ones (2012 schedule TBD) including a joint tech and foundation meeting

Wednesday morning at the summit was “breakfast with the board.” Look for some notes from that too soon.

Open Source is The Power of We (Blog Action Day)

Posted on October 15, 2012 by Rob H

This post is part of a world wide “blog action day” where thousands of bloggers post their unique insights about a single theme. For 2012, it’s the “power of we is as a celebration of people working together to make a positive difference in the world, either for their own communities or for people they will never meet half way around the world.”

I’ve choosing open source software because I think that we are establishing models for building ideas collaboratively that can be extended beyond technology into broader use. The way we solve open source challenges translates broadly because we are the tool makers of the global interaction.

I started using open source¹ as a way to solve a problem; I did not understand community or how groups of loosely connected people came together to create something new. Frankly, the whole process of creating free software seemed to be some hybrid combination of ninja coders and hippy hackers. That changed when I got involve on the ground floor of the OpenStack project (of which I am now a Foundation board member).

I was not, could not have been, prepared for the power and reality of community and collaboration that fuels OpenStack and other projects. We have the same problems as any non-profit project except that we are technologists: we can make new tools to solve our teaming and process problems.

It is not just that open source projects solve problems that help people. The idea of OpenStack and Hadoop being used by medical researches to find cures for cancer is important; however, the learning how to build collaboratively is another critical dimension. Our world is getting more connected and interconnected by technology, but the actual tools for social media are only in their earliest stages.

Not only are the tools evolving, the people using the tools are changing too! We are training each other to work together in ways that were beyond our imagine even 10 years ago. It’s the combination of both new technology and new skills that is resetting the rules for collaboration.

Just a few years ago, open source technology was considered low quality, risky and fringe. Today, open source projects like OpenStack and Hadoop are seen as more innovative and equally secure and supportable compared to licensed products. This transformation represents a surprising alignment and collaboration between individuals and entities that would normally be competing. While the motivation for this behavior comes from many sources, we all share the desire to do collaborative effectively.

I don’t think that we have figured out how to really do this the best way yet. We are making progress and getting better and better. We are building tools (like etherpad, wikis, irc, twitter, github, jenkins, etc) that improve collaboration. We are also learning building a culture of collaboration.

Right now, I’m on a train bound for the semi-annual OpenStack summit that brings a world wide audience together for 4½ days of community work. The discussions will require a new degree of openness from people and companies that are normally competitive and secretive about product development. During the summit, we’ll be doing more than designing OpenStack, we will be learning the new skills of working together. Perhaps those are the most important deliverables.

Open source projects combination of both new technologies and new skills creates the Power of We.

——————

PS¹: Open source software is a growing class of applications in which the authors publish the instructions for running the software publicly so that other people can use the software. Sometimes (but not always) this includes a usage license that allows other people to run the software without paying the author royalties. In many cases, the author’s motivation is that other users will help them test, modify and improve the software so that improves more quickly than a single creator could do alone.

Rob Hirschfeld

On Computing, Containers, Cloud & Tech Culture

Category Archives: Events