Unknown's avatar

About Rob H

A Baltimore transplant to Austin, Rob thinks about ways of building scale infrastructure for the clouds using Agile processes. He sat on the OpenStack Foundation board for four years. He co-founded RackN enable software that creates hyperscale converged infrastructure.

Strategy is…

Strategy is confidence and execution.

If you’ve got a process that supports your strategy, you can execute. 

You can say no to the right things and focus on the important element  Strategy is not just knowing where you are going, it’s knowing which things to say no to along the way. 

We suffer from an over-abundance of opportunity.  Success often means knowing what not to do.

OpenStack Board Voting Starts! Three thoughts about the board and election

1/18 Note: I was re-elected! Thanks everyone for your support.

OpenStack Foundation members, I have three requests for you:

  1. vote (ballots went out by email already and expire on Friday)
  2. vote for me (details below)
  3. pick the board you want, not just the candidates.

And, of course, vote according to our code of conduct: “Members should not attempt to manipulate election results. Open debate is welcome, but vote trading, ballot stuffing and other forms of abuse are not acceptable.”

To vote, you must have been a member for 6 months. Going Mad Panda? Join right now so you can vote next time!

Even if you’re not a voter, you may find my comments useful in understanding OpenStack. Remember, these positions are my own: they do not reflect those of my employer (Dell).

Vote because it’s the strongest voice the community has about OpenStack governance

I’d strongly encourage you to commit code, come to the summit and participate in the lists and chats; however, voting is more. It shows measurable impact. It shows the vitality of the community.

OpenStack is exciting to me because it’s community driven.

I have been a strong and early leader in the community:

Vote for me because of my OpenStack Operations focus

My team at Dell has been laser focused on making OpenStack operable for over 2 years. My team acts like a lean start-up inside of Dell to get products out to market early.

We’ve worked with early operators and ground-breaking ecosystem partners (plus). We’ve helped many people get OpenStack running for real workloads. Consequently, I am highly accessible and accountable to a large number of users, operators and ecosystem vendors. Few other people in the OpenStack community have my breadth of exposure to real OpenStack deployments. My team and I have been dedicated, active participants in OpenStack long before Dell’s grand OpenStack strategy coalesced.

More importantly, I am committed to open source and open operations. The Apache 2 open source Crowbar project established the baseline (and served as the foundation) for other OpenStack deployment efforts. We don’t just talk about open source, we do open source: our team works in the open (my git account).

Vote for a board because diversity of views is important

OpenStack voting allows you to throw up to 8 votes to a single candidate. While you can put all your eggs into a single basket, I recommend considering a broader slate in voting.

We have an impressive and dedicated list of candidates who will do work for the community. I ask that you consider the board as a team. If you want operators, users, diversity, change, less corporate influence or any flavor of the above then think not just about the individual.

Why should I be part of your slate? I am not a blind cheerleader. I have voiced and driven community-focused positions about the board (dev cycle, grizzly >;; folsom, consensus).

One of my core activities has been to work on addressing community concerns about affinity voting. Gold and Platinum members have little incentive to address this issue. While I’d like to see faster action, it requires changing the by-laws. Any by-law changes need to be made carefully and they also require a vote of the electorate. If you want positive change, you need board members, like me, who are persistent and knowledgeable in board dynamics.

What have I done on the Board? Quite a bit…

It would be easy to get lost in a board of 24 members, but I have not. Armed with my previous board experience, I have been a vocal advocate for the community in board meetings without creating disruption or pulling us off topic.

Why re-elect me? For community & continuity

I am a strong and active board member and I work hard to represent all OpenStack constituents: developers, operators, users and ecosystem vendors.

Working on a board is a long-term exercise. Positions and actions I’ve taken today may take months to come to fruition. I would like the opportunity continue that work and see it through.

OpenStack Board Interview Series by Rafael Knuth

Rafael Knuth, Dell Rafael Knuth (@RafaelKnuth with Dell as EMEA Social Media Community Manager) has set an ambitious objective to interview all 24 of the OpenStack Foundation Board members. I had the privilege of being the first interview posted (Japanese version!).

Here’s the series so far:

  1. Rob Hirschfeld (me!) of Dell (my summary)
  2. Boris Renski of Mirantis
  3. Randy Bias of CloudScaling
  4. Jim Curry of Rackspace
  5. Yujie Due of 99cloud
  6. Tim Bell of CERN
  7. Joshua McKenty of Piston
  8. Lew Tucker of Cisco
  9. Alan Clark (Chair) of SUSE  
  10. Joseph George of Dell (added 5/22)

These are excellent interviews and worth reviewing. They give balanced (often critical) views of OpenStack, its progress and challenges.

I update the list as he adds them.

Don’t complicate my cloud! It’s just infrastructure with an API

Getty MazeI’ve been “in cloud” for over 13 years (@dmcrory and I submitted patents using it starting in 2001) and I’m continually amazed at how complicated people want to make it.

For my role at Dell, I’m continually invited to seasons of meetings to define cloud, cloud architecture and cloud strategy. The reason these meetings go on and on is that everyone wants to make cloud complicated when it’s really very simple.

Cloud is infrastructure with an API.

That’s it. Everything else is just a consequence of having infrastructure with an API because API provides the ability to provide remote control.

What else do people try to lump into cloud?  Here are some of my topic cloud obfuscators:

  • (inter)network.  Yes, networks make an API interesting.  They are just an essential component but they are not cloud.  Most technologies are interesting because of networks: can we stop turning everything networked into cloud?  Thanks to nonsensical mega-dollar marketing campaigns, I despair this is a moot point.
  • as-a-service.  That’s another way of saying “accessible via an API.”  We have many flavors of Platform, Data, Application, Love or whatever as a Service.  That means they have a API.  Infrastructure as a Service (IaaS) is a cloud.
  • virtualization.  VMs were the first good example of hardware with an API; however, we had virtual containers (on Mainframes!) long before we had “cloud.”    They make cloud easier but they are not cloud.
  • pay-as-you-go (service pricing).  This is a common cloud requirement but it’s really a business model.  If someone builds a private cloud then it is still a cloud.

  • multi-tenant.  Another common requirement where we expect a cloud to be able to isolate users.  I agree that this is a highly desirable attribute of a good API implementation; however, it’s not essential to a cloud.  For example, most public clouds do not have true network isolation model.
  • elastic demand.  IMHO, another word for API driven provisioning.
  • live migration.  This is a cool feature often implemented on top of virtualization, but it’s not cloud.  We were doing live migrate with shared storage and clusters before anyone heard of cloud.   I don’t think this is cloud at all but someone out there does so I included it in the list.
  • security.  Totally important consideration and required for deployments large and small, but not presence/lack does not make something cloud.

We start talking about these points and then forget the whole API thing.  These items are important, but they do not make it “a cloud.”  When Dave McCrory and I first discussed API Infrastructure as “cloud,” it was driven by the fact that you could hide the actual infrastructure behind the API.  The critical concept was that the API allowed a you to manage a server anywhere from anywhere.

When Amazon offered the first EC2 service, it had to be a cloud because the servers were remote. It was not a cloud because it was on the internet; plenty of other companies were offering hosted servers. It was a cloud because their offering allowed required operators to use and API to interact with the infrastructure.  I remember that EC2’s lack of UI (and SLA) causing many to predict it would be a failure; instead, it sparked a revolution.

I’m excited now because we’re entering a new generation of cloud where Infrastructure APIs include networking and storage in addition to compute.  Mix in some of the interesting data and network services and we’re going to have truly dynamic and powerful clouds.  More importantly, we going to have some truly amazing applications.

What do you think?  Is API a sufficient definition of cloud in your opinion?

PS: Yes, if you have a physical server/network/store that is completely controllable by an API then you’ve got a cloud on your hands.

I respectfully disagree – we are totally aligned on your lack of understanding

Team FacesOccasionally, my journeys into Agile and Lean process force me down to its foundation: cultural fit.  Frankly, there is nothing more central to the success of a team than culture. That’s especially true about Lean because of the humility and honesty required. If your team is not built on a foundation of trust and shared values then it’s impossible keep having the listening and responsive dialog with our customers.

Successful teams have to be honest about taking negative feedback and you cannot do that without trust.

Trust is built on working out differences. Ideally, it would be as simple as “we agree” or “we disagree.” In an ideal world, every team would be that binary.    Remember, that no team always agrees – it’s how we resolve those differences that makes the team successful.  That’s something we know as “diversity” and it’s like annealing of steel to increase its strength.

Unfortunately, there are four  modes of agreement and two are team poison.

  1. Yes: We agree! Let’s get to work!
  2. No: We disagree! Let’s figure out what’s different so that we’re stronger!
  3. Artificial Warfare:  We disagree!  While we are fundamentally aligned, everyone else thinks that the team does not have consensus and ignores the teams decisions.  We also waste a lot of time talking instead of acting.
  4. Artificial Harmony: We agree!  But then we don’t support each other in getting the work done or message alignment.  We never spend time talking about the real issues so we constantly have to redo our actions.

I’ve never seen a team that is as simple as agree/disagree but I’ve been at companies (Surgient) that tried to build a culture to support trust and conflict resolution (based on Lencioni’s excellent 5 dysfunctions book).  However, there’s a major gap between a team that needs to build trust through healthy conflict and one that wraps itself in the dysfunctions of artificial harmony and warfare.

If you find yourself on a team with this problem then you’ll need management by-in to fix it.  I have not seen it be a self-correcting problem.  I’d love to hear if you’ve gotten yourself healthy from a team with these issues.

Signs of artificial agreement syndrome include

  1. Lack of broad participation – discussions are dominated by a few voices
  2. Discussions that always seem to run to the meta topic instead of the actual problem
  3. Issues are not resolved and come up over and over
  4. People are still upset after the meeting because issues have not been resolved
  5. People have different versions of events
  6. Lack of trust for some people to speak for the group
  7. Outcomes of decision making meetings are surprises
  8. Lack of results or missed commitments by the team

Lean Process’ strength is being Honest and Humble

Lean process and methodology is important to me because I think it is central to the work that we are doing in the community.  Even more, it’s changing how my team at Dell creates and delivers products for customers?
This post may be long, but my answer to “why Lean” ends up being very simple: Lean process is honest and humble.
I believe Lean process is more honest because it assumes a lack of knowledge.  It’s more “truthy” to admit there are a lot of things that we don’t know (we can’t know!) until we’ve started doing the work.  It’s very hard to admit we don’t have answers for things until we are further along because we want to feel like  experts and we to lock deliveries.
The “building software is like building a house” analogy is often used to claim that Lean lacks the design “blue prints” that other processes have.  The argument goes that builders needed to understand how the entire house works with structural support, plumbing lines and electrical circuits and things like that.  However, if I was going to build a house I would still leave a lot of things to the last-minute.  The process of building a house evolves so that the basic outlines of structural elements are known. In a lot of cases the position of rooms, the outlets, air conditioning ducts, a lot of the functional components, even windows and doors while they are often placed in the design can easily be moved and changed as you go through.  You can do a walk-through of a house after it’s been framed out and make all sorts of changes and adjustments.  As things go forward in the design of the house things become more and more difficult to change. You are building a brick façade, moving the windows within the façade are very difficult. However, interior places they aren’t.  Likewise, I don’t want to order my life-gem counter-tops from the blue-prints – it’s much safer to order off actual measurements.
Software projects are also building projects. You build a façade, you build a structure and within that structure you have a lot of flexibility. As you go you make more decisions and your choices become more limited. But, that is the nature of building.  For that reason, saying “we don’t know everything we want” is not just good practice, it is much more honest.
But honesty is not enough for a strong Lean process.  The need for humility in Lean architects and business people really stands out.  The Lean process is humble because it starts with the assumption that we don’t really understand the value, drivers, interests and features that make our product special.
We need very strong ideas and a vision; however, we need to be motivated by making something that is significant to other people.  They are the ones who give it value.
We have to give up the idea that we can convince someone who our idea will be significant to them – we have to show and collaborate instead.  The most important thing in building any project and taking any product to market is listening to the people who are using your product and understanding what their needs.  Instead of telling them what they need;  show them something interesting, interact with them and get their opinion.
Contrast that to waterfall methodology where the assumption is that we can put smart people in a room, have them figure out what the requirements are, build a team, get everything ready to go and then start executing.  That assumption is highly optimized and seems very efficient, but it has a huge amount of hubris in the process.  The idea that we can sit down two years in advance of market need and identify what those features and capabilities are seems outrageous to me in the current technology market.  It is so much harder to try to get that information correct and then execute on it that get a directional statement and begin and then get feedback and interact, it is a world of difference between the two processes.
Ultimately, Lean process about having requirements that are less defined or well-known.  It’s driven by giving respect to the people consuming the product.  We can hear their ideas and their reactions.  Where the users’ input can be evaluated and taken in to account.  It’s about collaboration.
Humility it not just about listening and collecting feedback: it is about interacting and building relationships.
So just as our customers are building a relationship with our product, they are also building a relationship with the people creating that product. And that relationship is what drives the product forward and what makes it a great product and it is what gives you a strong and loyal customer base, rather than dictating, “This is what you wanted. Here it is. I hope you enjoy it.”
This is a completely different and powerful way of delivering product.  I believe that honesty and humility in a Lean process inherently creates stronger products and ones that are both faster delivered and better suited to their markets.

double Block Head with OpenStack+Equallogic & Crowbar+Ceph

Block Head

Whew….Yesterday, Dell announced TWO OpenStack block storage capabilities (Equallogic & Ceph) for our OpenStack Essex Solution (I’m on the Dell OpenStack/Crowbar team) and community edition.  The addition of block storage effectively fills the “persistent storage” gap in the solution.  I’m quadrupally excited because we now have:

  1. both open source (Ceph) and enterprise (Equallogic) choices
  2. both Nova drivers’ code is in the open at part of our open source Crowbar work

Frankly, I’ve been having trouble sitting on the news until Dell World because both features have been available in Github before the announcement (EQLX and Ceph-Barclamp).  Such is the emerging intersection of corporate marketing and open source.

As you may expect, we are delivering them through Crowbar; however, we’ve already had customers pickup the EQLX code and apply it without Crowbar.

The Equallogic+Nova Connector

block-eqlx

If you are using Crowbar 1.5 (Essex 2) then you already have the code!  Of course, you still need to have the admin information for your SAN – we did not automate the configuration of the storage system, but the Nova Volume integration.

We have it under a split test so you need to do the following to enable the configuration options:

  1. Install OpenStack as normal
  2. Create the Nova proposal
  3. Enter “Raw” Attribute Mode
  4. Change the “volume_type” to “eqlx”
  5. Save
  6. The Equallogic options should be available in the custom attribute editor!  (of course, you can edit in raw mode too)

Want Docs?  Got them!  Check out these > EQLX Driver Install Addendum

Usage note: the integration uses SSH sessions.  It has been performance tested but not been tested at scale.

The Ceph+Nova Connector

block-ceph

The Ceph capability includes a Ceph barclamp!  That means that all the work to setup and configure Ceph is done automatically done by Crowbar.  Even better, their Nova barclamp (Ceph provides it from their site) will automatically find the Ceph proposal and link the components together!

Ceph has provided excellent directions and videos to support this install.

The Atlantic magazine explains why Lean process rocks (and saves companies $$)

GearsI’m certain that the Atlantic‘s Charles Fishman was not thinking software and DevOps when he wrote the excellent article about “The Insourcing Boom.”  However, I strongly recommend reading this report for anyone who is interested in a practical example of the inefficiencies of software lean process (If you are impatient, jump to page 2 and search for toaster).

It’s important to realize that this article is not about software! It’s an article about industrial manufacturing and the impact that lean process has when you are making stuff.  It’s about how US companies are using Lean to make domestic plants more profitable than Asian ones.  It turns out that how you make something really matters – you can’t really optimize the system if you treat major parts like a black box.

When I talk about Agile and Lean, I am talking about proven processes being applied broadly to companies that want to make profit selling stuff. That’s what this article is about

If you are making software then you are making stuff! Your install and deploy process is your assembly line. Your unreleased code is your inventory.

This article does a good job explaining the benefits of being close to your manufacturing (DevOps) and being flexible in deployment (Agile) and being connected to customers (Lean).  The software industry often acts like it’s inventing everything from scratch. When it comes to manufacturing processes, we can learn a lot from industry.

Unlike software, industry has real costs for scrap and lost inventory. Instead of thinking “old school” perhaps we should be thinking of it as the school of hard knocks.

My Dilemma with Folsom – why I want to jump to G

When your ship sailsThese views are my own.  Based on 1×1 discussions I’ve had in the OpenStack community, I am not alone.

If you’ve read my blog then you know I am a vocal and active supporter of OpenStack who is seeking re-election to the OpenStack Board.  I’m personally and professionally committed to the project’s success. And, I’m confident that OpenStack’s collaborative community approach is out innovating other clouds.

A vibrant project requires that we reflect honestly: we have an equal measure of challenges: shadow free fall Dev, API vs implementation, forking risk and others.  As someone helping users deploy OpenStack today, I find my self straddling between a solid release (Essex) and a innovative one (Grizzly). Frankly, I’m finding it very difficult to focus on Folsom.

Grizzly excites me and clearly I’m not alone.  Based on pace of development, I believe we saw a significant developer migration during feature freeze free fall.

In Grizzly, both Cinder and Quantum will have progressed to a point where they are ready for mainstream consumption. That means that OpenStack will have achieved the cloud API trifecta of compute-store-network.

  • Cinder will get beyond the “replace Nova Volume” feature set and expands the list of connectors.
  • Quantum will get to parity with Nova Network, addresses overlapping VM IPs and goes beyond L2 with L3 feature enablement like  load balancing aaS.
  • We are having a real dialog about upgrades while the code is still in progress
  • And new projects like Celio and Heat are poised to address real use problems in billing and application formation.

Everything I hear about Folsom deployment is positive with stable code and significant improvements; however, we’re too late to really influence operability at the code level because the Folsom release is done.  This is not a new dilemma.  As operators, we seem to be forever chasing the tail of the release.

The perpetual cycle of implementing deployment after release is futile, exhausting and demoralizing because we finish just in time for the spotlight to shift to the next release.

I don’t want to slow the pace of releases.  In Agile/Lean, we believe that if something is hard then we do should it more.  Instead, I am looking at Grizzly and seeing an opportunity to break the cycle.  I am looking at Folsom and thinking that most people will be OK with Essex for a little longer.

Maybe I’m a dreamer, but if we can close the deployment time gap then we accelerate adoption, innovation and happy hour.  If that means jilting Folsom at the release altar to elope with Grizzly then I can live with that.

Interview Transcript about OpenStack, Foundation, Dell & Crowbar

Rafael Knuth, Dell Rafael Knuth (@RafaelKnuth with Dell as EMEA Social Media Community Manager) has set an ambitious objective to interview all 24 of the OpenStack Foundation Board members.  I have the privilege of being the first interview posted (Japanese version!).   Here’s the whole series.

This is not puff interview – We spent an hour together and Rafael did not shy away from asking hard questions like “Why did Dell jump into OpenStack?” and “is VMware a threat to OpenStack?”  Rather than posting the whole transcript (it’s posted here), I’m including the questions (as a teaser) below.   There is some real meat in these answers about OpenStack, Dell, Crowbar and challenges facing the project.

WARNING: My job is engineering, not marketing.  You may find my answers (which are MY OWN ANSWERS) to be more direct that you are expecting.  If you find yourself needing additional circumlocution then simply close your browser and move on.

Some highlights…

Dell’s interest in OpenStack has been very pragmatic. OpenStack is something we really see a market need for.

Rackspace …  runs on OpenStack pretty much off trunk …  That’s exactly the type of vibrant community we want to see.  At the same time, there is a growing community that wants to use OpenStack distributions with support, certifications and they are fine with being 6 months behind OpenStack off trunk. That’s good, and we want that shadow, we want that combination of pure minded early adopters and less sophisticated OpenStack users both working together.

We are working with different partners to bring OpenStack to different customers in different ways. It is confusing. Your question about Dell Crowbar was right … it is targeted at a certain class of users, and I don’t want enterprise customers who expect a lot of shiny chrome and zero touch. That’s not the target by now for Dell Crowbar. We definitely need that sort of magic decoder page to help customers understand our commercial offering.

Questions from Interview:

  1. Dell is one of the very early contributors to OpenStack. Why is Dell engaging in this project?
  2. How does Dell contribute to OpenStack?
  3. Let’s talk a bit about Dell Crowbar, your team’s deployment mechanism for OpenStack.
  4. Let’s talk a bit about OpenStack raw vs. OpenStack distributions.
  5. What are the biggest barriers to OpenStack adoption as of now?
  6. What does a customer specifically need to do when moving from OpenStack Essex to Folsom for example?
  7. My next question is around proof of concept versus production, Rob. How are customers using OpenStack and can you give examples for both scenarios?
  8. I hear very often two different statements: “Open Stack is an alternative to Amazon.” The other is: “OpenStack is an alternative to VMware … maybe, hopefully in two or three years from now.”  Which of both statements is true?
  9. How do you view VMware joining OpenStack. Is it a threat to OpenStack or does VMware add value to the project?
  10. Let us speak about market adoption. Who are the early adopters of OpenStack? And when do you expect OpenStack to hit the tipping point for mass market adoption?
  11. Rob, for all those interested in Dell’s commercial offering around OpenStack … can you give a brief overview?
  12. Dell TechCenter that provides customers an overview over our OpenStack offering: Dell Crowbar as our DevOps tool in its various shapes and forms, OpenStack distros we support … cloud services we build around OpenStack … hardware capabilities optimized for OpenStack.
  13. What are the challenges for the OpenStack Board of Directors?