Category Archives: OpenStack

DevOps approaches to upgrade: Cube Visualization

Posted on by
14

I’m working on my OpenStack summit talk about DevOps upgrade patterns and got to a point where there are three major vectors to consider:

  1. Step Size (shown as X axis): do we make upgrades in small frequent steps or queue up changes into larger bundles? Larger steps mean that there are more changes to be accommodated simultaneously.
  2. Change Leader (shown as Y axis): do we upgrade the server or the client first? Regardless of the choice, the followers should be able to handle multiple protocol versions if we are going to have any hope of a reasonable upgrade.
  3. Safeness (shown as Z axis): do the changes preserve the data and productivity of the entity being upgraded? It is simpler to assume to we simply add new components and remove old components; this approach carries significant risks or redundancy requirements.

I’m strongly biased towards continuous deployment because I think it reduces risk and increases agility; however, I laying out all the vertices of the upgrade cube help to visualize where the costs and risks are being added into the traditional upgrade models.

Breaking down each vertex:

  1. Continuous Deploy – core infrastructure is updated on a regular (usually daily or faster) basis
  2. Protocol Driven – like changing to HTML5, the clients are tolerant to multiple protocols and changes take a long time to roll out
  3. Staged Upgrade – tightly coordinate migration between major versions over a short period of time in which all of the components in the system step from one version to the next together.
  4. Rolling Upgrade – system operates a small band of versions simultaneously where the components with the oldest versions are in process of being removed and their capacity replaced with new nodes using the latest versions.
  5. Parallel Operation – two server systems operate and clients choose when to migrate to the latest version.
  6. Protocol Stepping – rollout of clients that support multiple versions and then upgrade the server infrastructure only after all clients have achieved can support both versions.
  7. Forced Client Migration – change the server infrastructure and then force the clients to upgrade before they can reconnect.
  8. Big Bang – you have to shut down all components of the system to upgrade it

This type of visualization helps me identify costs and options. It’s not likely to get much time in the final presentation so I’m hoping to hear in advance if it resonates with others.

PS: like this visualization? check out my “magic 8 cube” for cloud hosting options.

5 things keeping DevOps from playing well with others (Chef, Crowbar and Upstream Patterns)

Posted on by
6

Sharing can be hardSince my earliest days on the OpenStack project, I’ve wanted to break the cycle on black box operations with open ops. With the rise of community driven DevOps platforms like Opscode Chef and Puppetlabs, we’ve reached a point where it’s both practical and imperative to share operational practices in the form of code and tooling.

Being open and collaborating are not the same thing.

It’s a huge win that we can compare OpenStack cookbooks. The real victory comes when multiple deployments use the same trunk instead of forking.

This has been an objective I’ve helped drive for OpenStack (with Matt Ray) and it has been the Crowbar objective from the start and is the keystone of our Crowbar 2 work.

This has proven to be a formidable challenge for several reasons:

  1. diverging DevOps patterns that can be used between private, public, large, small, and other deployments -> solution: attribute injection pattern is promising
  2. tooling gaps prevent operators from leveraging shared deployments -> solution: this is part of Crowbar’s mission
  3. under investing in community supporting features because they are seen as taking away from getting into production -> solution: need leadership and others with join
  4. drift between target versions creates the need for forking even if the cookbooks are fundamentally the same -> solution: pull from source approaches help create distro independent baselines
  5. missing reference architectures interfere with having a stable baseline to deploy against -> solution: agree to a standard, machine consumable RA format like OpenStack Heat.

Unfortunately, these five challenges are tightly coupled and we have to progress on them simultaneously. The tooling and community requires patterns and RAs.

The good news is that we are making real progress.

Judd Maltin (@newgoliath), a Crowbar team member, has documented the emerging Attribute Injection practice that Crowbar has been leading. That practice has been refined in the open by ATT and Rackspace. It is forming the foundation of the OpenStack cookbooks.

Understanding, discussing and supporting that pattern is an important step toward accelerating open operations. Please engage with us as we make the investments for open operations and help us implement the pattern.

OpenStack Board Voting Starts! Three thoughts about the board and election

Posted on by
3

1/18 Note: I was re-elected! Thanks everyone for your support.

OpenStack Foundation members, I have three requests for you:

  1. vote (ballots went out by email already and expire on Friday)
  2. vote for me (details below)
  3. pick the board you want, not just the candidates.

And, of course, vote according to our code of conduct: “Members should not attempt to manipulate election results. Open debate is welcome, but vote trading, ballot stuffing and other forms of abuse are not acceptable.”

To vote, you must have been a member for 6 months. Going Mad Panda? Join right now so you can vote next time!

Even if you’re not a voter, you may find my comments useful in understanding OpenStack. Remember, these positions are my own: they do not reflect those of my employer (Dell).

Vote because it’s the strongest voice the community has about OpenStack governance

I’d strongly encourage you to commit code, come to the summit and participate in the lists and chats; however, voting is more. It shows measurable impact. It shows the vitality of the community.

OpenStack is exciting to me because it’s community driven.

I have been a strong and early leader in the community:

Vote for me because of my OpenStack Operations focus

My team at Dell has been laser focused on making OpenStack operable for over 2 years. My team acts like a lean start-up inside of Dell to get products out to market early.

We’ve worked with early operators and ground-breaking ecosystem partners (plus). We’ve helped many people get OpenStack running for real workloads. Consequently, I am highly accessible and accountable to a large number of users, operators and ecosystem vendors. Few other people in the OpenStack community have my breadth of exposure to real OpenStack deployments. My team and I have been dedicated, active participants in OpenStack long before Dell’s grand OpenStack strategy coalesced.

More importantly, I am committed to open source and open operations. The Apache 2 open source Crowbar project established the baseline (and served as the foundation) for other OpenStack deployment efforts. We don’t just talk about open source, we do open source: our team works in the open (my git account).

Vote for a board because diversity of views is important

OpenStack voting allows you to throw up to 8 votes to a single candidate. While you can put all your eggs into a single basket, I recommend considering a broader slate in voting.

We have an impressive and dedicated list of candidates who will do work for the community. I ask that you consider the board as a team. If you want operators, users, diversity, change, less corporate influence or any flavor of the above then think not just about the individual.

Why should I be part of your slate? I am not a blind cheerleader. I have voiced and driven community-focused positions about the board (dev cycle, grizzly >;; folsom, consensus).

One of my core activities has been to work on addressing community concerns about affinity voting. Gold and Platinum members have little incentive to address this issue. While I’d like to see faster action, it requires changing the by-laws. Any by-law changes need to be made carefully and they also require a vote of the electorate. If you want positive change, you need board members, like me, who are persistent and knowledgeable in board dynamics.

What have I done on the Board? Quite a bit…

It would be easy to get lost in a board of 24 members, but I have not. Armed with my previous board experience, I have been a vocal advocate for the community in board meetings without creating disruption or pulling us off topic.

Why re-elect me? For community & continuity

I am a strong and active board member and I work hard to represent all OpenStack constituents: developers, operators, users and ecosystem vendors.

Working on a board is a long-term exercise. Positions and actions I’ve taken today may take months to come to fruition. I would like the opportunity continue that work and see it through.

OpenStack Board Interview Series by Rafael Knuth

Posted on by
2

Rafael Knuth, Dell Rafael Knuth (@RafaelKnuth with Dell as EMEA Social Media Community Manager) has set an ambitious objective to interview all 24 of the OpenStack Foundation Board members. I had the privilege of being the first interview posted (Japanese version!).

Here’s the series so far:

  1. Rob Hirschfeld (me!) of Dell (my summary)
  2. Boris Renski of Mirantis
  3. Randy Bias of CloudScaling
  4. Jim Curry of Rackspace
  5. Yujie Due of 99cloud
  6. Tim Bell of CERN
  7. Joshua McKenty of Piston
  8. Lew Tucker of Cisco
  9. Alan Clark (Chair) of SUSE  
  10. Joseph George of Dell (added 5/22)

These are excellent interviews and worth reviewing. They give balanced (often critical) views of OpenStack, its progress and challenges.

I update the list as he adds them.

double Block Head with OpenStack+Equallogic & Crowbar+Ceph

Posted on by
9

Block Head

Whew….Yesterday, Dell announced TWO OpenStack block storage capabilities (Equallogic & Ceph) for our OpenStack Essex Solution (I’m on the Dell OpenStack/Crowbar team) and community edition.  The addition of block storage effectively fills the “persistent storage” gap in the solution.  I’m quadrupally excited because we now have:

  1. both open source (Ceph) and enterprise (Equallogic) choices
  2. both Nova drivers’ code is in the open at part of our open source Crowbar work

Frankly, I’ve been having trouble sitting on the news until Dell World because both features have been available in Github before the announcement (EQLX and Ceph-Barclamp).  Such is the emerging intersection of corporate marketing and open source.

As you may expect, we are delivering them through Crowbar; however, we’ve already had customers pickup the EQLX code and apply it without Crowbar.

The Equallogic+Nova Connector

block-eqlx

If you are using Crowbar 1.5 (Essex 2) then you already have the code!  Of course, you still need to have the admin information for your SAN – we did not automate the configuration of the storage system, but the Nova Volume integration.

We have it under a split test so you need to do the following to enable the configuration options:

  1. Install OpenStack as normal
  2. Create the Nova proposal
  3. Enter “Raw” Attribute Mode
  4. Change the “volume_type” to “eqlx”
  5. Save
  6. The Equallogic options should be available in the custom attribute editor!  (of course, you can edit in raw mode too)

Want Docs?  Got them!  Check out these > EQLX Driver Install Addendum

Usage note: the integration uses SSH sessions.  It has been performance tested but not been tested at scale.

The Ceph+Nova Connector

block-ceph

The Ceph capability includes a Ceph barclamp!  That means that all the work to setup and configure Ceph is done automatically done by Crowbar.  Even better, their Nova barclamp (Ceph provides it from their site) will automatically find the Ceph proposal and link the components together!

Ceph has provided excellent directions and videos to support this install.

My Dilemma with Folsom – why I want to jump to G

Posted on by
13

When your ship sailsThese views are my own.  Based on 1×1 discussions I’ve had in the OpenStack community, I am not alone.

If you’ve read my blog then you know I am a vocal and active supporter of OpenStack who is seeking re-election to the OpenStack Board.  I’m personally and professionally committed to the project’s success. And, I’m confident that OpenStack’s collaborative community approach is out innovating other clouds.

A vibrant project requires that we reflect honestly: we have an equal measure of challenges: shadow free fall Dev, API vs implementation, forking risk and others.  As someone helping users deploy OpenStack today, I find my self straddling between a solid release (Essex) and a innovative one (Grizzly). Frankly, I’m finding it very difficult to focus on Folsom.

Grizzly excites me and clearly I’m not alone.  Based on pace of development, I believe we saw a significant developer migration during feature freeze free fall.

In Grizzly, both Cinder and Quantum will have progressed to a point where they are ready for mainstream consumption. That means that OpenStack will have achieved the cloud API trifecta of compute-store-network.

  • Cinder will get beyond the “replace Nova Volume” feature set and expands the list of connectors.
  • Quantum will get to parity with Nova Network, addresses overlapping VM IPs and goes beyond L2 with L3 feature enablement like  load balancing aaS.
  • We are having a real dialog about upgrades while the code is still in progress
  • And new projects like Celio and Heat are poised to address real use problems in billing and application formation.

Everything I hear about Folsom deployment is positive with stable code and significant improvements; however, we’re too late to really influence operability at the code level because the Folsom release is done.  This is not a new dilemma.  As operators, we seem to be forever chasing the tail of the release.

The perpetual cycle of implementing deployment after release is futile, exhausting and demoralizing because we finish just in time for the spotlight to shift to the next release.

I don’t want to slow the pace of releases.  In Agile/Lean, we believe that if something is hard then we do should it more.  Instead, I am looking at Grizzly and seeing an opportunity to break the cycle.  I am looking at Folsom and thinking that most people will be OK with Essex for a little longer.

Maybe I’m a dreamer, but if we can close the deployment time gap then we accelerate adoption, innovation and happy hour.  If that means jilting Folsom at the release altar to elope with Grizzly then I can live with that.

Interview Transcript about OpenStack, Foundation, Dell & Crowbar

Posted on by
6

Rafael Knuth, Dell Rafael Knuth (@RafaelKnuth with Dell as EMEA Social Media Community Manager) has set an ambitious objective to interview all 24 of the OpenStack Foundation Board members.  I have the privilege of being the first interview posted (Japanese version!).   Here’s the whole series.

This is not puff interview – We spent an hour together and Rafael did not shy away from asking hard questions like “Why did Dell jump into OpenStack?” and “is VMware a threat to OpenStack?”  Rather than posting the whole transcript (it’s posted here), I’m including the questions (as a teaser) below.   There is some real meat in these answers about OpenStack, Dell, Crowbar and challenges facing the project.

WARNING: My job is engineering, not marketing.  You may find my answers (which are MY OWN ANSWERS) to be more direct that you are expecting.  If you find yourself needing additional circumlocution then simply close your browser and move on.

Some highlights…

Dell’s interest in OpenStack has been very pragmatic. OpenStack is something we really see a market need for.

Rackspace …  runs on OpenStack pretty much off trunk …  That’s exactly the type of vibrant community we want to see.  At the same time, there is a growing community that wants to use OpenStack distributions with support, certifications and they are fine with being 6 months behind OpenStack off trunk. That’s good, and we want that shadow, we want that combination of pure minded early adopters and less sophisticated OpenStack users both working together.

We are working with different partners to bring OpenStack to different customers in different ways. It is confusing. Your question about Dell Crowbar was right … it is targeted at a certain class of users, and I don’t want enterprise customers who expect a lot of shiny chrome and zero touch. That’s not the target by now for Dell Crowbar. We definitely need that sort of magic decoder page to help customers understand our commercial offering.

Questions from Interview:

  1. Dell is one of the very early contributors to OpenStack. Why is Dell engaging in this project?
  2. How does Dell contribute to OpenStack?
  3. Let’s talk a bit about Dell Crowbar, your team’s deployment mechanism for OpenStack.
  4. Let’s talk a bit about OpenStack raw vs. OpenStack distributions.
  5. What are the biggest barriers to OpenStack adoption as of now?
  6. What does a customer specifically need to do when moving from OpenStack Essex to Folsom for example?
  7. My next question is around proof of concept versus production, Rob. How are customers using OpenStack and can you give examples for both scenarios?
  8. I hear very often two different statements: “Open Stack is an alternative to Amazon.” The other is: “OpenStack is an alternative to VMware … maybe, hopefully in two or three years from now.”  Which of both statements is true?
  9. How do you view VMware joining OpenStack. Is it a threat to OpenStack or does VMware add value to the project?
  10. Let us speak about market adoption. Who are the early adopters of OpenStack? And when do you expect OpenStack to hit the tipping point for mass market adoption?
  11. Rob, for all those interested in Dell’s commercial offering around OpenStack … can you give a brief overview?
  12. Dell TechCenter that provides customers an overview over our OpenStack offering: Dell Crowbar as our DevOps tool in its various shapes and forms, OpenStack distros we support … cloud services we build around OpenStack … hardware capabilities optimized for OpenStack.
  13. What are the challenges for the OpenStack Board of Directors?

Seeking OpenStack Foundation Board Seat for 2013 (please nominate me)

Posted on by
5
I am seeking another term on the OpenStack Foundation Board.  Please consider nominating me for this position.
The following is the profile information I provided as part of the nomination process.  If you are looking for insights into where OpenStack is going then these question (especially the later ones) will be interesting.

Provide Brief Biography of Yourself

I have been involved in Cloud for over 12 years and launch some of the earliest Cloud companies.  My educational background (Duke and LSU) is in computer science and systems engineering (Mechanical/Industrial) with a focus on distributed systems.  I have always found deployment to be vitally important in development – that lead me to found a SaaS start-up in 1999 and had made me a DevOps advocate.  In addition to core cloud technologies, I am an Agile/Lean process evangelist who strongly believes that how you build and deliver is just as important as what you deliver.
Currently, I am a principal engineer at Dell leading our OpenStack Cloud project  and also a founder of the Crowbar project.  In that role, I am in constant contact with OpenStack users, ecosystem developers and vendors world-wide; consequently, I have a very broad perspective on use and technical needs for OpenStack and related Cloud technologies.

What is your relationship to OpenStack, and why is its success important to you? What would you say is your biggest contribution to OpenStack’s success to date?

I have been involved in OpenStack at the earliest stages and was a key influencer in Dell’s decision to be an initial sponsor.  Further, I formulated Dell’s operations/deployment focused Lean strategy that helped create an early focus on OpenStack operations.   This support was a critical catalyst to building market momentum and we continue help drive operations and user focused requirements for OpenStack.  Providing a very open and community driven DevOps focus has been my biggest contribution (see Crowbar).
I also serve the community in many ways.  In addition to being elected to the 2012 Board, I founded the Austin OpenStack User Group (next meeting 12/6!), have been spoken at every Summit, co-Chaired the Operations Track at the Grizzly Summit and create community awareness of OpenStack through my blog, corporate work, and social media activity.
I have invested considerable time in OpenStack and made professional commitments to its success.  It’s not just about software and an awesome community – I am personally invested in OpenStack winning.

Describe your experience with other non profits or serving as a board member. How does your experience prepare you for the role of a board member?

I am currently serving on the OpenStack Foundation Board and have been an active advocate for a collaborative process and open communication (http://wp.me/pF6d2-vc, http://wp.me/pF6d2-w9).  I hope continue on the board so that we can stay focused on the critical issues at hand.

I have over four years of serving as Secretary for my city’s public Texas 4-B Commercial Development Corporation which governed by state open meetings standards.  This work has proven directly relevant to the OpenStack foundation because the open governance requirements map very well to our communities transparency expectations.

What do you see as the Board’s role in OpenStack’s success?

The board’s job is to ensure that community and collaboration remain OpenStack’s core strength as an open source project.  As adoption and footprint increases there is enormous pressure on the community to try and serve both general and very specialized interests.  Most critically, we must find ways to balance competing financial interests within the community.
The board must ensure that OpenStack provides commercial opportunity because I believe that incent continued investment; however, we cannot let profit drive the community away from our open source values.
I believe the Board must monitor the community’s progress to ensure we maintain this balance.  It is our responsibility to make adjustments, influence changes and take responsibility for driving an innovative and collaborative culture.

What do you think the top priority of the Board should be in 2013?

There are two inter-related top priorities for the Board: we must help deliver guidelines for which projects are truly “core” OpenStack and help move towards a certifiable API specification for OpenStack (instead of implementation).   While distinct, I believe both items must be solved together.  These changes are essential to foster innovation and adoption of OpenStack.
While these issues will occupy the Board (and Technical/User committees) for 2013, my personal priority for OpenStack in general remains focused on operators and users.  I believe we have to make substantial progress on upgrades, migration and operational readiness.  These issues continue to create a serious barrier to adoption.

Impressions from OpenStack Foundation board meeting 10/15

Posted on by
1

20121018-115319.jpgI spent the first day of the OpenStack summit doing my service as a Foundation board member. While you need to wait for a summary by Jonathan Bryce, the Foundation Executive Director, and/or the official minutes to get the details of our actions, I think that it’s reasonable to share my impressions.

In some ways, the chemistry of the board is as important as our actions at this point. We are still working thorough start up issues and, more importantly, learning how to work together. The board is not only big, 24 members, it is also has many companies that often compete (power of we).

Here are my top 5 impressions:

    1. Transparency. The bylaws are crafted to ensure a openness and our meetings are broadcast to the maximum extent possible. Even with that background, there is a consistent self-check and discussion about increasing transparency so that the community is included.
    2. Humor. It’s a great sign of progress that we are laughing together because it shows that we trust and respect each other.
    3. Frustration. Recognition that we have important decisions to make and a degree of impatience to make them. Boards are subtle: we spend a lot of time setting up the right structures that allow us to make hard decisions quickly. Also, grumbling that we’d overlapped the board meeting and the summit.
    4. Consensus. This something I committed to help build. I feel that our actions reflect both healthy discussion as individuals and desire to work together as a board.
    5. Leadership. Looking back from our first meeting, it is clear that Jonathan and Alan have fully assumed their roles. I’m also seeing how the Foundation team (Mark, Lauren, Stephano) are smoothly supporting operations even though we’re really just weeks from the Foundation launch.

I hope that gives you some insights into the board meeting and that more of you can join the broadcast for the next ones (2012 schedule TBD) including a joint tech and foundation meeting

Wednesday morning at the summit was “breakfast with the board.” Look for some notes from that too soon.

Open Source is The Power of We (Blog Action Day)

Posted on by
2

This post is part of a world wide “blog action day” where thousands of bloggers post their unique insights about a single theme. For 2012, it’s the “power of we is as a celebration of people working together to make a positive difference in the world, either for their own communities or for people they will never meet half way around the world.”

I’ve choosing open source software because I think that we are establishing models for building ideas collaboratively that can be extended beyond technology into broader use. The way we solve open source challenges translates broadly because we are the tool makers of the global interaction.

I started using open source¹ as a way to solve a problem; I did not understand community or how groups of loosely connected people came together to create something new. Frankly, the whole process of creating free software seemed to be some hybrid combination of ninja coders and hippy hackers. That changed when I got involve on the ground floor of the OpenStack project (of which I am now a Foundation board member).

I was not, could not have been, prepared for the power and reality of community and collaboration that fuels OpenStack and other projects. We have the same problems as any non-profit project except that we are technologists: we can make new tools to solve our teaming and process problems.

It is not just that open source projects solve problems that help people. The idea of OpenStack and Hadoop being used by medical researches to find cures for cancer is important; however, the learning how to build collaboratively is another critical dimension. Our world is getting more connected and interconnected by technology, but the actual tools for social media are only in their earliest stages.

Not only are the tools evolving, the people using the tools are changing too! We are training each other to work together in ways that were beyond our imagine even 10 years ago. It’s the combination of both new technology and new skills that is resetting the rules for collaboration.

Just a few years ago, open source technology was considered low quality, risky and fringe. Today, open source projects like OpenStack and Hadoop are seen as more innovative and equally secure and supportable compared to licensed products. This transformation represents a surprising alignment and collaboration between individuals and entities that would normally be competing. While the motivation for this behavior comes from many sources, we all share the desire to do collaborative effectively.

I don’t think that we have figured out how to really do this the best way yet. We are making progress and getting better and better. We are building tools (like etherpad, wikis, irc, twitter, github, jenkins, etc) that improve collaboration. We are also learning building a culture of collaboration.

Right now, I’m on a train bound for the semi-annual OpenStack summit that brings a world wide audience together for 4½ days of community work. The discussions will require a new degree of openness from people and companies that are normally competitive and secretive about product development. During the summit, we’ll be doing more than designing OpenStack, we will be learning the new skills of working together. Perhaps those are the most important deliverables.

Open source projects combination of both new technologies and new skills creates the Power of We.

——————

PS¹: Open source software is a growing class of applications in which the authors publish the instructions for running the software publicly so that other people can use the software. Sometimes (but not always) this includes a usage license that allows other people to run the software without paying the author royalties. In many cases, the author’s motivation is that other users will help them test, modify and improve the software so that improves more quickly than a single creator could do alone.