Crowbar 2 Status Update > I can feel the rumble of the engines

two

Crowbar Two

While I’ve been more muted on our Crowbar 2 progress since our pivot back to CB1 for Grizzly, it has been going strong and steady.  We took advantage of the extra time to do some real analysis about late-binding, simulated annealing, emergent services and functional operations that are directly reflected in Crowbar’s operational model (yes, I’m working on posted explaining each concept).

We’re planning Crowbar 2 hack-a-thon in Hong Kong before the OpenStack Ice House Summit (11/1-3).  We don’t expect a big crowd on site, but the results will be fun to watch remote and it should be possible to play along (watch the crowbar list for details).

In the mean time, I wanted to pass along this comprehensive status update by Crowbar’s leading committer, Victor Lowther:

It has been a little over a month since my last status report on
Crowbar 2.0, so now that we have hit the next major milestone
(installing the OS on a node and being able to manage it afterwards),
it is time for another status report.

Major changes since the initial status report:

* The Crowbar framework understands node aliveness and availability.
* The Network barclamp is operational, and can manage IPv4 and IPv6 in
  the same network.
* delayed_jobs + a stupidly thin queuing layer handle all our
  long-running tasks.
* We have migrated to postgresql 9.3 for all our database needs.
* DHCP and DNS now utilize the on_node_* role hooks to manage their
  databases.
* We support a 2 layer deployment tree -- system on top, everything
  else in the second layer.
* The provisioner can install Ubuntu 12.04 on other nodes.
* The crowbar framework can manage other nodes that are not in
  Sledgehammer.
* We have a shiny installation wizard now.

In more detail:

Aliveness and availability:

Nodes in the Crowbar framework have two related flags that control
whether the annealer can operate on them.

Aliveness is under the control of the Crowbar framework and
encapsulates the framework's idea of whether any given node is
manageable or not.  If a node is pingable and can be SSH'ed into as
root without a password using the credentials of the root user on
the admin node, then the node is alive, otherwise it is dead.
Aliveness is tested everytime a jig tries to do something on a node
-- if a node cannot be pinged and SSH'ed into from at least one of
its addresses on the admin network, it will be marked as
dead.  When a node is marked as dead, all of the noderoles on that
node will be set to either blocked or todo (depending on the state of
their parent noderoles), and those changes will ripple down the
noderole dependency graph to any child noderoles.

Nodes will also mark themselves as alive and dead in the course of
their startup and shutdown routines.

Availability is under the control of the Crowbar cluster
administrators, and should be used by them to tell Crowbar that it
should stop managing noderoles on the node.  When a node is not
available, the annealer will not try to perform any jig runs on a
node, but it will leave the state of the noderoles alone.

A node must be both alive and available for the annealer to perform
operations on it.

The Network Barclamp:

The network barclamp is operational, with the following list of
features:

* Everything mentioned in Architecture for the Network Barclamp in
  Crowbar 2.0
* IPv6 support.  You can create ranges and routers for IPv6 addresses
  as well as IPv4 addresses, and you can tell a network that it should
  automatically assign IPv6 addresses to every node on that network by
  setting the v6prefix setting for that network to either:
  * a /64 network prefix, or
  * "auto", which will create a globally unique RFC4193 IPv6 network
    prefix from a randomly-chosen 40 bit number (unique per cluster
    installation) followed by a subnet ID based on the ID of the
    Crowbar network.
  Either way, nodes in a Crowbar network that has a v6prefix will get
  an interface ID that maps back to their FQDN via the last 64 bits of
  the md5sum of that FQDN. For now, the admin network will
  automatically create an RFC4193 IPv6 network if it is not passed a
  v6prefix so that we can easily test all the core Crowbar components
  with IPv6 as well as IPv4.  The DNS barclamp has been updated to
  create the appropriate AAAA records for any IPv6 addresses in the
  admin network.

Delayed Jobs and Queuing:

The Crowbar framework runs all jig actions in the background using
delayed_jobs + a thin queuing layer that ensures that only one task is
running on a node at any given time.  For now, we limit ourselves to
having up to 10 tasks running in the background at any given time,
which should be enough for the immediate future until we come up with
proper tuning guidelines or auto-tuning code for significantly larger
clusters.

Postgresql 9.3:

Migrating to delayed_jobs for all our background processing made it
immediatly obvious that sqlite is not at all suited to handling real
concurrency once we started doing multiple jig runs on different nodes
at a time. Postgresql is more than capable of handling our forseeable
concurrency and HA use cases, and gives us lots of scope for future
optimizations and scalability.

DHCP and DNS:

The roles for DHCP and DNS have been refactored to have seperate
database roles, which are resposible for keeping their respective
server roles up to date.  Theys use the on_node_* roles mentioned in
"Roles, nodes, noderoles, lifeycles, and events, oh my!" along with a
new on_node_change event hook create and destroy DNS and DHCP database
entries, and (in the case of DHCP) to control what enviroment a node
will PXE/UEFI boot into.  This gives us back the abiliy to boot into
something besides Sledgehammer.

Deployment tree:

Until now, the only deployment that Crowbar 2.0 knew about was the
system deployment.  The system deployment, however, cannot be placed
into proposed and therefore cannot be used for anything other than
initial bootstrap and discovery.  To do anything besides
bootstrap the admin node and discover other nodes, we need to create
another deployment to host the additional noderoles needed to allow
other workloads to exist on the cluster.  Right now, you can only
create deployments as shildren of the system deployment, limiting the
deployment tree to being 2 layers deep.

Provisioner Installing Ubuntu 12.04:

Now, we get to the first of tqo big things that were added in the last
week -- the provisioner being able to install Ubuntu 12.04 and bring
the resulting node under management by the rest of the CB 2.0
framework.  This bulds on top of the deployment tree and DHCP/DNS
database role work.  To install Ubuntu 12.04 on a node from the web UI:

1: Create a new deployment, and add the provisioner-os-install role to
that deployment.  In the future you will be able to edit the
deployment role information to change what the default OS for a
deployment should be.
2: Drag one of the non-admin nodes onto the provisioner-os-install
role.  This will create a proposed noderole binding the
provisioner-os-install role to that node, and in the future you would
be able to change what OS would be installed on that node by editing
that noderole before committing the deployment.
3: Commit the snapshot.  This will cause several things to happen:
  * The freshly-bound noderoles will transition to TODO, which will
    trigger an annealer pass on the noderoles.
  * The annealer will grab all the provisioner-os-install roles that
    are in TODO, set them in TRANSITION, and hand them off to
    delayed_jobs via the queuing system.
  * The delayed_jobs handlers will use the script jig to schedule a
    reboot of the nodes for 60 seconds in the future and then return,
    which will transition the noderole to ACTIVE.
  * In the crowbar framework, the provisioner-os-install role has an
    on_active hook which will change the boot environment of the node
    passed to it via the noderole to the appropriate os install state
    for the OS we want to install, and mark the node as not alive so
    that the annealer will ignore the node while it is being
    installed.
  * The provisioner-dhcp-database role has an on_node_change handler
    that watches for changes in the boot environment of a node.  It
    will see the bootenv change, update the provisioner-dhcp-database
    noderoles with the new bootenv for the node, and then enqueue a
    run of all of the provisioner-dhcp-database roles.
  * delayed_jobs will see the enqueued runs, and run them in the order
    they were submitted.  All the runs sholuld happen before the 60
    seconds has elapsed.
  * When the nodes finally reboot, the DHCP databases should have been
    updated and the nodes will boot into the Uubntu OS installer,
    install, and then set their bootenv to local, which will tell the
    provisioner (via the provisioner-dhcp-database on_node_change
    hook) to not PXE boot the node anymore.
  * When the nodes reboot off their freshly-installed hard drive, they
    will mark themselves as alive, and the annealer will rerun all of
    the usual discovery roles.
The semi-astute observer will have noticed some obvious bugs and race
conditions in the above sequence of steps.  These have been left in
place in the interest of expediency and as learning oppourtunities for
others who need to get familiar with the Crowbar codebase.

Installation Wizard:

We have a shiny installation that you can use to finish bootstrapping
your admin node.  To use it, pass the --wizard flag after your FQDN to
/opt/dell/bin/install-crowbar when setting up the admin node, and the
install script will not automatically create an admin network or an
entry for the admin node, and logging into the web UI will let you
customize things before creating the initial admin node entry and
committing the system deployment.  

Once we get closer to releasing CB 2.0, --wizard will become the default.

OpenStack Core Online Forum, Oct 16 13:30 UTC / Oct 22 0100 UTC

Go Online!OpenStack Community, you are invited on an online discussion about OpenStack Core on October 16th at UTC 13:30 (8:30 am US Central) and October 22nd at UTC 0100 (8:00 pm US Central)

At the next OpenStack Foundation Board meeting, we will be setting a timeline for implementing an OpenStack Core Definition process that promotes a clear and implementation driven metric for deciding which projects should be considered “required.”  This is your chance to review and influence the process!

We’ll review the OpenStack Core Definition process (20 minutes) and then open up the channel for discussion using the IRC (#openstack-meeting) & Google Hangout on Air (link posted in IRC).

The forum will be coordinated through the IRC channel for links and questions.

Can’t make it?  The session was recorded > here!

Thinking about how to Implement OpenStack Core Definition

THIS POST IS #10 IN A SERIES ABOUT “WHAT IS CORE.”

Tied UpWe’ve had a number of community discussions (OSCON, SFO & SA-TX) around the process for OpenStack Core definition.  These have been animated and engaged discussions (video from SA-TX): my notes for them are below.

While the current thinking of a testing-based definition of Core adds pressure on expanding our test suite, it seems to pass the community’s fairness checks.

Overall, the discussions lead me to believe that we’re on the right track because the discussions jump from process to impacts.  It’s not too late!  We’re continuing to get community feedback.  So what’s next?

First…. Get involved: Upcoming Community Core Discussions

These discussions are expected to have online access via Google Hangout.  Watch Twitter when the event starts for a link.

Want to to discuss this in your meetup? Reach out to me or someone on the Board and we’ll be happy to find a way to connect with your local community!

What’s Next?  Implementation!

So far, the Core discussion has been about defining the process that we’ll use to determine what is core.  Assuming we move forward, the next step is to implement that process by selecting which tests are “must pass.”  That means we have to both figure out how to pick the tests and do the actual work of picking them.  I suspect we’ll also find testing gaps that will have developers scrambling in Ice House.

Here’s the possible (aggressive) timeline for implementation:

  • November: Approval of approach & timeline at next Board Meeting
  • January: Publish Timeline for Roll out (ideally, have usable definition for Havana)
  • March: Identify Havana must pass Tests (process to be determined)
  • April: Integration w/ OpenStack Foundation infrastructure

Obviously, there are a lot of details to work out!  I expect that we’ll have an interim process to select must-pass tests before we can have a full community driven methodology.

Notes from Previous Discussions (earlier notes):

  • There is still confusion around the idea that OpenStack Core requires using some of the project code.  This requirement helps ensure that people claiming to be OpenStack core have a reason to contribute, not just replicate the APIs.
  • It’s easy to overlook that we’re trying to define a process for defining core, not core itself.  We have spent a lot of time testing how individual projects may be effected based on possible outcomes.  In the end, we’ll need actual data.
  • There are some clear anti-goals in the process that we are not ready to discuss but will clearly going to become issues quickly.  They are:
    • Using the OpenStack name for projects that pass the API tests but don’t implement any OpenStack code.  (e.g.: an OpenStack Compatible mark)
    • Having speciality testing sets for flavors of OpenStack that are different than core.  (e.g.: OpenStack for Hosters, OpenStack Private Cloud, etc)
  • We need to be prepared that the list of “must pass” tests identifies a smaller core than is currently defined.  It’s possible that some projects will no longer be “core”
  • The idea that we’re going to use real data to recommend tests as must-pass is positive; however, the time it takes to collect the data may be frustrating.
  • People love to lobby for their favorite projects.  Gaps in testing may create problems.
  • We are about to put a lot of pressure on the testing efforts and that will require more investment and leadership from the Foundation.
  • Some people are not comfortable with self-reporting test compliance.   Overall, market pressure was considered enough to punish cheaters.
  • There is a perceived risk of confusion as we migrate between versions.  OpenStack Core for Havana seems to specific but there is concern that vendors may pass in one release and then skip re-certification.  Once again, market pressure seems to be an adequate answer.
  • It’s not clear if a project with only 1 must-pass test is a core project.  Likely, it would be considered core.  Ultimately, people seem to expect that the tests will define core instead of the project boundary.

What do you think?  I’d like to hear your opinions on this!

Taking OpenStack Core discussions to community

core flowTHIS POST IS #9 IN A SERIES ABOUT “WHAT IS CORE.”

We’ve been building up to a broad discussion about the OpenStack Core and I’d like to invite everyone in the OpenStack community to participate (review latest).

Alan Clark (Board Chairman) officially kicked off this open discussion with his post on the OpenStack blog last week.  And we’re trying to have face-to-face events for dialog like the Core meetup tonight in San Francisco.  Look for more to come!

Of course, this will also be a topic at the summit (Alan and I submitted two sessions about this).  The Board needs to move this forward in the November meeting, so NOW is the time to review and give us input.

refined: 10 OpenStack Core Positions

core flowTHIS POST IS #8 IN A SERIES ABOUT “WHAT IS CORE.”

Last week, I posted a streamlined visual of the core discussion that distilled the 12 positions into 10.  Here are reordered and cleaned up matching positions.  This should make it much easier to understand the context.

Note 11/3: The Core Definition is now maintained on the OpenStack Wiki.  This list may not reflect the latest changes.
  1. Implementations that are Core can use OpenStack trademark (OpenStack™)

    1. This is the legal definition of “core” and the  why it matters to  the community.

    2. We want to make sure that the OpenStack™ mark means something.

    3. The OpenStack™ mark is not the same as the OpenStack brand; however, the Board uses it’s control of the mark as a proxy to help manage the brand.

  2. Core is a subset of the whole project

    1. The OpenStack project is supposed to be a broad and diverse community with new projects entering incubation and new implementations being constantly added.  This innovation is vital to OpenStack but separate from the definition of Core.

    2. There may be other marks that are managed separately by the foundation, and available for the platform ecosystem as per the Board’s discretion

    3. “OpenStack API Compatible ” mark not part of this discussion and should be not be assumed.

  3. Core definition can be applied equally to all usage models

    1. There should not be multiple definitions of OpenStack depending on the operator (public, private, community, etc)

    2. While expected that each deployment is identical, the differences must be quantifiable

  4. Claiming OpenStack requiring use of designated upstream code

    1. Implementation’s claiming the OpenStack™ mark must use the OpenStack upstream code (or be using code submitted to upstream)

    2. You are not OpenStack, if you pass all the tests but do not use the API framework

    3. This prevents people from using the API without joining the community

    4. This also surfaces bit-rot in alternate implementations to the larger community

    5. This behavior improves interoperability because there is more shared code between implementation

  5. Projects must have an open reference implementation

    1. OpenStack will require an open source reference base plug-in implementation for projects (if not part of OpenStack, license model for reference plug-in must be compatible).

    2. Definition of a plug-in: alternate backend implementations with a common API framework that uses common _code_ to implement the API

    3. This expects that projects (where technically feasible) are expected to implement a plug-in or extension architecture.

    4. This is already in place for several projects and addresses around ecosystem support, enabling innovation

    5. Reference plug-ins are, by definition, the complete capability set.  It is not acceptable to have “core” features that are not functional in the reference plug-in

    6. This will enable alternate implementations to offer innovative or differentiated features without forcing changes to the reference plug-in implementation

    7. This will enable the reference to expand without forcing other  alternate implementations to match all features and recertify

  6. Vendors may substitute alternate implementations

    1. If a vendor plug-in passes all relevant tests then it can be considered a full substitute for the reference plug-in

    2. If a vendor plug-in does NOT pass all relevant test then the vendor is required to include the open source reference in the implementation.

    3. Alternate implementations may pass any tests that make sense

    4. Alternate implementations should add tests to validate new functionality.

    5. They must have all the must-pass tests (see #10) to claim the OpenStack mark.

  7. OpenStack Implementations are verified by open community tests

    1. Vendor OpenStack implementations must achieve 100% of must-have coverage?

    2. Implemented tests can be flagged as may-have requires list  [Joshua McKenty]

    3. Certifiers will be required to disclose their testing gaps.

    4. This will put a lot of pressure on the Tempest project

    5. Maintenance of the testing suite to become a core Foundation responsibility.  This may require additional resources

    6. Implementations and products are allowed to have variation based on publication of compatibility

    7. Consumers must have a way to determine how the system is different from reference (posted, discovered, etc)

    8. Testing must respond in an appropriate way on BOTH pass and fail (the wrong return rejects the entire suite)

  8. Tests can be remotely or self-administered

    1. Plug-in certification is driven by Tempest self-certification model

    2. Self-certifiers are required to publish their results

    3. Self-certified are required to publish enough information that a 3rd party could build the reference implementation to pass the tests.

    4. Self-certified must include the operating systems that have been certified

    5. It is preferred for self-certified implementation to reference an OpenStack reference architecture “flavor” instead of defining their own reference.  (a way to publish and agree on flavors is needed)

    6. The Foundation needs to define a mechanism of dispute resolution. (A trust but verify model)

    7. As an ecosystem partner, you have a need to make a “works against OpenStack” statement that is supportable

    8. API consumer can claim working against the OpenStack API if it works against any implementation passing all the “must have” tests(YES)

    9. API consumers can state they are working against the OpenStack API with some “may have” items as requirements

    10. API consumers are expected to write tests that validate their required behaviors (submitted as “may have” tests)

  9. A subset of tests are chosen by the Foundation as “must-pass”

    1. An OpenStack body will recommend which tests are elevated from may-have to must-have

    2. The selection of “must-pass” tests should be based on quantifiable information when possible.

    3. Must-pass tests should be selected from the existing body of “may-pass” tests.  This encourages people to write tests for cases they want supported.

    4. We will have a process by which tests are elevated from may to must lists

    5. Potentially: the User Committee will nominate tests that elevated to the board

  10. OpenStack Core means passing all “must-pass” tests

    1. The OpenStack board owns the responsibility to define ‘core’ – to approve ‘musts’

    2. We are NOT defining which items are on the list in this effort, just making the position that it is how we will define core

    3. May-have tests include items in the integrated release, but which are not core.

    4. Must haves – must comply with the Core criteria defined from the IncUp committee results

    5. Projects in Incubation or pre-Incubation are not to be included in the ‘may’ list

Visualizing the OpenStack Core discussion points

THIS POST IS #6 IN A SERIES ABOUT “WHAT IS CORE.”

As we take the OpenStack Core discussion to a larger audience, I was asked to create the summary version the discussion points.  We needed a quick visual way to understand how these consensus statements interconnect and help provide context.  To address this need, I based it on a refined 10 core positions to create the following OpenStack Core flowchart.

core flow

The flow diagram below is grouped into three main areas: core definition (green), technical requirements (blue), and testing impacts (orange).

  1. Core Definition (green) walks through the fundamental scope and premise of the “what is core” discussion.  We are looking for the essential OpenStack: the parts that everyone needs and nothing more.  While OpenStack can be something much larger, core lives at the heart of the use-case venn diagram.  It’s the magical ice cream flavor that everyone loves like Triple Unicorn Rainbow Crunch.
  2. Technical Requirements (blue) covers some of the most contentious parts of the dialog.  This section states the expectation that OpenStack™ implementations must use parts the OpenStack code (you can’t just provide a compatible API).  It goes further to expect that we will maintain an open reference implementation and also identify places where parts of the code can be substituted with alternate implementations.  Examples of alternate implementations are plug-ins, API extensions, different hypervisors, and alternate libraries.
  3. Testing Impacts (orange) reviews some of the important new thinking around Core.  These points focus on the use of OpenStack community tests (e.g.: Tempest) to validate the total code base.  We expect users to be able to self-administer these tests or rely on an external validation.  Either way, we do not expect all tests to pass for all configurations; instead, the Foundation will identify a subset of the tests as required or must-pass.  The current thinking is that these must-pass tests will become the effective definition of OpenStack™ Core.

I hope this helps connect the dots on the core discussions so far.

I’d like to clean-up the positions to match the flow chart and cross reference.  Stay tuned!  This flowchart is a work in process – updates and suggestions are welcome!

READ POST IS #7: WHERE IS THIS GOING?

OpenStack leaders learning by humility, doing and being good partners

With the next OpenStack Board meeting on Thursday (5/30/13 agenda) and Mark McLoughlin’s notes crossing my desk, I was reminded of still open discussion topics around OpenStack leadership.  Reminder: except for executive sessions, OpenStack Board Meetings are open (check agenda for details).

2013-03-11_20-01-50_458

Many of the people and companies involved in OpenStack are new to open source projects. Before OpenStack, I had no direct experience building a community like we’ve built together with OpenStack or I’ve been leading with Crowbar. There is no Collaborative Open Source Communities for Dummies book (I looked).

I am not holding myself, OpenStack or Crowbar up as shining examples of open source perfection. Just the opposite, we’ve had to learn the hard way about what works and what fails. I attribute our successes to humility to accept feedback and willingness to ask for help.

But being successful in the small (like during OpenStack Cactus) is different than where we are heading.  In the small, everyone was an open source enthusiast and eager collaborator.  In the large, we should be asking the question “how will we teach people to join and build an open source community?”

The answer is that collaboration must be modeled by the OpenStack leadership.

At the Summit, I was talking with fellow board director Sean Roberts (Yahoo!) and I think he made this point very simply:

“Being in open source is a partnership. If you don’t bring something to the partnership then you’re a user not a partner. We love users but we need to acknowledge the difference.” (Sean Roberts, OpenStack Director)

OpenStack will succeed by building a large base of users; consequently, we need our leaders to be partners in the community.

Thanks! I’m enjoying my conversation with you

I write because I love to tell stories and to think about how actions we take today will impact tomorrow.  Ultimately, everything here is about a dialog with you because you are my sounding board and my critic.  I appreciate when people engage me about posts here and extend the conversation into other dimensions.  Feel free to call me on points and question my position – that’s what this is all about.

Thank you for being at part of my blog and joining in.  I’m looking forward to hearing more from you.

During the OpenStack Summit, I got to lead and participate in some excellent presentations and panels.  While my theme for this summit was interoperability, there are many other items discussed.

I hope you enjoy them.

Did one of these topics stand out?  Is there something I missed?  Please let me know!

We need better Gold Member criteria to help building OpenStack culture

bunny slippersDuring last OpenStack board meeting, we started a dialog that will be continued over the rest of the year.  It concerns how/if we should apply our criteria to measure the contributions of companies that are applying to become Gold members.

I believe that we should see many contribution “footprints” for companies in Foundation leadership positions.  These footprints do not have to be code in github: there are many visible ways to contribute to OpenStack including internal installs, delivered product, community meetups, open source support around code, service to the community through speaking and sponsoring and, of course, code too.

At this point in the OpenStack evolution, there is so much going on that it is easy to leave footprints because there are so many ways to engage.  Footprints are tangible evidence of community leadership and the currency of collaboration.  OpenStack thrives because we are committed to working together, being transparent in our actions and providing service to the project beyond our own needs.

I believe OpenStack Foundation’s new gold members will are great additions to our growing community; however, we need to be increasingly deliberate in accepting new Gold members to make sure that they have a history of demonstrating a culture of open source leadership and contribution.  

These applications deserve careful consideration for several reasons:

  1. there are a limited number of gold level positions (16 of the 24 are now occupied)
  2. there is no practical way to remove a gold member (but only 8 are elected to the board)
  3. there is a perception (by the applicants) that they gain additional credibility through gold membership
  4. gold and platinum members are the leaders of our community so everyone will models their behavior

It is important to remember that there is no limit or barrier (beyond $) to joining at the corporate sponsors level. So, being a gold member means that companies are seeking a broader leadership role in the project.

Over the next months, Simon Anderson (committee chair, Dreamhost) will be leading me and several other board members in an effort to refine of our Gold member review criteria.  I’ll post own list shortly and I’m interested in hearing from you about what type of “footprints” we should be considered in this process.

OpenStack steps toward Interopability with Temptest, RAs & RefStack.org

Pipes are interoperableI’m a cautious supporter of OpenStack leading with implementation (over API specification); however, it clearly has risks. OpenStack has the benefit of many live sites operating at significant scale. The short term cost is that those sites were not fully interoperable (progress is being made!). Even if they were, we are lack the means to validate that they are.

The interoperability challenge was a major theme of the Havana Summit in Portland last week (panel I moderated) .  Solving it creates significant benefits for the OpenStack community.  These benefits have significant financial opportunities for the OpenStack ecosystem.

This is a journey that we are on together – it’s not a deliverable from a single company or a release that we will complete and move on.

There were several themes that Monty and I presented during Heat for Reference Architectures (slides).  It’s pretty obvious that interop is valuable (I discuss why you should care in this earlier post) and running a cloud means dealing with hardware, software and ops in equal measures.  We also identified lots of important items like Open OperationsUpstreamingReference Architecture/Implementation and Testing.

During the session, I think we did a good job stating how we can use Heat for an RA to make incremental steps.   and I had a session about upgrade (slides).

Even with all this progress, Testing for interoperability was one of the largest gaps.

The challenge is not if we should test, but how to create a set of tests that everyone will accept as adequate.  Approach that goal with standardization or specification objective is likely an impossible challenge.

Joshua McKenty & Monty Taylor found a starting point for interoperability FITS testing: “let’s use the Tempest tests we’ve got.”

We should question the assumption that faithful implementation test specifications (FITS) for interoperability are only useful with a matching specification and significant API coverage.  Any level of coverage provides useful information and, more importantly, visibility accelerates contributions to the test base.

I can speak from experience that this approach has merit.  The Crowbar team at Dell has been including OpenStack Tempest as part of our reference deployment since Essex and it runs as part of our automated test infrastructure against every build.  This process does not catch every issue, but passing Tempest is a very good indication that you’ve got the a workable OpenStack deployment.