This post comes with a WARNING LABEL… THE FOLLOWING SELECTION CRITERIA ARE PRELIMINARY TO GET FEEDBACK AND HELP VALIDATE THE PROCESS.

UPDATE 5/7/14 > see the OFFICIAL version.

ORIGINAL TEXT

---

As part of the DefCore work, we have the challenge of taking all the Tempest tests and figuring out which ones are the “must-pass” tests that will define core (our note pages).  We want to have a very transparent and objective process for picking the tests so we need to have well defined criteria and a selection process.

Figuring out the process will be iterative.  The list below represents a working set of selection criteria that are applied to the tests.  The DefCore committee will determine relative weights for the criteria after the tests have been scored because it was clear in discussion that not all of these criteria should have equal weight.

Once a test passes the minimum criteria score and becomes “must-pass” the criteria score does not matter – the criteria are only used for selecting tests. As per the Core principles, passing all “must-pass” test will be required to be considered core.

So what are these 13 preliminary criteria (source)?

1. Test is required stable for >2 releases (because things leaving Core are bad)

• the least number/amount of must pass tests as possible (due to above)
• but noting that the number will increase over time
• least amount of change from current requirements as possible (nova, swift 2 versions)
• (Acknowledge that deprecation is punted for now, but can be executed by TC)

2. Where the code being tested has an designed area of alternate implementation (extension framework) as per the Core Principles, there should be parity in capability tested across extension implementations

• Test is not configuration specific (test cannot meet criteria if it requires a specific configuration)
• Test does not require an non-open extension to pass (only the OpenStack code)

3. Capability being tested is Service Discoverable (can be found in Keystone and via service introspection) – MONTY TO FIX WORDING around REST/DOCS, etc.

• Nearly core or “compatible” clouds need to be introspected to see what’s missing
• Not clear at this point if it’s project or capability level enforced.  Perhaps for Elephant it’s project but it should move to capability for later

4A, 4B & 4C. Candidates are widely used capabilities

• 4A favor capabilities that are supported by multiple public cloud providers and private cloud products
  • Allow the committee to use expert judgement to promote capabilities that need to resolve the “chicken-and-egg”
  • Goals are both diversity and quantity of users

• 4B. Should be included if supported by common tools (Ecosystem products includes)

• 4C. Should be included if part of common libraries (Fog, Apache jclouds, etc)

5. Test capabilities that are required by other must-pass tests and/or depended on by many other capabilities

6. Should reflect future technical direction (from the project technical teams and the TC)

• Deprecated capabilities would be excluded (or phased out)
• This could potentially become a “stick” if used incorrectly because we could force capabilities

7. Should be well documented, particularly the expected behavior.

• includes the technical references for others in the project as well as documentation for the users and or developers accessing the feature or functionality

8. A test that is a must-pass test should stay a must-pass test (makes must-pass tests sticky release per release)

9. A test for a Capability with must-pass tests is more likely to be considered must-pass

10 Capabilities is unique and cannot be build out of other must-pass capabiliies

• Candidates favor capabilities that users cannot implement if given the presence of other capabilities
• consider the pain to users if a cloud doesn’t have the capability – not so much pain if they can run it themselves
• “Unique capabilities that cannot be build out of other must-pass capabilities should not be considered as strongly”

11. Tests do not require administrative right to execute

We expect these criteria to change based on implementation experience and community input; however, we felt that further discussion without implementation was getting diminishing returns.  It’s import to remember that all of the criteria are not equal, they will have relative weights to help drive tune the results.