We have a spec that has requirements for implementations. It would be nice to have a test suite that could be used to judge if an implementation is compatible with the specification.
The conformance tests should:
- have an assertion to cover each MUST, MUST NOT, etc in the spec that will fail the suite if the assertion fails
- have an assertion to cover each SHOULD, SHOULD NOT, etc in the spec, but do not fail the suite if the assertion fails
- have a negative test to cover each MAY, RECOMMENDED, etc in the spec, but do not fail the suite if the assertion fails
- make no assumptions as to ambiguities in the spec (those should be files as issues against the spec)
- make no assumptions as to the internals of the implementation, only the spec defined interfaces should be tested
- have minimal prerequisites (ideally none) other than a service binding implementation running in a cluster
- remove any state they add to the cluster during the tests
Initially, we should focus on the core specification. Spec extensions should discrete conformance tests that can be run independently.
We have a spec that has requirements for implementations. It would be nice to have a test suite that could be used to judge if an implementation is compatible with the specification.
The conformance tests should:
Initially, we should focus on the core specification. Spec extensions should discrete conformance tests that can be run independently.