Skip to content

Cover every published binding construct with a checked-in example - #105

Open
reissjason wants to merge 1 commit into
eclipse-thingweb:mainfrom
MultiTechSystems:test/lorawan-wot-conformance-corpus
Open

reissjason wants to merge 1 commit into
eclipse-thingweb:mainfrom
MultiTechSystems:test/lorawan-wot-conformance-corpus

Conversation

@reissjason

Copy link
Copy Markdown
Contributor

What this adds

examples/conformance/ — twelve hand-written Thing Descriptions, one binding construct each, plus tests/test_conformance_corpus.py, which fails if any construct the binding publishes has no example.

Why

Coverage of the vocabulary currently tracks vendor adoption, not the vocabulary. A lorav: term, a wire type or a layout is exercised only if some manufacturer happens to ship a schema that uses it. Terms that no vendor has adopted are published, documented, accepted by lorawan-form.schema.json — and have never been decoded once.

What the two existing corpora actually reach:

curated examples/*.td.json after this PR
lorav: terms 20 / 24 24 / 24
payload layouts 3 / 4 4 / 4
lorav:wireType values 5 / 32 30 / 32
lorav:endian values 1 / 2 2 / 2
lorav:presentWhen shapes 1 / 2 2 / 2
lorav:derived descriptors 0 / 5 5 / 5
lorav:byteLength forms 0 / 2 2 / 2

examples/devices/ reaches more, but it cannot play this role: it is generated, gitignored and pinned to a submodule, so nothing in it is reviewed and nothing in it is stable across a bump. Two things were covered in neither corpus at all — the ctv layout, and 19 of the 32 published wire types.

This mirrors, one layer up, the _language-conformance/ fixtures in device-payload-schema, which hold the four reference interpreters to the same behaviour for constructs no device schema uses — and which found two cross-language defects when they were added.

What it found

Building the corpus turned up two published constructs that do not work. Both are filed separately and are deliberately not papered over with a fixture:

Both are recorded in UNCOVERED_WIRE_TYPES and examples/conformance/README.md, with a test asserting that gap list stays accurate — it fails if a listed type gains a fixture or leaves the schema.

How the fixtures are built

Every expected value in a *.vectors.json here was computed from the wire bytes by hand and is then held against the reference interpreter, rather than recorded from whatever the interpreter returned. Three of my hand-computed values disagreed with the interpreter on the first run; two of those disagreements were the defects above.

The fixtures are picked up by the existing suite through conftest.example_tds() / vector_files(), so each one is schema-validated, decoded and round-tripped like any other example. Discovery is an explicit union of examples/*.td.json and examples/conformance/*.td.json, never a recursive glob, so the gitignored generated catalog cannot steer the checked-in suite.

Incidental fixes

tests/snapshot.py and tests/test_decode.py both resolved a vectors file's "td" key against examples/ rather than against the vectors file's own directory. That only worked while every example sat in one flat directory; both now resolve beside the vectors file.

The golden snapshot is re-recorded: 19 payload schemas and 37 decoded vectors, with the conformance entries under a conformance/ prefix.

Verification

1270 passed  ->  1498 passed, 55 skipped
ruff check   ->  All checks passed
ruff format  ->  28 files already formatted

The guard was mutation-checked: deleting layout-ctv-default-tags.td.json and derived-compute-guard.td.json produces exactly

FAILED test_every_supported_layout_has_an_example[ctv]
FAILED test_every_derived_descriptor_has_an_example[compute]
FAILED test_every_derived_descriptor_has_an_example[guard]

🤖 Generated with Claude Code

The binding publishes 24 lorav: terms, four payload layouts, 32 wire types,
two byte orders, two presentWhen shapes and five derived-value descriptors.
Until now a construct was exercised only if some vendor happened to ship a
schema using it, so coverage of the vocabulary tracked vendor adoption rather
than the vocabulary itself.

The seven curated examples reached 20 of the 24 terms, three of the four
layouts, five of the 32 wire types, one of the two presentWhen shapes and none
of the five derived descriptors. examples/devices/ reaches most of the rest,
but it is generated, gitignored and pinned to a submodule: nothing in it is
reviewed, nothing in it is stable, and ctv plus 19 wire types appear in neither
corpus at all.

examples/conformance/ closes that with one hand-written Thing Description per
construct, each with vectors whose expected values were computed from the wire
bytes by hand and are then held against the reference interpreter, rather than
recorded from whatever it returned. The existing suite picks them up through
conftest.example_tds()/vector_files(), so each fixture is schema-validated,
decoded and round-tripped like any other example.

tests/test_conformance_corpus.py is what keeps the coverage from decaying. It
walks the vocabulary, the layout set, the wire-type enum and the derived-value
descriptors and fails if any published member has no example, so minting a term
without an example that exercises it breaks the suite. A published term nobody
has decoded once is a term nobody knows works.

Discovery is an explicit union of examples/*.td.json and
examples/conformance/*.td.json rather than a recursive glob: examples/devices/
and examples/generated/ move with the submodule pin and must not steer the
checked-in suite.

Two published constructs deliberately have no fixture, because one would have
to encode wrong behaviour in order to pass. Both are filed as issues and
recorded in examples/conformance/README.md and UNCOVERED_WIRE_TYPES, with a
test asserting that list stays accurate:

- bool/xsd:boolean. The reference interpreter reads bool as a bit of the byte
  under the cursor and does not advance the cursor without 'consume'. The
  binding has no term for 'bit' or 'consume' and converter._WIRE_WIDTH gives
  bool a width of 1, so a second boolean re-reads the first one's byte and
  every later field is read one byte early.
- Mixed lorav:endian within one Thing Description. converter._typed() emits a
  be_/le_ type prefix for a field that disagrees with the schema default; the
  interpreter has no such prefix and rejects the field. Across 167 generated
  and 7 curated Thing Descriptions, zero emit it, so the path had never run.

Two test-side bugs surfaced on the way. tests/snapshot.py and test_decode.py
both resolved a vectors file's "td" against examples/ rather than against the
vectors file's own directory, which only worked while every example sat in one
flat directory. The golden snapshot is re-recorded: 19 payload schemas and 37
decoded vectors, the conformance ones under a conformance/ prefix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Hao-Siemens

Hao-Siemens commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Thank you very much for the PR!

I ran the tests and they passes. In generall, I support adding these conformance tests alongside the real-device tests. Also future developers can treat this as labled examples. Though, we show binding-feature coverage, not real device compatibility here. My opinion is to prefer real device with derived vectors for all the labeled examples where available and use synthetic cases for specific gaps and edge conditions.

Beside, my AI agent found one mismatch: polynomial documentation (the form schema, ontology, and README describe coefficients in ascending order (c0 + c1x + c2x^2)) with the interpreter’s descending coefficient order. Please align this before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants