Repository navigation
Cover every published binding construct with a checked-in example - #105
Open
reissjason wants to merge 1 commit into
Open
reissjason wants to merge 1 commit into
reissjason wants to merge 1 commit into
Conversation
The binding publishes 24 lorav: terms, four payload layouts, 32 wire types, two byte orders, two presentWhen shapes and five derived-value descriptors. Until now a construct was exercised only if some vendor happened to ship a schema using it, so coverage of the vocabulary tracked vendor adoption rather than the vocabulary itself. The seven curated examples reached 20 of the 24 terms, three of the four layouts, five of the 32 wire types, one of the two presentWhen shapes and none of the five derived descriptors. examples/devices/ reaches most of the rest, but it is generated, gitignored and pinned to a submodule: nothing in it is reviewed, nothing in it is stable, and ctv plus 19 wire types appear in neither corpus at all. examples/conformance/ closes that with one hand-written Thing Description per construct, each with vectors whose expected values were computed from the wire bytes by hand and are then held against the reference interpreter, rather than recorded from whatever it returned. The existing suite picks them up through conftest.example_tds()/vector_files(), so each fixture is schema-validated, decoded and round-tripped like any other example. tests/test_conformance_corpus.py is what keeps the coverage from decaying. It walks the vocabulary, the layout set, the wire-type enum and the derived-value descriptors and fails if any published member has no example, so minting a term without an example that exercises it breaks the suite. A published term nobody has decoded once is a term nobody knows works. Discovery is an explicit union of examples/*.td.json and examples/conformance/*.td.json rather than a recursive glob: examples/devices/ and examples/generated/ move with the submodule pin and must not steer the checked-in suite. Two published constructs deliberately have no fixture, because one would have to encode wrong behaviour in order to pass. Both are filed as issues and recorded in examples/conformance/README.md and UNCOVERED_WIRE_TYPES, with a test asserting that list stays accurate: - bool/xsd:boolean. The reference interpreter reads bool as a bit of the byte under the cursor and does not advance the cursor without 'consume'. The binding has no term for 'bit' or 'consume' and converter._WIRE_WIDTH gives bool a width of 1, so a second boolean re-reads the first one's byte and every later field is read one byte early. - Mixed lorav:endian within one Thing Description. converter._typed() emits a be_/le_ type prefix for a field that disagrees with the schema default; the interpreter has no such prefix and rejects the field. Across 167 generated and 7 curated Thing Descriptions, zero emit it, so the path had never run. Two test-side bugs surfaced on the way. tests/snapshot.py and test_decode.py both resolved a vectors file's "td" against examples/ rather than against the vectors file's own directory, which only worked while every example sat in one flat directory. The golden snapshot is re-recorded: 19 payload schemas and 37 decoded vectors, the conformance ones under a conformance/ prefix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Thank you very much for the PR! I ran the tests and they passes. In generall, I support adding these conformance tests alongside the real-device tests. Also future developers can treat this as labled examples. Though, we show binding-feature coverage, not real device compatibility here. My opinion is to prefer real device with derived vectors for all the labeled examples where available and use synthetic cases for specific gaps and edge conditions. Beside, my AI agent found one mismatch: polynomial documentation (the form schema, ontology, and README describe coefficients in ascending order (c0 + c1x + c2x^2)) with the interpreter’s descending coefficient order. Please align this before merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
examples/conformance/— twelve hand-written Thing Descriptions, one binding construct each, plustests/test_conformance_corpus.py, which fails if any construct the binding publishes has no example.Why
Coverage of the vocabulary currently tracks vendor adoption, not the vocabulary. A
lorav:term, a wire type or a layout is exercised only if some manufacturer happens to ship a schema that uses it. Terms that no vendor has adopted are published, documented, accepted bylorawan-form.schema.json— and have never been decoded once.What the two existing corpora actually reach:
examples/*.td.jsonlorav:termslorav:wireTypevalueslorav:endianvalueslorav:presentWhenshapeslorav:deriveddescriptorslorav:byteLengthformsexamples/devices/reaches more, but it cannot play this role: it is generated, gitignored and pinned to a submodule, so nothing in it is reviewed and nothing in it is stable across a bump. Two things were covered in neither corpus at all — thectvlayout, and 19 of the 32 published wire types.This mirrors, one layer up, the
_language-conformance/fixtures indevice-payload-schema, which hold the four reference interpreters to the same behaviour for constructs no device schema uses — and which found two cross-language defects when they were added.What it found
Building the corpus turned up two published constructs that do not work. Both are filed separately and are deliberately not papered over with a fixture:
bool/xsd:boolean. The reference interpreter readsboolas a bit of the byte under the cursor and does not advance the cursor withoutconsume. The binding has no term forbitorconsume, andconverter._WIRE_WIDTHgivesboola width of 1, so a second boolean re-reads the first one's byte and every later field is read one byte early.lorav:endianin one Thing Description.converter._typed()emits abe_/le_type prefix for a field disagreeing with the schema default; the interpreter has no such prefix and rejects the field. Across 167 generated and 7 curated Thing Descriptions, zero emit it, so the path had never run.Both are recorded in
UNCOVERED_WIRE_TYPESandexamples/conformance/README.md, with a test asserting that gap list stays accurate — it fails if a listed type gains a fixture or leaves the schema.How the fixtures are built
Every expected value in a
*.vectors.jsonhere was computed from the wire bytes by hand and is then held against the reference interpreter, rather than recorded from whatever the interpreter returned. Three of my hand-computed values disagreed with the interpreter on the first run; two of those disagreements were the defects above.The fixtures are picked up by the existing suite through
conftest.example_tds()/vector_files(), so each one is schema-validated, decoded and round-tripped like any other example. Discovery is an explicit union ofexamples/*.td.jsonandexamples/conformance/*.td.json, never a recursive glob, so the gitignored generated catalog cannot steer the checked-in suite.Incidental fixes
tests/snapshot.pyandtests/test_decode.pyboth resolved a vectors file's"td"key againstexamples/rather than against the vectors file's own directory. That only worked while every example sat in one flat directory; both now resolve beside the vectors file.The golden snapshot is re-recorded: 19 payload schemas and 37 decoded vectors, with the conformance entries under a
conformance/prefix.Verification
The guard was mutation-checked: deleting
layout-ctv-default-tags.td.jsonandderived-compute-guard.td.jsonproduces exactly🤖 Generated with Claude Code