Skip to content

ProvenanceBench: curate pairs from #45 #48

Description

@aykoooo

What & why

Refs #36
Refs #45

Grow provenancebench/pairs.yaml from the #45 pair table (~100–200 rows): dense and MoE transformation kinds with cites, revision pins, and label_confidence.

Scope

  • Import rows from ProvenanceBench — benchmark for the fingerprint search loop #45 table (finetune, quant, abliterate, expert drop, merge, …).
  • Add controls: neg_cross, neg_mistag (card claims ref but weights are not derived — compliance-critical), neg_adapter_only, unsupported shape-mismatch pairs.
  • Target ~25–35% non-related gold in full suite; keep smoke balance tests (≥40% controls in smoke).
  • Pin ref_rev / sus_rev on smoke cases.
  • Anchor sync with smokes/fixtures.yaml where applicable.

Out of scope

  • Autoresearch agent loop wiring (docs only until scoring issue lands).
  • Hub network validation in CI (offline schema tests stay).

Test plan

  • Pair count and kind coverage documented on issue
  • test_pairs_schema and anchor sync tests pass
  • Smoke label-balance test passes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions