Skip to content

ProvenanceBench: scoring and verifier alignment #47

Description

@aykoooo

What & why

Refs #36

Define how ProvenanceBench scores detect.py output and how that maps to aibom_verifier verdicts (verified_derivative, fraudulent_claim, incompatible, unsupported, insufficient_evidence).

Scope

  • Document gold y vs detector pred (related / unrelated / abstain) and confidence float role.
  • Threshold policy: when does a float become confirm vs reject vs abstain?
  • Map bench labels to verifier taxonomy (table in issue or wiki).
  • Optional: per-signal floats in run log for triage; calibration metrics later (not v1 gate).
  • label_confidence: disputed rows excluded from primary bal_acc or scored separately.

Out of scope

  • Implementing real weight signals in detect.py (separate work).
  • Per-run random negative generation.

Test plan

  • Written spec accepted on issue
  • Eval harness metrics align with spec (or doc explains gap)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions