fix: check-claim accepts a numeric claim only when every number in it is supported

CHANGELOG.md

@@ -11,6 +11,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).
 
 ### Added
 
+- Two labelled claim sets for `check-claim` (`benchmarks/claims-v1.jsonl`: 60 paraphrased numeric claims; `benchmarks/claims-copy-v1.jsonl`: 60 claims copied from the same 30 abstracts, the unsupported half with one number, unit, direction, or binding changed), each frozen before the engine was run on it, and `scripts/benchmark_claims.py`, which runs a set through `check-file` on a fresh cache and reports the false-`ACCEPT` rate, `ACCEPT` precision and recall with Wilson intervals. It needs the network and is not run in CI.
 - An end-to-end skill evaluation (`benchmarks/e2e/`): eight manuscripts (Markdown, LaTeX + BibTeX, Korean) with 40 planted errors and 65 correct references, a harness that runs each through an isolated `claude -p` session with and without the skill (no user hooks, plugins, or MCP servers), a stdlib scorer, and the 2026-10-09 results: 80/80 planted errors caught with the skill and 77/80 without, no false alarms in either, at about twice the time per run. It needs the network and a logged-in Claude Code, and is not run in CI.
 - A browser page (`web/`) that runs `check-bib` on a pasted list or a `.bib`/`.ris`/`.txt` file with no server: the engine runs in Pyodide (pinned 314.0.7 from jsDelivr) inside a Web Worker and calls CrossRef and doi.org directly. It shows progress, the same verdict tables and legend as the HTML report, a Korean/English toggle, and downloads the same HTML report the CLI writes (in English). In Korean, the page shows each reason in Korean from a template table (`web/reasons.js`) with DOIs, years, and titles kept as written, and shows any reason it cannot translate in English marked "(원문)"; `tests/test_web_reasons.py` fails when the engine gains a `check-bib` reason without a Korean template. Nothing is cached or stored in the browser. `scripts/build_web.py` builds the site; `.github/workflows/pages.yml` deploys it once Pages is enabled and the `PAGES_ENABLED` repository variable is `true`.
 
@@ -21,6 +22,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).
 
 ### Fixed
 
+- `check-claim` and `check-file` no longer accept a claim with a wrong number next to a right one. The engine bound one quantity of a claim and ignored the rest, so "a service lifetime of 61 years at 80 °C" was accepted from "16 years at 80 °C". An `ACCEPT` now also needs every number in the claim, with its unit, prefix (`mA` vs `A`, `MPa` vs `mPa`), sign, and comparator, to be entailed by a number in the same evidence clause or sentence, and a direction word such as "higher than" or "decreased" that is not a number's own comparator to appear there. An approximate figure ("~5000 cycles") no longer supports an exact claim. Numbers that are part of names (`Fe2C`, `COVID-19`, `cm−1`) and citation markers (`[12]`, `(Lee et al., 2024)`) are not counted. On 60 claims copied from 30 abstracts with one change each, false `ACCEPT` went from 4/30 to 0/30 and `ACCEPT` on the supported claims from 14/30 to 16/30.
+- A claim that the abstract states word for word is accepted when the sentence goes on to qualify a different item: "1.28 MPa tensile strength with 573% elongation" was `WARN` because the scope check read "at −25 °C" from a later item of the same list. A value inside parentheses is now scoped to its own `;`-separated list item. Claims with a percentage (other than strain, which keeps its own checks) can now be accepted word for word.
 - `check-bib` no longer attaches a reference without a DOI to a different CrossRef record that shares its title. Supplementary-information components are never taken as the cited work; replies, comments, and retraction or correction notices ("Retraction—<title>", "Authors' reply") join review reports as records about another work; when several records match, the one whose first author agrees wins over search order (a 1998 *Lancet* letter titled like Wakefield's paper no longer hides the retraction); a title that only mostly appears in the reference must come with an agreeing first author; a reference that reads as a book (publisher or edition, no journal volume and page) matches only book records, not journal reviews of the book; and a reference that says preprint prefers the preprint record.
 - A Korean (or Chinese/Japanese) reference with a title is no longer treated as title-less: the title-less check counts Hangul words and CJK runs, not only Latin words, and leaves out a leading list of Hangul or Han author names. A Korean reference whose title disagrees with the Korean title CrossRef stores for its DOI is reported as naming another paper; before, it was explained as "The reference has no article title".
 

benchmarks/README.md

@@ -25,6 +25,7 @@ measure again after the tool is tuned against v1's misses.
   naming: `2026-10-08-34da678.json` (before #27) and `2026-10-08-7910447.json` (after #27,
   before the held-out set existed).
 - `../scripts/benchmark_references.py`: runs a set (standard library only).
+- `claims-v1.jsonl`, `claims-copy-v1.jsonl`, and `../scripts/benchmark_claims.py`: the claim sets and their runner (see [Claim set](#claim-set-claims-v1jsonl)).
 - `../scripts/render_scorecard.py`: draws `.github/assets/scorecard-{light,dark}.svg`
   from a held-out results file, with the development-set result as a footnote. The README chart
   is drawn from `results/2026-10-09-8170e66-holdout-v2.json` with
@@ -62,6 +63,74 @@ set, before the fix) and at `fe8e810` (after it), without `REF_VERIFY_MAILTO`.
 No fabricated item became PASS, no real item lost its PASS, and no retracted item lost
 `PAPER_RETRACTED` in any set. Results: `results/2026-10-09-{6f4386b,fe8e810}-*.json`.
 
+## Claim set (`claims-v1.jsonl`)
+
+60 numeric claims for measuring `ref-verify check-file` (the same check as `check-claim`):
+30 the abstract supports and 30 it does not, six each of a wrong number, a wrong unit or
+unit prefix, a reversed direction (above/below, rose/fell, more/less than), a number that
+belongs to another sample or quantity in the same abstract, and a claim with several
+quantities where one is wrong. Each of the 30 papers gives one supported and one
+unsupported claim. The papers are recent materials, chemistry, energy, sensor, water,
+concrete, and clinical papers with a CrossRef abstract; none appears in any other set,
+the E2E sets, or the test fixtures. The set was frozen at the commit that added it, before
+the claim engine was run on it.
+
+`claims-copy-v1.jsonl` has 60 more claims on the same 30 papers, written the other way:
+the supported claim copies a span of the abstract (a few tidy subscript or exponent
+spacing, such as `NH 3` to `NH3`), and the unsupported claim is that span with exactly one
+thing changed, six of each error type. This is how a citing sentence drafted from an
+abstract goes wrong. It was added after the first run of `claims-v1.jsonl`, where the
+unchanged engine accepted none of the 60 paraphrased claims, true or false, and so could not
+show a false ACCEPT; it was frozen before the engine was run on it.
+
+| Field | Meaning |
+|---|---|
+| `id` | `c1-NN` (`cc1-NN` in the copy set). |
+| `doi`, `claim` | What a user would pass to `check-claim`. |
+| `label` | `SUPPORTED` or `NOT_SUPPORTED`, by a reading of the abstract. |
+| `error_type` | For unsupported claims: `wrong_number`, `wrong_unit`, `wrong_direction`, `wrong_subject`, or `multi_quantity`. |
+| `evidence_quote` | An exact substring of the abstract that decides the label. |
+| `abstract_source`, `abstract_sha256` | The abstract the label was read from, as `ref-verify` fetches it. |
+| `verified`, `notes` | How the label was checked, and why a claim is wrong. |
+
+Scoring: `ACCEPT` is the only verdict counted as "supported"; `WARN` and `REJECT` both
+count as "not accepted". The headline is the false-ACCEPT rate on unsupported claims; the
+runner also reports ACCEPT precision and recall with 95% Wilson intervals. A row whose
+fetched abstract no longer matches `abstract_sha256` is listed under `abstract_changed`.
+
+```bash
+python3 scripts/benchmark_claims.py
+python3 scripts/benchmark_claims.py --dataset benchmarks/claims-copy-v1.jsonl
+```
+
+### Multi-quantity claim fix (2026-10-09)
+
+Each set was run on a fresh cache before the fix (`claims-v1` at `6c44979`,
+`claims-copy-v1` at `7d6a148`; the engine is the same in both) and after it (`2768bca`),
+without `REF_VERIFY_MAILTO`. No abstract changed between runs.
+
+| Set | Run | False ACCEPT (unsupported claims) | ACCEPT on supported claims | ACCEPT precision |
+|---|---|---|---|---|
+| claims-copy-v1 | before | 4/30 (5–30%) | 14/30 (30–64%) | 14/18 (55–91%) |
+| claims-copy-v1 | after | **0/30 (0–11%)** | 16/30 (36–70%) | 16/16 (81–100%) |
+| claims-v1 | before | 0/30 (0–11%) | 0/30 (0–11%) | no ACCEPT |
+| claims-v1 | after | 0/30 (0–11%) | 0/30 (0–11%) | no ACCEPT |
+
+Parentheses are 95% Wilson intervals. The four false ACCEPTs before the fix were one of
+each kind except a swapped subject: a cycle count changed beside a right efficiency
+(multi-quantity), an exponent changed (`10−4` to `10−3`), a unit changed in a secondary
+quantity (`42 ms` to `42 s`), and "higher than" turned into "lower than". The direction-word
+rule that catches the last one was added after this run showed it, so the
+`wrong_direction` row of the copy set is partly in-sample.
+
+On the copy set, three supported claims became ACCEPT (word-for-word claims carrying a
+percentage) and one became WARN: `cc1-27`, whose second number sits in the next clause
+of a sentence that goes on to "whereas ... (from 0.000486 to 1.17 MPa)", so the scope
+check sees "from". The paraphrased claims-v1 stays at no ACCEPT either way: `check-claim`
+accepts only claims whose wording follows the abstract, so it measures what is
+explicitly stated, not what a reader would accept as a paraphrase. Results:
+`results/2026-10-09-{6c44979,7d6a148,2768bca}-claims*.json`.
+
 ## Item fields
 
 | Field | Meaning |
@@ -143,8 +212,8 @@ None so far for `references-holdout-v1.jsonl`, `references-holdout-v2.jsonl`, or
   exposed (#27) were developed against it, so the stored results are in-sample; a new,
   held-out set is the honest next measurement.
 
-- Whether a paper supports a claim (`check-claim`); only reference existence and
-  metadata are scored here.
+- Whether a paper supports a claim (`check-claim`), in the reference sets; only reference
+  existence and metadata are scored there. `claims-v1.jsonl` covers numeric claims.
 - Non-English literature beyond a handful of Korean items (journal papers, theses, and
   one paper cited by its Korean title).
 - Full text, page numbers, volume/issue, or journal names: `check-bib` compares title,