Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- An end-to-end skill evaluation (`benchmarks/e2e/`): eight manuscripts (Markdown, LaTeX + BibTeX, Korean) with 40 planted errors and 65 correct references, a harness that runs each through an isolated `claude -p` session with and without the skill (no user hooks, plugins, or MCP servers), a stdlib scorer, and the 2026-10-09 results: 80/80 planted errors caught with the skill and 77/80 without, no false alarms in either, at about twice the time per run. It needs the network and a logged-in Claude Code, and is not run in CI.
- A browser page (`web/`) that runs `check-bib` on a pasted list or a `.bib`/`.ris`/`.txt` file with no server: the engine runs in Pyodide (pinned 314.0.7 from jsDelivr) inside a Web Worker and calls CrossRef and doi.org directly. It shows progress, the same verdict tables and legend as the HTML report, a Korean/English toggle, and downloads the same HTML report the CLI writes (in English). In Korean, the page shows each reason in Korean from a template table (`web/reasons.js`) with DOIs, years, and titles kept as written, and shows any reason it cannot translate in English marked "(원문)"; `tests/test_web_reasons.py` fails when the engine gains a `check-bib` reason without a Korean template. Nothing is cached or stored in the browser. `scripts/build_web.py` builds the site; `.github/workflows/pages.yml` deploys it once Pages is enabled and the `PAGES_ENABLED` repository variable is `true`.

### Fixed

- `check-bib` no longer attaches a reference without a DOI to a different CrossRef record that shares its title. Supplementary-information components are never taken as the cited work; replies, comments, and retraction or correction notices ("Retraction—<title>", "Authors' reply") join review reports as records about another work; when several records match, the one whose first author agrees wins over search order (a 1998 *Lancet* letter titled like Wakefield's paper no longer hides the retraction); a title that only mostly appears in the reference must come with an agreeing first author; a reference that reads as a book (publisher or edition, no journal volume and page) matches only book records, not journal reviews of the book; and a reference that says preprint prefers the preprint record.
- A Korean (or Chinese/Japanese) reference with a title is no longer treated as title-less: the title-less check counts Hangul words and CJK runs, not only Latin words, and leaves out a leading list of Hangul or Han author names. A Korean reference whose title disagrees with the Korean title CrossRef stores for its DOI is reported as naming another paper; before, it was explained as "The reference has no article title".

## [1.3.1] — 2026-10-08

### Changed
Expand Down
18 changes: 17 additions & 1 deletion benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ fabricated, retracted, and unindexed references against live CrossRef.

| Set | Items | Role |
|---|---|---|
| `references-regress-v3.jsonl` | 23 | **Regression set for the search-path and Hangul-title fixes.** Not a held-out score: it holds the failure cases found by the E2E evaluation (#37) and holdout-v2 (a same-title letter or review taken for the paper, an SI component taken for the paper, a book matched to a journal review of it, a Korean reference with a title read as title-less) and close-call controls where the cited record really is a letter, a review, a book, or a preprint. Frozen before the before/after runs. |
| `references-holdout-v2.jsonl` | 99 | **Held-out set v2.** Written and committed before any run, after #30 and #33 were tuned on holdout-v1's misses. It shares no paper (DOI or title) with the other two sets and stresses title-less physics/chemistry citations (13 real, 8 fabricated), group authors, particle surnames, Korean-titled Korean journal papers with Hangul authors, books, conference papers, retractions marked in CrossRef only through Retraction Watch data, and retractions CrossRef cannot mark (KISTI, ISTIC, and DataCite DOIs). Measured once, on 1.3.1; the README scorecard is not yet switched to it. |
| `references-holdout-v1.jsonl` | 86 | **Held-out set v1, now in-sample:** its misses were used to tune #30 and #33. Originally written and committed before the tool was run on it; no paper in it appears in the development set, and it uses other journals, fields, and citation styles (IEEE, Nature, Harvard, Chicago, biblatex, and physics/chemistry styles that omit the article title). The README headline numbers and chart come from this set. |
| `references-v1.jsonl` | 142 | **Development set (used while fixing the tool).** Its misses drove the matching fixes in #27, so its scores are in-sample and are shown only as a secondary line. |
Expand Down Expand Up @@ -44,6 +45,21 @@ and a FASEB abstract DOI aliased to its withdrawal notice), reported separately:
four got `PAPER_RETRACTED`. Per-item results:
`results/2026-10-09-8170e66-{holdout-v2,holdout-v1,v1}.json`.

## Wrong-record and Hangul-title fix (2026-10-09)

Each set was run on a fresh cache at `6f4386b` (main `e045a86` plus the frozen regress-v3
set, before the fix) and at `fe8e810` (after it), without `REF_VERIFY_MAILTO`.

| Set | Before | After |
|---|---|---|
| regress-v3 | real 10/12 PASS, retracted 0/1, 1 wrong-record PASS, 2 unindexed books PASSed on a review | real 12/12 PASS, retracted 1/1, no wrong-record PASS |
| holdout-v2 | real 40/45, fabricated 29/29, retracted 8/8, unindexed REJECT 0/13 | unchanged; 2 references no longer attach to a wrong record |
| holdout-v1 (in-sample) | real 40/40, fabricated 26/26, retracted 10/10, unindexed REJECT 0/10 | unchanged; 2 no longer attach to a wrong record |
| v1 (in-sample) | real 66/66, fabricated 43/43, retracted 16/16, unindexed REJECT 1/17 | unchanged; 3 no longer attach to a wrong record |

No fabricated item became PASS, no real item lost its PASS, and no retracted item lost
`PAPER_RETRACTED` in any set. Results: `results/2026-10-09-{6f4386b,fe8e810}-*.json`.

## Item fields

| Field | Meaning |
Expand Down Expand Up @@ -117,7 +133,7 @@ rerun on the same commit can differ slightly from a stored result.

## Label corrections

None so far for `references-holdout-v1.jsonl` or `references-holdout-v2.jsonl`.
None so far for `references-holdout-v1.jsonl`, `references-holdout-v2.jsonl`, or `references-regress-v3.jsonl`.

## What this does not measure

Expand Down
Loading
Loading