Skip to content

feat(spec-review): record the ratifier on decisions — agent-chosen vs human-ratified (FAFF-1079) - #1121

Merged
alechill merged 8 commits into
mainfrom
FAFF-1079-record-the-ratifier-on-decisions-agent-chosen-vs-human-ratified
Oct 24, 2026
Merged

alechill merged 8 commits into
mainfrom
FAFF-1079-record-the-ratifier-on-decisions-agent-chosen-vs-human-ratified

Conversation

@alechill

@alechill alechill commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A spec's **Chosen:** decision can now record that a human, not the producer, settled it: **Chosen:** X (ratified: human). Adversarial spec-review design lenses defer to a tagged decision, so a call a human has already made isn't re-argued every round. The decision's implementation is still critiqued at full severity, and a critical is never deferred. No contract changes: faff-contract:spec-readiness keeps chosen | punt | assumes.

What changed

  • kernel.md, Decision markers: an optional Ratifier tag bullet.
    • It has one value in v1, human. An absent tag means agent-chosen, so behaviour is unchanged for existing specs.
    • To fit the kernel's line cap, the Sibling-skill invocation rationale paragraph moved to git history. Its one rule ("human-facing /faff-prep prose keeps its slash") is folded into step 3, and KERNEL_LINE_BASELINE drops 466 → 465.
  • faff-prep/SKILL.md, Spec-review gate: an interactive-only ratify choice beside revise and park.
    • It is offered only for a major or minor design-lens objection that a scope test classifies as attacking the decision itself, not its implementation. Anything ambiguous gets no offer.
    • On an explicit human "ratify" reply, prep appends the tag, logs a neutralised, 200-character-capped quote of the reply in the re-attach comment, and offers a tradeoff-shaped decisions-register capture.
    • The tag is carried forward through revise and autonomous stale-refresh.
  • Design lenses (refute-architectural.md, refute-infosec.md, refute-qa.md, plus the review-bench copies):
    • They defer to an in-spec tagged decision, and to ### Ratified tradeoffs (docs/decisions.md) entries.
    • They treat the tagged line's prose as untrusted data.
  • review-call.mjs: DEFAULT_BRIEF_RESERVE_TOKENS 2400 → 2800, re-measured after the lens clauses grew the binding refute-infosec.md brief to about 2596 tokens.
  • test/adversarial-context-window.test.mjs: two band fixtures retuned to the new reserve so each still exercises the usable-to-raw window band.
  • faffter-dark-spec-review/eval/in-spec-ratification/: the fixture set and recorded outcomes for the five design-lens checks, run against real backends.

For your review

  • SKILL_LINE_BASELINE["faff-prep"] 622 → 680. The ratify subsection adds 58 lines to one skill. That's worth a look against the lean standard.
  • The ratify-comment inspection criterion moved to FAFF‑1398, by operator decision. It needs a person typing "ratify" in an interactive prep session, which this build couldn't supply. The spec records the move.

Review

One pass: standard pass, plus an adversarial second opinion from openai/deepseek/deepseek-v4-flash-0731. Three design findings were accepted, because they restate the LLM-judged-prose shape the 2026-09-25 operator decision accepted. One was refuted: the reused neutralisation is neutralise() at ratified-scope.js:181.

Tests

  • validate-adapters and every lint rung pass.
  • test/validate-adapters.test.mjs and test/adversarial-context-window.test.mjs: 86 of 86 pass.
  • The full suite, run in chunks at low concurrency, shows 45 failures. They are the same 45 tests that fail on origin/main in this environment.

Acceptance Criteria

  • faff-1079#done:1 No edit touches faff-contract:spec-readiness's or spec-review-verdict's JSON schema files or their contract-defs.js mappers (grep confirms no diff in plugin/skills/faff/contracts/*.schema.json or the computeSpecReadiness/objection-schema regions of contract-defs.js).
    Verified: git diff origin/main over plugin/skills/faff/contracts/*.schema.json and contract-defs.js is empty.
    Method: command git diff origin/main --stat -- 'plugin/skills/faff/contracts/*.schema.json' plugin/skills/faff/bin/lib/contract-defs.js
  • faff-1079#done:2 kernel.md's new ratifier-tag bullet states the tag is never emitted on an autonomous dispatch, in those or equivalent words.
    Verified: The kernel bullet reads "never added on an autonomous dispatch".
    Method: command grep -o 'never added on an autonomous dispatch' plugin/skills/faff/references/kernel.md
  • faff-1079#done:3 kernel.md's Decision-markers list contains a Ratifier tag (optional) bullet in the same list as the existing Punt ownership (optional) bullet (adjacent placement is a readability nicety for the PR reviewer to check, not a mechanically-verifiable position), documenting (ratified: human) as a closed-vocabulary, single-value, prose-only suffix.
    Verified: The Ratifier tag (optional) bullet is in kernel.md's Decision-markers list; kernel.md measures 465 against KERNEL_LINE_BASELINE 465 and faff validate-adapters exits 0
    Method: command faff validate-adapters
  • faff-1079#done:4 The bullet states explicitly that a tagged **Chosen:** still emits { "marker": "chosen" }.
    Verified: The kernel bullet states a tagged Chosen still emits { "marker": "chosen" }.
    Method: command grep -o '{ "marker": "chosen" }' plugin/skills/faff/references/kernel.md
  • faff-1079#done:5 faff-prep/SKILL.md's Spec-review gate section offers a third "ratify" choice alongside revise/park for a major/minor design-lens objection that cleanly names a specific already-**Chosen:** decision and passes the decision-vs-implementation scope test, in interactive mode only.
    Verified: faff-prep's Spec-review gate gained Ratify a design-lens objection in place (interactive only), gated on major/minor plus the scope test.
    Method: command grep -c 'passes the scope test below' plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:6 The same text states an objection that is ambiguous or mixes decision- and implementation-level critique is never offered ratify — it routes through revise/park exactly as today.
    Verified: The same section states an ambiguous or mixed objection is never offered ratify and routes through revise/park.
    Method: command grep -c 'ambiguous between the two, or mixes both' plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:7 The same text states a critical objection is never offered this path.
    Verified: Procedure step 1 presents a critical as today with no ratify offer.
    Method: command grep -c 'severity == "critical" -> present as today' plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:8 The same text states autonomous mode never presents this offer.
    Verified: Procedure step 5 states the offer never fires in autonomous mode.
    Method: command grep -c 'Autonomous mode: never fires' plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:9 The same text states a ratify does not consume a loop-cap iteration.
    Verified: Procedure step 4d states a ratify does NOT consume a loop-cap iteration.
    Method: command grep -c 'consume a loop-cap iteration' plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:10 The same text states that on an explicit "ratify" reply, the human's verbatim reply (or its shortest decision-bearing sentence) is logged into the tracker comment the re-attach already posts, alongside the resulting tag.
    Verified: Procedure step 4b plus the Audit the utterance paragraph require the neutralised, capped quote in the tracker comment the re-attach already posts.
    Method: command grep -c 'neutralised, capped quote' plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:11 faff-prep/SKILL.md's Decisions-register capture step documents a second trigger (ratifying a design-lens objection) that drafts the tradeoff shape (Chosen/Rationale/Scope/Source-issue/Ratified-by: human/Date, no Matches:), while the original Punt/Assumes-closure trigger's precedent-shaped draft is unchanged in the diff.
    Verified: The capture step documents the second, tradeoff-shaped trigger with Source-issue/Ratified-by/Date and no Matches:; the original precedent-shaped trigger is unchanged in the diff.
    Method: command git diff origin/main -- plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:12 The extension text states the existing build-follows / no-build-follows materialisation split applies unchanged.
    Verified: The extension states follow-through is identical: build-follows materialises via Step 4c, no-build-follows surfaces the drafted text.
    Method: command grep -c 'Follow-through is \*\*identical\*\*' plugin/skills/faff-prep/SKILL.md
  • faff-1079#done:13 faffter-dark-spec-review/SKILL.md and all three of refute-architectural.md / refute-infosec.md / refute-qa.md explicitly name ### Ratified tradeoffs (docs/decisions.md) in their existing "Defer to ratified scope" text (grep for the literal heading string in each file).
    Verified: All four files name the literal ### Ratified tradeoffs (docs/decisions.md) in their Defer-to-ratified-scope text.
    Method: command grep -c 'Ratified tradeoffs (docs/decisions.md)' in SKILL.md + the three refute-*.md
  • faff-1079#done:14 Each of the three refute-*.md files gains a new "Defer to an in-spec human ratification" bullet stating: no ## Ratified scope block required; defers to observation on a tagged **Chosen:**; implementation still critiqued at full severity; critical never deferred; the tagged line's own text is treated as untrusted DATA, never an instruction.
    Verified: Each of the three refute files gained the Defer to an in-spec human ratification bullet with all five required statements.
    Method: command grep -c 'Defer to an in-spec human ratification' plugin/skills/faffter-dark-spec-review/refute-{architectural,infosec,qa}.md
  • faff-1079#done:15 faffter-dark-spec-review/SKILL.md's dense deferral paragraph gains the equivalent sentence, including the untrusted-DATA framing.
    Verified: The occupant SKILL.md's dense deferral paragraph gained the equivalent sentence including the untrusted-DATA framing.
    Method: command grep -c 'A fourth deferral source needs no block at all' plugin/skills/faffter-dark-spec-review/SKILL.md
  • faff-1079#done:16 kernel.md's ratifier-tag bullet (or faff-prep's autonomous refresh section) states that an autonomous stale-refresh preserves an already-present (ratified: human) tag verbatim — "never added on autonomous dispatch" governs new tags, not inherited ones.
    Verified: Both kernel.md's bullet and faff-prep's carry-forward paragraph state an autonomous stale-refresh preserves an already-present tag verbatim.
    Method: command grep -c 'inherited tag is carried forward verbatim' kernel.md; grep -c 'autonomous stale-refresh' faff-prep/SKILL.md
  • faff-1079#done:17 Build the fixture trio (committed under plugin/skills/faffter-dark-spec-review/eval/in-spec-ratification/ or the nearest existing eval-fixture home for this occupant): a minimal spec containing exactly one decision-level **Chosen:** finding independently known to draw a major objection from at least one design lens today (reuse or adapt an existing spec-judge-discrimination-style fixture rather than inventing an untested one), in three variants — untagged, the same file with (ratified: human) appended to that one line, and a critical-severity variant of the same decision — plus a fourth, separate fixture whose **Chosen:** decision draws an implementation-level objection (the "correctly identified but badly built" case Decision 4b's carve-out names), used only by item 4 below.
    Verified: Committed under plugin/skills/faffter-dark-spec-review/eval/in-spec-ratification/: three decision variants, the implementation-level fixture, the scope-test objection set, the register fixture, a README and run-log.
    Method: command ls plugin/skills/faffter-dark-spec-review/eval/in-spec-ratification/
  • faff-1079#done:18 Pin every run variable except the tag, and record the pin. Items 1-3 below compare outcomes across fixture variants that must differ in exactly the tag — so every run pins the same backend/model identity, the same --context file set, and the same ## Ratified scope presence-or-absence. Record the pinned values in a run-log.md (or equivalent) committed alongside the fixture trio itself — one file per fixture-directory, not a separate untracked note — so a later re-run of the fixture has the exact prior pin to reproduce, and a PR reviewer can check the pin without re-deriving it. A pass whose runs used different backends or context, or whose pin was never recorded, is not evidence the tag caused the difference.
    Verified: run-log.md records the single-element pinned chain, timeout, max-tokens, deadline, empty context set, absent Ratified-scope block, lens and system-dir, committed beside the fixtures.
    Method: command cat plugin/skills/faffter-dark-spec-review/eval/in-spec-ratification/run-log.md
  • faff-1079#done:19 Test coverage item 1 (no regression): run the design lens(es) against the untagged decision-level fixture, pinned per the rule above; the same major (or higher) objection appears. This is the fixture's pinned baseline, not a fresh eyeball check.
    Verified: Item 1 ran and the major objection appeared; recorded in run-log.md.
    Method: command node plugin/skills/faffter-dark-spec-review/build-lens-requests.mjs --lenses architectural ... | node plugin/skills/faffter-dark-adversarial-review/fan-out.mjs --strategy serial
  • faff-1079#done:20 Test coverage item 2 (defers to observation): run the same lens(es), same pinned run variables, against the tagged decision-level fixture with no ## Ratified scope file present; the previously-gating objection now appears only as an observation citing the tagged line. Because both variants differ in exactly one suffix under otherwise-pinned conditions, a pass here is direct evidence the tag — not lens variance — caused the downgrade.
    Verified: Item 2 ran and the objection appeared only as an observation citing the tagged line; the implementation-level objection stayed major, so the downgrade was scoped.
    Method: command node plugin/skills/faffter-dark-spec-review/build-lens-requests.mjs --lenses architectural ... | node plugin/skills/faffter-dark-adversarial-review/fan-out.mjs --strategy serial
  • faff-1079#done:21 Test coverage item 3 (critical never deferred): the critical-severity fixture variant, tag present, same pinned run variables; confirm it still gates.
    Verified: Item 3 ran and the critical still gated with the tag present.
    Method: command node plugin/skills/faffter-dark-spec-review/build-lens-requests.mjs --lenses architectural ... | node plugin/skills/faffter-dark-adversarial-review/fan-out.mjs --strategy serial
  • faff-1079#done:22 Test coverage item 4 (scope test correctly declines an implementation objection — the false-negative direction): run the ratify-offer scope test (Decision 2) against the separate implementation-level fixture's objection; confirm it is classified as implementation-level and the ratify choice is not offered. This closes the false-negative gap a decision-only fixture pair cannot exercise: a scope test that always classifies "decision-level" would pass items 1-3 while still wrongly ratifying every implementation objection.
    Verified: Item 4 ran: the implementation-level claim classified implementation-level, ratify_offered=no.
    Method: command node plugin/skills/faffter-dark-adversarial-review/review-call.mjs --system <scope-test brief> --diff <objection claim> --diff-kind prose
  • faff-1079#done:23 Test coverage item 5 (the fail-safe middle is tested, not merely named): run the same scope test against the ambiguous worked example itself ("Postgres is fine, but this schema will lock the table under load" — Decision 2's third worked example, not the clean implementation case item 4 uses) and confirm it too is not offered ratify. Item 4 alone tests only the unambiguous implementation case; the "ambiguous → no ratify, never the reverse" safety claim is specifically about the boundary case, so the boundary case is what must be exercised — a scope test that classifies unambiguous cases correctly but defaults an ambiguous one to "decision-level" would pass item 4 while failing the exact case the asymmetry exists to protect.
    Verified: Item 5 ran: the ambiguous worked example classified ambiguous, ratify_offered=no. A decision-level positive control returned yes, so the test is not a constant no.
    Method: command node plugin/skills/faffter-dark-adversarial-review/review-call.mjs --system <scope-test brief> --diff <objection claim> --diff-kind prose
  • faff-1079#done:24 Contract-extraction check (the "byte-identical" claim, named and run, not just asserted): for the same fixture's untagged and tagged variants, extract both specs' faff-contract:spec-readiness block and diff them byte-for-byte; the only permitted difference is inside the (ratified: human) suffix text itself, never in the emitted decisions[]/confidence JSON. This is the concrete check the "no contract regression" scenario (section 5) names but does not itself run.
    Verified: Contract-extraction check ran: both variants' extractions are identical and the validator output is byte-identical.
    Method: command faff contract spec-readiness < <extraction> ; diff <untagged-out> <tagged-out>
  • faff-1079#done:25 A docs/decisions.md fixture carrying a ratified_tradeoff entry for a synthetic issue confirms faff ratified-scope --assemble emits the ### Ratified tradeoffs subsection and faff ratified-scope --validate still passes (no regression to the existing selftest suite in ratified-scope.js/decisions.js).
    Verified: The register fixture round-tripped through --assemble and --validate; the ratified-scope.js and decisions.js selftests both still pass.
    Method: command node plugin/skills/faff/bin/lib/ratified-scope.js --selftest ; node plugin/skills/faff/bin/lib/decisions.js --selftest

Closes FAFF-1079

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Ready to land. Run this locally in a real terminal:

faff effects declare --run-dir .faff/anchors/run-20261023-014034-graft-FAFF-1079 --issue FAFF-1079 --step merge <<'EOF'
[{"kind":"merge","target":"pr:1121","reversible":true}]
EOF
faff merge-gate --pr 1121 --issue FAFF-1079 --run-dir .faff/anchors/run-20261023-014034-graft-FAFF-1079 --level L2 \
  --execute --merge-args "--squash --delete-branch"

…t-chosen vs human-ratified

Signed-off-by: Alec Hill <alec@shftwst.dev>
… human-ratified (FAFF-1079)

A `**Chosen:**` marker recorded that a decision closed, never who closed it, so the
spec-review design lenses could not tell an agent's own inference from a human's live
say-so and spent review rounds re-litigating settled calls.

Adds the `(ratified: human)` suffix as a prose-only convention and wires the one consumer
that already has a defer-or-pass-through mechanism:

- the kernel's decision-marker dialect gains a `Ratifier tag (optional)` bullet — closed
  single-value vocabulary, never a `faff-contract:spec-readiness` field, never added on an
  autonomous dispatch, inherited tags carried forward verbatim;
- faff-prep's Spec-review gate gains a third `ratify` path behind a decision-vs-implementation
  scope test, with a neutralised and 200-character-capped quote of the human's reply in the
  tracker comment the re-attach already posts;
- the Decisions-register capture step gains a second, tradeoff-shaped trigger;
- the three design-lens refuters and the occupant's own deferral paragraph name the
  `### Ratified tradeoffs (docs/decisions.md)` subsection and gain a fourth deferral clause
  that reads the tag off the spec body, needing no `## Ratified scope` block.

Neither contract schema is touched: a tagged `**Chosen:**` still emits `{ "marker": "chosen" }`,
verified byte-for-byte through `faff contract spec-readiness`.

A committed fixture set under the occupant's own eval home carries the evidence, with its pins
and recorded outcomes: the untagged variant draws a `major`, the tagged variant records the same
objection as an `observation` citing the tagged line while its implementation half stays `major`,
and the `critical` variant still gates with the tag present.

Two consequences of the required prose growth are carried here rather than hidden: the
faff-prep line-cap baseline rises with its justification, and the adversarial brief-token
reserve is re-measured against the grown design-lens briefs.

Signed-off-by: Alec Hill <alec@shftwst.dev>
…ng rationale out (FAFF-1079)

Operator decision 2026-10-22: free kernel lines by moving prose, not by raising
KERNEL_LINE_BASELINE. The Sibling-skill invocation section's "Why this exists"
paragraph is rationale, so it leaves the kernel. Its one rule (human-facing
"type /faff-prep" prose keeps its slash) is folded into step 3. The baseline is
lowered to the new measured 465 so the ratchet stays tight.

The rationale, preserved here: a fixed /faff-prep string is correct in at most
one install mode. The Skill tool takes a skill name, and that name differs per
mode, so every delegation resolves against the live available-skills list
rather than a frozen literal. That keys off whatever the canonical name is,
which composes with a future skill rename without editing the rule.

Also retunes two band fixtures in test/adversarial-context-window.test.mjs
(tests 50 and 51) to the re-measured 2800-token brief reserve, so each still
exercises the usable-to-raw window band it exists to pin.

Signed-off-by: Alec Hill <alec@shftwst.dev>
…-FAFF-1079

Signed-off-by: Alec Hill <alec@shftwst.dev>
@alechill
alechill force-pushed the FAFF-1079-record-the-ratifier-on-decisions-agent-chosen-vs-human-ratified branch from 73bae86 to c900371 Compare October 24, 2026 07:35
…-1398

Signed-off-by: Alec Hill <alec@shftwst.dev>
…AFF-1398

Signed-off-by: Alec Hill <alec@shftwst.dev>
…ratifier-on-decisions-agent-chosen-vs-human-ratified

Signed-off-by: Alec Hill <alec@shftwst.dev>

# Conflicts:
#	plugin/skills/faff/bin/lib/validate-adapters.js
…ratifier-on-decisions-agent-chosen-vs-human-ratified

Signed-off-by: Alec Hill <alec@shftwst.dev>

Signed-off-by: Alec Hill <alec@shftwst.dev>
@alechill
alechill merged commit e895df5 into main Oct 24, 2026
14 checks passed
@alechill
alechill deleted the FAFF-1079-record-the-ratifier-on-decisions-agent-chosen-vs-human-ratified branch October 24, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant