[spMohanty]

Readies the kit for the Phase 2 public release. The enabling change is one dependency pin; most of the rest of the diff follows from it.

Why the pin moved

	before	after
whestbench	>=0.15.0,<0.16.0	>=0.16.0,<0.17.0
flopscope	>=0.11.0,<0.12.0	>=0.12.0,<0.13.0

whestbench 0.16.0 is the first release whose defaults are the Phase 2 round. Under 0.15.0 the CLI applied Phase 1 limits while the documentation described Phase 2:

whest run default	0.15.0	0.16.0
shape	256 × 32	1024 × 16
flop_budget	2.72e11	2⁴¹ = 2,199,023,255,552
wall_time_limit_s	60	120
setup_timeout_s	(flag did not exist)	5.0
residual_wall_time_limit_s	—	0.4
lambda_flops_per_second	1e11	0.0, so C_m = F_m
dataset-less --n-samples	2,560,000	200,000

Every one of those is now the CLI default, so a plain uv run whest run rehearses the Phase 2 parameters with no flags at all. Several pages carried explicit "this becomes true at 0.16.0" hedges, written when it was not; they are gone.

flopscope 0.12.0 reprices ~40 operations relative to 0.11.0, in both directions, but none are operations the bundled examples use (matmul, tagged einsum, stats.norm, astype, pointwise). Every documented FLOP figure is therefore unchanged, and tests/test_flopscope_cost_docs.py is the gate that keeps it so.

pyproject.toml records the ordering rule it is following: the kit should never report FLOP counts a submission is not scored on, so it tracks the graded meter rather than the newest release.

New: docs/reference/rounds.md

The kit had no page describing the round it is in, so a participant reading a Phase 1 number somewhere had no way to place it. The new page puts all three rounds side by side — dataset tag, shape, predict() return shape, forward-pass cost, budget, λ, residual cap, wall cap, residual model — plus how to reproduce a score from an earlier round and why pre-Phase-2 local numbers do not compare.

It also records the ratio of budget to forward-pass cost:

round	budget ÷ one forward pass
v1-warmup	64,850
v1-phase1	64,850
v2-phase2	65,536

Each round's budget corresponds to approximately 65,000 forward passes. Phase 2's budget is 8.08× Phase 1's, and one Phase 2 forward pass costs 8× one Phase 1 forward pass, so the number of passes a submission can afford is unchanged across rounds.

tests/test_rounds_doc.py (19 tests) asserts every cell of that table against whestbench.budget.ROUNDS at runtime, so a round change upstream fails the build. The two forward-pass rows are derived rather than read, since RoundConfig has no such field, and the page states this.

Why Phase 2 drops the residual price

The kit asserted C_m = F_m in three places and explained it in none, which leaves anyone who remembers Phase 1's λ unable to tell whether the rate was removed deliberately or is simply missing. rounds.md now carries the reasoning and the other two sites link to it:

• Pricing permits the purchase of unmetered compute. Under λ, arithmetic performed outside flopscope was billable rather than disallowed: it had a defined cost, and a submission that paid it stayed within the rules. Phase 2 prohibits computation outside flopscope, so there is no transaction for a rate to price. The remaining use of residual time is plumbing, which a cap bounds directly.
• The priced score depended on the grading machine. R_m is wall-clock time, so under λ > 0 an identical submission produced different values of C_m on different hardware and under different load. F_m is derived analytically from tensor shapes and dtypes and does not vary by machine.
• A combined budget did not state a FLOP limit. Under λ, B bounded a sum of FLOPs and priced seconds, so the FLOPs available to a submission depended on how much residual time it spent.

The page also states that the 0.4 s cap is a limit rather than an allowance: residual time below it does not affect the score, and exceeding it zeroes that MLP's predictions.

Fixed: whest package silently dropped shipped weights

The kit's own .gitignore contained *.npz, and whest package reads .gitignore and .whestignore as one pattern list with no ! negation. Following docs/how-to/ship-weights.md produced:

Status: success

…and a tarball containing estimator.py and manifest.json, with weights.npz absent and no warning in the output. The omission first surfaced at grading time as a setup() error, which fails the whole submission rather than one MLP.

No file in this repository is an .npz; examples/04_shipped_weights.py writes to a temporary directory. The pattern therefore excluded only participant files, including the workflow documented in ship-weights.md. Removed, with a comment recording why it should not return. Verified before and after.

Fixed: an incorrect FAQ entry on process isolation

faq.md said Stage 4 and the grader "run each MLP in a fresh process — that state is gone." Measured under both runners: one pid, one module import, one setup(), one instance, state carried across MLPs. --runner subprocess is identical to --runner local here.

The remedy it recommended, caching through SetupContext.scratch_dir, raises inside setup(): that field is None on every 0.16.0 path, and a setup() exception fails the whole submission.

The entry now describes worker-level state, which the kit did not cover: state carries between calls, but a submission is served by several worker processes (approximately 5-15 setup() runs), each handling only the MLPs assigned to it, and a replaced worker starts from an empty state. A cache built across MLPs is therefore populated locally and empty on several grader workers.

Corrections

Corrections made against the shipped 0.16.0 harness, by theme:

• setup() does not run once per submission — once per worker process, several workers per submission, and a replaced worker re-runs it. Corrected in six places. It changes the advice for anyone amortising an expensive setup across the suite.
• Defaults described as flags you must pass — --wall-time-limit, --residual-wall-time-limit and the new --setup-timeout are already the Phase 2 values.
• Re-measured numbers — reductions bill N−1 (var bills 4·N), fnp.eye(1024) is 2,048 at its float64 default, stats.norm per-element costs, the einsum symmetry saving. The Phase 1 example ratio was quoted as ~160×, a float64-era figure compared against a float32 one; measured like-for-like it is 137.6×.
• Report fields — four emitted on every run were undocumented (whestbench_version, results.per_layer_mse, per_mlp[].mlp_name, per_mlp[].per_layer_mse); traceback was listed as error-only when it is always present; error_code is not stable across runners.
• Commands — every fenced whest … is now uv run whest …; a bare whest version can resolve to a different install entirely.
• Links — one dead URL (/participants/me/customize → /participants/me/edit); 141 internal links checked, 0 broken.

local_engine.py now models the 120 s wall cap and the 0.4 s residual gate, calls setup() outside the budget context and teardown() after the sweep, and states in its docstring what it still does not model.

Style guide

The kit adopts the Google developer documentation style guide.

Headings and titles move to sentence case. Capitalization only: GitHub lowercases anchors, so the wording is unchanged and every existing link still resolves. The emoji prefixes are kept.

Prose moves to second person and active voice with the actor named, figurative language is replaced by the mechanism it stood for, conditions precede instructions, and link text describes its destination. Word-level changes follow the guide's word list: e.g. becomes "for example", i.e. becomes "that is", etc. is replaced by naming the remaining items, and simply, easy and filler uses of just are removed. Occurrences of just meaning "only" are kept, as in "Stage 1: just python".

No number, flag, path, field name, command or error string changed, and no fenced content was edited.

Refreshed demo

assets/demo.cast was recorded under Phase 1: width=256 depth=32, FLOP Budget 2.72e+11, memory_limit_mb=65536, and an unpinned dataset resolving to main. Re-recorded at the Phase 2 shape.

The two recorders had drifted apart — one pinned @v1-phase1, the other pinned nothing — which is how the cast went stale unnoticed. Both now pin @v2-phase2, and make demo-cast-headless SOURCE=. can record a local checkout so a pre-merge release can be captured accurately.

Verification

check	result
uv run pytest tests/ -q	47 passed
uv run ruff check .	clean
CI	green, including the new Stage 4 and Stage 5 smokes
examples 01–04, estimator.py, --baseline	all run
whest validate	passes on all five bundled estimators
internal documentation links	43 files, 0 broken

Baseline scores over the full 100-MLP mini split at @v2-phase2:

estimator	final_layer_mse	all_layers_mse
zeros baseline	0.909506	0.763593
01_random	0.685614	0.538962
02_mean_propagation	2.21617e-04	1.61058e-04
03_covariance_propagation	4.05038e-06	2.22912e-06

These match the figures already published in concepts/scoring-model.md, so that table is unchanged. The sentence above it, describing the numbers as a Phase 1 measurement, was incorrect and is removed.