Skip to content

perf: faster witness proofs and proof checks - #152

Merged
flyq merged 19 commits into
mainfrom
liquan/perf/proof-opt
Nov 13, 2026
Merged

flyq merged 19 commits into
mainfrom
liquan/perf/proof-opt

Conversation

@flyq

@flyq flyq commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Faster SALT witness proofs and proof checks. Measured end to end with mega-reth's replay-full pipeline over 5000 mainnet blocks on a 32-vCPU box: per witness at one worker 33 → about 6 ms (v1.0.5 → this branch), and the pipeline's throughput from 100 to 172 blocks/s on these salt changes alone; on the stateless validator against production salt v1.0.6, a mainnet head block goes from 15.3 to 8.2 ms (witness verification 9.9 → 2.7 ms) and 16-worker throughput from 225 to 463 blocks/s. Proof and witness bytes are unchanged: every optimization is pinned by byte-for-byte tests against the pre-existing code paths. The branch is trimmed to the mechanisms with a measured end-to-end win: a per-mechanism ablation and a replay-full + validator run took out everything worth under 2% (records in mega-reth data/salt/salt-152-ablation-2026-09-21.zh-CN.md and data/salt/salt-152-e2e-2026-09-21.zh-CN.md), and the cache refresh moved to its own stacked PR, #156.

A rebuilt node polynomial now enters the cache only after a check that it opens to its commitment. The pipeline figures above were measured before that check existed and have not been re-run; what it costs on a cache miss is under Measurements.

Changes

  • Fixed-base MSMs and a shared CRS (ad33ac8): ipa::create_with_precomp computes every IPA round's L/R as fixed-base MSMs over the original generators with wNAF tables instead of folding the generator vector, MultiPoint::open_with_committer reuses the process-wide committer, DEFAULT_CRS is built once, the remaining variable-base MSMs use a windowed parallel Pippenger, plus a proof/witness bench (salt/benches/proof.rs).
  • Node polynomial cache (62741fe, d357608, salt/src/proof/subtrie.rs): a process-wide cache keyed by physical node id and validated by commitment bytes, so a witness skips the 256-child range read and the map-to-field pass for every node whose commitment it has seen; query construction keeps node/point order either way (the transcript hashes it). The shared committer carries Q as base n, so the IPA's per-round blinding terms are fixed-base too.
  • Cache admission and eviction (7c709ef, 3bb7086): a rebuilt polynomial is cached only if it opens to the commitment its node was read at, so a store whose commitments and contents come from different states cannot poison the cache for consistent ones. All the polynomials a call rebuilds are checked together: one 128-bit challenge per node, drawn from a blake3 hash of every commitment and polynomial in the batch, and commit(Σ r_i·poly_i) compared with Σ r_i·C_i; only a batch that fails is checked node by node. A full shard evicts one polynomial per new node and passes over those used since it last considered them, instead of dropping everything it holds. NodePolyCache is public (global(), capacity(), set_capacity(), clear()); a capacity of zero turns the cache and its check off. Its locks park under std, as Lazy does.
  • Pippenger bucket sizing (911cc37, banderwagon/src/msm.rs): signed digits never index the upper half of a window's 2^c bucket array, yet the reduction walked it; every window now takes 2^(c-1) buckets (the last one 2^last_bits, since it absorbs the carry). Platform-independent, and it lands on the sizes a proof check hits: the IPA's closing 274-point MSM and a head witness's few hundred to two thousand commitments.
  • Trimmed after measurement (784d695, 6d5298f, 94f1c4a): ScalarWindows, all five rayon chunk floors, and the refresh (moved to perf(proof): refresh the node-polynomial cache from each block's own transition #156). The cold-cache bench groups went with them and are back (a015ed1): they are what times the miss path, and they clear the cache through NodePolyCache::clear.
  • Review cleanups (1f97191, 44e3094): one thread_chunk_size! macro for the per-thread chunking idiom and its zero guard; CRS::committer builds the G ++ [Q] tables the provers require and create_with_precomp asserts that shape (its variable-base blinding fallback is gone); Committer::msm replaces ipa::fixed_base_msm, and Committer::msm_dense commits a dense polynomial without building a term list; create_sub_trie reads one ParentNode per parent in node order and resolve_polys takes that slice; MultiPoint::open moves its CRS instead of cloning it; DEFAULT_CRS back to pub(crate).

API changes

  • Breaking, ipa-multipoint: ProverQuery::poly is an Arc<LagrangeBasis> (was LagrangeBasis), so a caller that owns the polynomial writes poly: poly.into(). ipa::multi_scalar_mul_par is removed; banderwagon::multi_scalar_mul already splits its windows across threads. Neither mega-reth nor stateless-validator uses either.
  • Added: salt::NodePolyCache, Committer::msm_dense, and a public LagrangeBasis::values, beside the prover entry points listed above.

Testing

On this head: cargo test (salt lib: 225 passed), cargo test --no-default-features, cargo test --features test-bucket-resize, cargo check -p salt --target riscv64imac-unknown-none-elf --no-default-features, cargo clippy --all-targets -- -D warnings and cargo fmt --check clean. cargo-mutants over the salt lines changed by 3bb7086: no survivors and no timeouts.

New tests: cached_node_polynomials_reproduce_the_witness_byte_for_byte, create_with_precomp_matches_create (both committer shapes), open_with_committer_matches_open, msm_windowed_matches_arkworks_for_every_window_width, make_digits_stays_within_the_bucket_counts, msm_dense_matches_msm_and_the_reference; for the cache, inconsistent_store_does_not_poison_witnesses_of_a_consistent_store, batch_check_passes_only_when_every_polynomial_opens_to_its_commitment, batch_challenges_depend_on_every_commitment_and_polynomial, resolve_polys_caches_only_the_rebuilds_that_open_to_their_commitment, resolve_polys_rebuilds_everything_when_the_cache_is_off, node_poly_cache_evicts_what_was_not_used_since_the_hand_passed and node_poly_cache_capacity_bounds_trims_and_turns_the_cache_off.

On the testbed (at 784d695 and earlier, before the consistency check), 5000 blocks pass on every configuration with 0 errors across 75 runs, and the validator reproduces every parent root.

Measurements

The validator and pipeline runs are all on one box (AMD EPYC 9B45, 16 cores / 32 threads), runs strictly sequential, arms interleaved in alternating order.

Validator, end to end — same stateless-validator code, only salt swapped; witnesses served from memory, blocks from a local full node; p50 per block. Head window: 2000 mainnet blocks, 19–41 tx. Dense window: 34k–44k tx per block. v1.0.6 and d357608 from the 09-16 run, d357608 again and 784d695 from the 09-21 run (d357608 reproduces within 2%); the verifier path of 784d695 is the same as this head's.

salt head, 1 worker: blocks/s verify ms total ms head, 16 workers: blocks/s CPU ms/block dense, 1 worker: verify ms CPU s/block
v1.0.6 (production) 63 9.9 15.3 225 134 180 11.3
this PR before bucket sizing (d357608) 117 2.85 8.3 433 71 156 10.2
this PR (784d695) 118 2.7 8.2 463 66 150 10.1

EVM replay (2.8 ms head / 355 ms dense) and salt_update (2.5 ms head; 84 → 82 → 80 ms dense) do not move with the salt arm, so removing ScalarWindows cost the validator's state-root step nothing measurable. All 34 + 12 runs validated every block (execution plus state root against the header) with 0 ERROR lines.

Witness pipeline, end to end — mega-reth liquan/perf/pipeline-opt replay-full over 5000 mainnet blocks (25827013..25832012), one binary per salt arm, 3 configurations × 5 rounds, 75 runs, 0 errors, run-to-run spread ±0.3%. Median blocks/s, and the per-block witness build time in ms; M = concurrent witness builders.

salt arm M1 blocks/s build ms M6 blocks/s build ms M8 blocks/s build ms
cache off 130.5 7.36 214.1 23.82 222.7 31.38
cache on, refresh ignored (≈ this PR before the consistency check) 145.5 6.55 248.3 19.60 257.9 24.68
cache + refresh (784d695) 152.7 6.25 279.4 15.91 290.2 18.30
784d695 + ScalarWindows + three chunk floors (= before the trim, 4e613ce) 154.6 6.15 282.2 15.84 293.5 18.13
784d695 − its last two chunk floors (= #156 at the time) 154.8 6.15 278.2 16.53 286.4 20.68

The cache is worth +11.5% / +16% / +16%. It gains most under concurrency because the pool is saturated there and the CPU it saves turns into wall time one for one; with it off the state-root stage slows as well (2.86 → 3.74 ms at M8), since the extra witness work takes CPU from it. Everything the trim removed is within 1.4% either way.

These runs predate the consistency check (7c709ef), and the pipeline has not been re-run with it. The check runs once per call for the nodes that missed, so the first row is unaffected and the others pay it in proportion to their miss rate: more than half of a witness's nodes at this PR's 44% hit rate, a handful at #156's 98%. This PR's head is the second row without the two chunk floors and with the check, a combination not run by itself, so the second row overstates it by what the check costs on the pipeline.

Per-mechanism ablation — salt's criterion benches, one mechanism switched off per arm against the full branch (at 4e613ce, before the trim and before the consistency check); the figure is how much slower the bench gets, i.e. what the mechanism saves (parenthesised = not significant, p > 0.05). "cold" rows clear the node cache before every iteration.

bench full branch fixed-base IPA off Q blinding off node cache off bucket sizing off
salt_proof/create/16 2.07 ms +231% +23% +15% (−2.0%)
salt_proof/create/256 3.43 ms +141% +17% +46% (−0.3%)
salt_proof/create/2048 7.98 ms +55% +4.1% +64% −1.0%
salt_proof/create-cold/256 4.98 ms +96% +11% (0.0%) (−0.9%)
salt_proof/create-cold/2048 12.77 ms +37% +2.6% +1.7% +1.4%
witness/create/256 4.11 ms +116% +11% +37% (−1.0%)
witness/create/1024 7.25 ms +64% +9.2% +49% (+0.7%)
witness/create-cold/1024 10.72 ms +43% +4.0% (+0.3%) −1.1%
salt_proof/check/16 2.01 ms (0.0%) (+0.1%) (+0.1%) +27%
salt_proof/check/256 2.29 ms (−1.0%) (−0.5%) (−1.0%) +6.6%
witness/verify/256 2.38 ms +1.9% (+1.1%) +1.3% +7.8%
witness/verify/1024 5.02 ms (−0.3%) (−0.7%) (−0.1%) +11%

At 4e613ce a cold cache cost nothing (lookup + insert were below the noise), and each arm moves only the benches it should. Since then a miss also pays for the consistency check, measured next. The arms for what this PR no longer carries were within ±3.3% on every proof bench and mostly not significant: ScalarWindows (+1.3% / +2.4% on update 10000 KVs at 1 / 8 threads), the max(64) shape floor, and the four other chunk floors.

Cache misses and the consistency check — SaltProof::create on a 50,000-key MemStore, release build, Apple M4 Pro (14 cores); "cold" is the first proof of a key set (every node a miss), "warm" the same proof again (every node a hit). Rebuilding a node from a MemStore is nearly free, so the cold column isolates the check.

320 keys / 550 nodes: cold, 14 threads cold, 1 thread warm, 14 threads warm, 1 thread 2048 keys / 3503 nodes: cold, 14 threads cold, 1 thread
no check (75cdab8) 5.0 ms 18.2 ms 4.4 ms 17.0 ms 10.6 ms 39.5 ms
one fixed-base MSM per miss (7c709ef) 57 ms 485 ms 4.6 ms 18.3 ms 325 ms 2.8 s
one check per call (this head) 7.6 ms 28.7 ms 4.5 ms 17.1 ms 17.7 ms 86.5 ms

A miss cost about 0.85 ms of CPU under the per-node check and costs under 20 µs now. With criterion on the same machine at this head: salt_proof/create/256 5.16 ms and create-cold/256 9.32 ms; witness/create/256 5.61 ms and create-cold/256 9.89 ms.

Per commit, earlier runs — per witness at one worker, mainnet shape (321 keys / 347 internal / 215 leaf nodes): v1.0.5 33.2 ms → fixed-base MSMs 7.43 → cache + fixed-base blinding 6.0; pipeline (W32/K4/M6, mega-reth before its own pipeline work): 100 → 150 → 172 blocks/s from the salt changes alone. msm_windowed alone (M4 Pro, release): 274 points 525 → 418 µs, 1,000 points 880 → 798, 2,335 points 2,046 → 1,718, 8,192 points 4,884 → 4,564.

Records in mega-reth: data/salt/salt-opt-e2e-validator-2026-09-16.zh-CN.md, data/salt/salt-152-ablation-2026-09-21.zh-CN.md, data/salt/salt-152-e2e-2026-09-21.zh-CN.md.

Notes

  • The node polynomial cache is process-wide and bounded to 32768 polynomials by default (64 shards × 512, up to ~256 MB of LagrangeBasis); NodePolyCache::global().set_capacity(n) changes the bound, and 0 turns the cache off. Entries are validated by commitment, so a stale or foreign entry is only ever a miss. It is only read and written by proof creation; a validator never touches it.
  • Callers should still keep the store unchanged while a proof is created: a proof built from a store whose commitments and contents disagree does not verify. The consistency check only keeps such a store from leaving polynomials in the cache that later proofs would reuse.
  • Without the refresh the cache is filled by rebuilds alone, so a node the previous block wrote is rebuilt once under its new commitment (hit rate 44% on mainnet; perf(proof): refresh the node-polynomial cache from each block's own transition #156 lifts it to 98%).
  • Follow-ups kept on a backup branch, each measured: a forward patch of cache misses from the previous block's transition (+5% at six workers, needs a mega-reth counterpart), fusing each trie level's delta and accumulate passes into one parallel section (state-root step 2.0 → 1.4 ms in isolation), a compile fix for the enable-hugepages feature, and the perf phase-timing feature used for the measurements.
  • Consumers: mega-reth and stateless-* pin salt by tag; mega-reth's replay drivers call Witness::create_with_refresh, so they follow perf(proof): refresh the node-polynomial cache from each block's own transition #156 rather than this branch.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Performance Benchmark Comparison

Compared 5 benchmark(s) against the latest main baseline.

Detailed Comparison
Benchmark Baseline Throughput (Kelem/s) New Throughput (Kelem/s) Change
update 10000 KVs/1 threads 80.86 82.00 +1.41%
update 10000 KVs/2 threads 156.31 157.48 +0.75%
update 10000 KVs/4 threads 288.85 289.82 +0.34%
update 10000 KVs/8 threads 473.63 470.59 -0.64%
update 10000 KVs/16 threads 595.72 597.08 +0.23%

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Mutation testing - PASS

Mutation score: 100.0% (1/1 viable mutants killed)

  • caught: 1
  • survived, real gaps: 0
  • timed out, inconclusive: 0
  • suppressed: 0
  • unviable: 0
  • timeout total: 0

No unsuppressed survivors or timeouts remain.

flyq and others added 5 commits September 7, 2026 17:38
Restores the 2026-08 proof-optimization work that lived as uncommitted
changes on main (kept alive in stash@{0}, "proof-bench work", 2026-08-25)
onto main 945d5bd. Content is unchanged apart from rustfmt on the bench.

- banderwagon: hand-rolled window-parallel Pippenger MSM (msm_windowed)
  that avoids arkworks' thread-pool traffic at SALT's MSM sizes;
  multi_scalar_mul skips the batch inversion when the bases are already
  affine; Committer::num_bases.
- ipa-multipoint: create_with_precomp computes every IPA L/R point as a
  fixed-base MSM over the original CRS generators (byte-identical
  proofs); MultiPoint::open_with_committer and commit_dense route the
  polynomial commitments through the precomputed tables; queries share
  their polynomial via Arc; verification sums scalars per commitment run
  so the final multiexp has one point per node instead of per query;
  serial fast paths for small powers_of_par and per-query scalar work;
  equivalence tests against the generic paths.
- salt: process-wide DEFAULT_CRS and shared_committer() so proof
  creation reuses the trie's fixed-base tables instead of rebuilding the
  CRS per call; shape.rs/subtrie.rs cut per-key allocations, read each
  bucket's metadata once, and cache default-commitment scalar
  conversions; new benches/proof.rs covering SaltProof::create/check and
  Witness::create/verify.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGFBjWNK8YFania1Bstst9
…ding base

- proof/subtrie: a process-wide node-polynomial cache keyed by physical
  node id and validated by commitment bytes (the commitment binds the
  children), so unchanged nodes skip the 256-child range read and the
  map-to-field pass. Query construction keeps node/point order either
  way: the transcript hashes it. Hit rate on mainnet ~44% internal /
  ~48% leaf (misses are the block's own write paths).
- ipa: the shared committer carries Q as base n, so the IPA's per-round
  blinding terms z*(w*Q) are fixed-base multiplications instead of two
  variable-base ones per round.
- banderwagon: ScalarWindows produces the wNAF digits on the fly, so
  mul_index no longer allocates a Vec per call.
- rayon chunk floors in the multiproof aggregate, the shape pass, the
  parent reads and the fixed-base MSM (8 terms): below that the dispatch
  costs more than the work.

Measured with mega-reth's replay-full pipeline (W32/K4/M6, 5000 mainnet
blocks, 32-vCPU box): 150 -> 172 blocks/s; per witness at M=1
7.43 -> 6.00 ms (IPA 2.41 -> 1.70, internal nodes 1.88 -> 1.37). Proofs
stay byte-identical (see the determinism test).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01722A9KuCsH6MnM83Mhax3Y
The node-polynomial cache only helps for nodes whose commitment did not
change since they were cached; every node on a block's write paths is a
miss (~55% of the internal nodes of a mainnet witness) that costs a
256-child range read and 256 map-to-field conversions.

A witness proves the pre-state, so the polynomials it resolves are
exactly the bases the block's transition transforms. `NodePolyRefresh`
carries that transition — the (old, new) commitment of every changed
node and the (old, new) value of every changed slot, the pair
`StateRoot::update_fin` produced — and `Witness::create_with_refresh`
(also `SaltWitness`/`SaltProof`) applies it after the witness's own
lookups: each changed parent's polynomial is copied, its changed
positions overwritten (hash-to-field of the new child commitment, or
the slot's new value), and the result inserted under the parent's new
commitment, so the next block's witness hits. Buckets whose capacity
changes are frozen for the block; subtree tops are keyed the way the
witness reads them; a parent read at any commitment other than the
transition's old one is skipped. Debug builds check every refreshed
entry: old + sum G_i*(s_i - base_i) == new, which commitment validation
alone cannot see. The existing `create` signatures are unchanged.

Measured on the replay-full pipeline: hit rate at one worker 44% -> 98%,
per-witness internal-node phase 1.39 -> 0.42 ms; at six workers the
witness build 19.6 -> 15.3 ms and the pipeline 265 -> 305 blocks/s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01722A9KuCsH6MnM83Mhax3Y
…-state guard

Six tests over `NodePolyRefresh` in subtrie.rs, each pinning a path that a
mutation of the refresh left the suite blind to:

- `refresh_uses_cached_bases_for_parents_the_witness_did_not_read`: a
  planned parent out of the witness's hand is skipped while the cache holds
  no entry for its old commitment, and refreshed from that entry once a
  pre-state witness has cached it.
- `refresh_applies_every_planned_parent`: with every base in hand,
  `apply_refresh` inserts exactly one entry per planned parent, each equal
  to the post-block rebuild.
- `witness_refresh_reports_every_planned_parent` (`perf`): a witness over
  every changed key reports `refreshed_nodes == plan.len()` on the calling
  thread, race-free against the shared cache.
- `refresh_rejects_a_witness_over_the_post_state` (debug): the
  `read_at_old` guard fires with its own message on a witness over the
  post-state.
- `refresh_skips_a_parent_read_at_a_foreign_commitment` (release): a
  witness over the pre-B view carrying block C's transition patches
  nothing, so the bucket root stays absent under C's commitment.
- `refresh_patches_a_rehashed_bucket_with_unchanged_capacity`: a nonce-only
  rehash of a two-level, 512-slot bucket is not frozen; its leaves, top and
  metadata leaf are planned at the moved slots and refreshed.

Each test fails under the mutation it targets (dropping the cache fallback,
skipping level-1/2 parents, freezing on any metadata change, accepting any
read commitment) and passes on the code as it is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01722A9KuCsH6MnM83Mhax3Y
The refresh doc comments referred to numbered invariants of a design note
outside the repository; the sentences now stand on their own.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01722A9KuCsH6MnM83Mhax3Y
@flyq
flyq force-pushed the liquan/perf/proof-opt branch from 8968473 to 6cfc1cd Compare September 7, 2026 09:57
@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Mutation testing - PASS

Mutation score: 100.0% (54/54 viable mutants killed)

  • caught: 54
  • survived, real gaps: 0
  • timed out, inconclusive: 0
  • suppressed: 0
  • unviable: 79
  • timeout total: 0

No unsuppressed survivors or timeouts remain.

flyq and others added 6 commits September 14, 2026 07:44
…est fixtures

Quality cleanup of the proof-opt changes; proof and witness bytes are unchanged
(blake3 digests of three serializations match before/after).

- subtrie: one resolve_polys helper replaces the duplicated internal/leaf cache
  blocks; in_hand carries the commitment each base was read at and is only
  filled with a refresh, so apply_refresh drops its parent_bytes parameter;
  multi_commitments_to_scalars takes node ids; empty_slot_scalar and
  slot_position are shared by refresh_plan and leaf_polynomial; the test-only
  query shims are gone; node_poly_cache::insert evicts only for a new node and
  frees evicted or replaced entries outside the shard lock.
- ipa-multipoint: open_with_committer takes &CRS (no CRS clone per proof);
  create_with_precomp uses a plain flag for Q in the tables and reuses its
  per-round term buffers; commit_dense falls back to commit_lagrange_poly;
  check calls multi_scalar_mul directly and multi_scalar_mul_par is deprecated;
  LagrangeBasis::values is public.
- DEFAULT_CRS lives in trie.rs next to the committer built from it, so the trie
  no longer depends on proof::prover.
- tests/bench: apply_block and two_block_fixture move to proof::test_utils and
  back the witness.rs refresh test; shared value/meta/store_with_two_slots
  helpers; the check bench builds its data with SaltWitness::create.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hold

`make_digits` recenters every window's digit to [-2^(c-1), 2^(c-1) - 1], so a
bucket index (|digit| - 1) never reaches the upper half of the 2^c array the
windowed MSM allocated: half of every window's bucket array was untouched, and
the reduction still walked it, paying one point addition per empty bucket
(arkworks skips the zero operand, so the cost was the `res += running_sum` of
each empty slot rather than both additions).

Each window now takes 2^(c-1) buckets. The last window is the exception: it
absorbs the outstanding carry instead of being recentered, so its digits run
0..=2^last_bits and it takes 2^last_bits buckets — the full 2^c when c divides
the 253-bit modulus width, which is the width the size heuristic picks for
8,193 to 16,384 points.

Results are unchanged (exact group arithmetic), so proofs stay byte-identical.
Measured on an M4 Pro, release, medians of two runs: 274 points (the IPA's
closing MSM, present in every proof check) 525 -> 418 us, 400 points 580 -> 484,
1,000 points 880 -> 798, 2,335 points 2,046 -> 1,718, 8,192 points 4,884 ->
4,564. The band that gains most is exactly the one a mainnet head witness lands
in: its commitments coalesce to a few hundred to a couple of thousand points.
None of this depends on AVX-512, so every platform gets it.

`msm_windowed` now picks `c` and delegates to `msm_windowed_with_c`, which lets
the new equivalence test cover every window width (and with it every bucket-count
branch) at a size that keeps a debug run cheap; the existing size-driven test
still covers the parallel path. A second test pins the digit bounds the bucket
counts rest on, and an ignored timing test reports the sizes above.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`benches/proof.rs` creates every proof over one store and one key set, so from
the second sample on, the process-wide node-polynomial cache serves every node
and the benchmark reports a 100% hit rate — a number no caller sees, and one
that hides the cost of the stages that feed the cache (store reads, map-to-field
passes, their rayon chunking) from anything measured against it.

`salt_proof/create-cold` and `witness/create-cold` clear the cache before every
iteration, so each sample pays for the nodes on its own paths. The warm groups
stay: production sits between the two, at 44% on mainnet without the refresh and
98% with it, so the pair brackets it.

`clear_node_poly_cache` is the hook the benches call. It is `#[doc(hidden)]` —
nothing in production wants to drop the cache — and clearing a shard frees its
map after the lock is released, like the eviction path does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ffect for

An ablation of every mechanism this branch adds (one arm per mechanism switched
off, salt's own criterion benches, 32-thread EPYC) found three that are either
inert at mainnet witness shapes or too small to carry their cost:

- `ScalarWindows` (allocation-free wNAF digits in `mul_index`): nothing on proof
  creation, +1.3% / +2.4% on `update 10000 KVs` at 1 / 8 threads, the first of
  which sits on the benchmark's noise floor. It also removed
  `calculate_prefetch_index`, which the batched committer this branch is about to
  be merged with builds on. `salt_committer.rs` goes back to what `ad33ac8` left.
- `MIN_PARENT_CHUNK = 8`: a floor binds only below `floor x threads` items, and a
  mainnet head witness reads ~560 parent commitments, so it never bound under 70
  threads. The parent list is never empty, so no floor is needed at all.
- The `max(8)` on the multiproof aggregate and the `max(64)` on
  `parents_and_points`: no measurable effect in isolation. Both become `max(1)`,
  which is what keeps an empty input away from `chunks(0)` (a panic) — the
  verifier reaches `parents_and_points` with whatever key set it was handed, and
  `MultiPoint::open` is a library entry point.

`MIN_NODE_CHUNK` and `MSM_MIN_CHUNK` stay: the first also guards the all-hits case
(an empty rebuild list, the norm once the refresh lifts the hit rate to 98%), the
second binds on every proof once the pool has more than 16 threads, and what both
are for — several proofs sharing one pool — is not something a one-proof-at-a-time
benchmark can see.

Outputs are unchanged; proofs stay byte-identical.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…hook

An end-to-end replay-full run (5000 mainnet blocks, 5 alternating rounds
per arm) put numbers on what the isolated ablation could not:

- `MIN_NODE_CHUNK` and `MSM_MIN_CHUNK` are worth 1.3% of pipeline
  throughput with 8 concurrent witness builds and cost 1.4% with one.
  Both become `.max(1)`, which only keeps an empty input off `chunks(0)`.
- The cold-cache bench groups and the public `clear_node_poly_cache` hook
  they needed (reverts 4e613ce): the pipeline run now answers the question
  they were added for, so the crate does not need the extra public symbol.
- `msm_windowed_timing`, an ignored timing harness.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…anch

`NodePolyRefresh` and the three `create_with_refresh` entry points leave
this branch for `liquan/perf/proof-refresh`, which carries them on top of
it unchanged. The refresh is 336 lines of product code and about 850 of
tests, close to half of this PR, and the part that most deserves a review
of its own; end to end it is worth 12.5% of replay-full throughput on top
of the cache, which stays here and is worth 16% by itself.

Without a refresh the cache is filled by rebuilds alone, so a node the
previous block wrote is rebuilt once under its new commitment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@flyq flyq changed the title perf: faster witness proofs and state-root updates perf: faster witness proofs and proof checks Sep 21, 2026
@flyq
flyq marked this pull request as ready for review September 22, 2026 01:22
Reuse, simplification, efficiency and altitude passes over the diff
against main; outputs are unchanged and the proof byte-identity tests
pin that.

- `thread_chunk_size!` (salt-macros): the `div_ceil(num_threads).max(1)`
  idiom and its zero-chunk guard live once, not at four sites with four
  comments; every chunking site uses it.
- `CRS::committer` builds the `G ++ [Q]` tables the provers require, and
  `create_with_precomp` asserts that shape: its `q` parameter, the
  table-shape sniffing, the debug assert and the variable-base blinding
  fallback no caller reached are gone. The trie's shared committer is
  built through it.
- `Committer::msm` (banderwagon) replaces `ipa::fixed_base_msm`, which
  multiproof reached across modules for a committer operation.
- `create_sub_trie`: one `ParentNode` (logical id, physical id, bytes,
  decoded commitment) per parent, read in parallel in node order; this
  drops the second `parent_bytes` map, the lookup closure with an
  unreachable error arm, the repeated `connect_parent_id` calls and the
  hash lookups per node, and `resolve_polys` takes the slice.
  `internal_polynomials` returns one polynomial per node instead of a
  flat buffer the caller copied out of. Key validation groups the sorted
  keys with `chunk_by`.
- `open_inner` derives `a`/`b` once for both IPA paths;
  `open_point_outside_of_domain` is inlined into its only (test) caller.
- Deleted or narrowed: `multi_scalar_mul_par` (deprecated, no callers),
  `LagrangeBasis::values` and `DEFAULT_CRS` back to `pub(crate)`,
  `shared_committer` returns `&'static`, `apply_block` returns nothing,
  `msm_windowed` stops truncating to the shorter slice (its caller asserts
  equal lengths), `slot_position` is shared with `shape.rs`, the arkworks
  agreement tests share one input builder.

Bench (M4 Pro, warm cache) against the tree before this commit:
`salt_proof/create/{16,256,2048}` and `witness/create/{256,1024}` within
noise. An intermediate version read the parents with an unbounded
`par_iter`, which cost 2% on `witness/create/256`; the read is bounded to
one chunk per thread again via `with_min_len`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@flyq
flyq requested review from Troublor and yilongli September 22, 2026 03:32
…h the suppressions

The mutation gate's three survivors sat in the cache's eviction condition
(`len >= MAX_PER_SHARD && !contains_key`): nothing filled a shard. That
cannot be tested deterministically against a process-wide cache that every
proof test writes to, so the shards become `NodePolyCache`, with `CACHE` as
the one global instance, and a test fills a private instance: a full shard
keeps its entries when a node it holds is re-inserted (at a new commitment,
which retires the old one) and is cleared only by a node it does not hold.

The suppression hygiene check has failed on this branch since 9/7: three
entries named `multi_commitments_to_scalars` and `process_leaf_node`, which
this branch rewrote (the `<` guard is now in `leaf_polynomial`; the `*`
sizing arithmetic no longer exists), and four `create_node_aligned_chunks`
entries were pinned to lines that `DEFAULT_CRS` and `shared_committer`
shifted by eleven. Re-pointed and re-pinned; the orphan check passes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread salt/src/proof/subtrie.rs Resolved
flyq and others added 3 commits October 14, 2026 22:48
- crs.rs, salt_committer.rs: put `CRS::committer` and `Committer::msm` where they
  no longer split `commit_lagrange_poly` and `add_affine_point` from their docs
- multiproof.rs: assert each query polynomial has one coefficient per CRS generator,
  as the old `LagrangeBasis` addition did, and check the committer's base count
  before any proving work in `open_with_committer`
- subtrie.rs: document that the node-polynomial cache relies on callers keeping the
  store unchanged while a proof is created

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ommitment

The node-polynomial cache is process-wide and keyed by (node id, commitment),
so a polynomial rebuilt from a store whose trie and state disagree and filed
under the commitment it was read at poisoned every later proof that read that
commitment, including proofs from consistent stores.

On a miss, check the rebuilt polynomial with one fixed-base MSM against the
commitment and insert it only if they match; a mismatching polynomial is still
used for the current proof. Hits stay free.

Add a regression test: a witness from a store with block 2's state but block
1's trie is created first, then a witness from a consistent block-1 store must
still verify.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Resolve conflicts with #150:
- shape.rs: keep the per-chunk record_key_paths helper and port main's
  bounded main-trie walk (MAIN_TRIE_LEVELS - 1 steps + root debug_assert).
- subtrie.rs: keep the batched internal_polynomials, with main's u64
  subtraction before narrowing; METADATA_BUCKET_LEVEL is pub(crate) and
  the unused SLOT_INDEX_MASK stays removed; drop a now-redundant cast.
- mutants/suppressions.toml: re-pin create_node_aligned_chunks lines to
  the merged trie.rs (905/914/911/916).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.08088% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
salt/src/proof/subtrie.rs 98.68% 3 Missing and 4 partials ⚠️
ipa-multipoint/src/math_utils.rs 25.00% 2 Missing and 1 partial ⚠️
Files with missing lines Coverage Δ
banderwagon/src/element.rs 95.52% <100.00%> (+0.10%) ⬆️
banderwagon/src/msm.rs 95.60% <100.00%> (+10.41%) ⬆️
banderwagon/src/salt_committer.rs 97.52% <100.00%> (+0.58%) ⬆️
ipa-multipoint/src/crs.rs 88.78% <100.00%> (+1.02%) ⬆️
ipa-multipoint/src/ipa.rs 99.39% <100.00%> (+0.27%) ⬆️
ipa-multipoint/src/lagrange_basis.rs 88.60% <100.00%> (-1.48%) ⬇️
ipa-multipoint/src/multiproof.rs 100.00% <100.00%> (ø)
salt/src/lib.rs 100.00% <100.00%> (ø)
salt/src/proof/prover.rs 98.54% <100.00%> (+<0.01%) ⬆️
salt/src/proof/shape.rs 98.54% <100.00%> (+0.12%) ⬆️
... and 5 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

flyq and others added 3 commits November 12, 2026 23:01
…thout cloning the CRS

`commit_dense` built a `Vec<(usize, Fr)>` of 256 terms for every commitment
to feed `Committer::msm`, and the node-polynomial check in salt did the
same. `Committer::msm_dense` takes the scalars as a slice, base `i` for
scalar `i`, and both callers use it. `LagrangeBasis::values` becomes public
so salt can pass a polynomial's evaluations without copying them.

`MultiPoint::open` takes the CRS by value, but since `open_inner` borrowed
it, the path without a committer cloned all 257 points for the IPA, which
folds the generators in place. `open_inner` now takes a `Cow`, so `open`
moves its CRS as it did before this branch.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…he cache around it

Checking each rebuilt polynomial with its own 256-term fixed-base MSM
(7c709ef) cost about 0.85 ms of CPU per cache miss, which made a cold proof
ten times slower: 320 keys over about 550 nodes took 57 ms instead of 5 ms
on 14 cores, and 485 ms instead of 18 ms on one thread.

A call's rebuilt polynomials are now checked together: with one 128-bit
challenge per node, `commit(Σ r_i·poly_i)` is compared with `Σ r_i·C_i`.
The challenges come from a blake3 hash of every commitment and polynomial in
the batch, so the check needs no entropy source and works under `no_std`.
Only a batch that fails is checked node by node, which caches exactly what
the per-node check cached. The same proof now takes 7.7 ms cold (29.5 ms on
one thread); the warm path is unchanged.

The cache itself, following the review of this branch:

- A full shard evicted everything it held, so the leaves one proof touched
  took the paths every proof walks with them. It now evicts one polynomial
  per new node and passes over those used since it last considered them.
- `NodePolyCache` is public: `global()`, `capacity()`, `set_capacity()` and
  `clear()`. A capacity of zero turns the cache off, and proof creation then
  skips the check as well.
- The shards use a lock whose waiters park under `std`, as `Lazy` already
  does; `no_std` keeps `spin`.
- `resolve_polys` takes the cache as a parameter, so tests check on a cache
  of their own that a consistent rebuild is cached and served, and that an
  inconsistent one is not.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The proof benches repeat one input, so after the first iteration every node
polynomial comes from the cache and the miss path is never timed, which is
how a tenfold slowdown on it went unnoticed. `salt_proof/create-cold` and
`witness/create-cold` clear the cache before each iteration, through
`NodePolyCache::clear` now that the cache has a public handle (6d5298f
dropped these groups together with the hook they needed).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@vincent-k2026 vincent-k2026 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, no verifier check is skipped or weakened. Please get a quick ack from Troublor on the batch-check rework in 3bb7086, add negative tests that hit the new MSM / run-merge path, and note the 256 MiB default cache in the release notes.

@shanchen97 shanchen97 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@flyq
flyq merged commit c124b45 into main Nov 13, 2026
17 checks passed
@flyq
flyq deleted the liquan/perf/proof-opt branch November 13, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants