Skip to content

fix(state): read a bucket's used-slot count from the store once per state - #161

Open
flyq wants to merge 6 commits into
mainfrom
liquan/fix/witness-used-count
Open

flyq wants to merge 6 commits into
mainfrom
liquan/fix/witness-used-count

Conversation

@flyq

@flyq flyq commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replaying a block on its witness rescanned a whole bucket on every insertion when the block resized that bucket. A stateless validator does this replay to derive the block's state updates.

EphemeralSaltState asked the store for a bucket's used-slot count on every insertion into an empty slot, and a resize reinserts every key of the bucket. A witness answers by looking up every slot of the bucket, so the replay cost capacity × keys lookups.

Every one of those calls returned the same answer. The store does not change while a state borrows it, and the state adds its own insertions and deletions on top of the store's count (usage_count_delta). This PR reads the store's count once per bucket per state and keeps it.

An earlier version of this PR cached the counts in Witness. The cache now lives in EphemeralSaltState, so every StateReader gets the fix and Witness is unchanged from main.

What changes

  • EphemeralSaltState gains a pub base_usage_count field. usage_count asks the store on first use and reuses the answer afterwards.
  • A failed read is not kept: the store is asked again, and its error reaches the caller unchanged. A partly witnessed bucket therefore behaves exactly as on main.
  • Witness, SaltWitness::bucket_used_slots, the witness bytes, proofs, state updates and state roots are unchanged.
  • SaltWitness::metadata returns InvalidFormat for a metadata slot witnessed as empty, where it hit unreachable!. The proof check rejects such a witness, so this only matters when a witness is read before it is verified.

Behaviour change to review

EphemeralSaltState now relies on the store not changing while the state borrows it.

usage_count_delta already needed that: a change applied to the store under a live state would be counted twice. But the count was re-read on every call, and two tests (test_usage_count, test_metadata) wrote to the store under a live state and expected the count to follow. They now read each change to the store through a new state.

A caller that writes to the store under a live state and then inserts through that state would get the count from the first read.

Measurements

Release build, Apple M4 Pro. One bucket is filled to its resize threshold (the real 80% load factor, test-bucket-resize with NUM_DATA_BUCKETS=1), then a block of 8 inserts resizes it. Timed: EphemeralSaltState::update_fin of that block on the block's witness. The PR columns are the best of 10 runs (3 for the two largest rows); the main column is a single run. Every run returns state updates equal to the full store's.

Resize Keys in the bucket main This PR, Witness This PR, bare SaltWitness
512 → 1024 409 2.05 ms 0.32 ms 0.33 ms
4096 → 8192 3,276 177.6 ms 4.06 ms 3.92 ms
16384 → 32768 13,107 5.40 s 18.2 ms 17.9 ms
65536 → 131072 52,428 105.7 s (earlier run) 95.3 ms 93.1 ms
131072 → 262144 104,857 not run 212.3 ms 204.4 ms

The first three main numbers were measured with the same harness in the same session. The 65536 number is from the earlier version of this PR and was not rerun.

What remains after the fix is the reinsertion itself, linear in the bucket's keys.

A block that resizes nothing costs what it does on main. No pass over the witnessed keys runs when a Witness is built, and an insertion into a partly witnessed bucket still scans up to the first slot that is not witnessed.

Testing

All of these pass on this head:

  • cargo test
  • cargo test --features test-bucket-resize with NUM_DATA_BUCKETS=2 and BUCKET_RESIZE_LOAD_FACTOR_PCT=1
  • cargo test --no-default-features, with and without test-bucket-resize
  • the random stress test with CI's settings
  • cargo check -p salt --target riscv64imac-unknown-none-elf --no-default-features
  • cargo clippy --all-targets -- -D warnings and cargo fmt --all -- --check
  • scripts/mutation_test.sh diff origin/main: 3 of 3 viable mutants killed, no timeouts, none unviable

Not run locally: the i686 job and the spec-gate packs. The canon pack targets proof/salt_witness.rs, where it mutates sort and dedup calls and comparators; the lines changed there have none.

New and changed tests:

  • test_usage_count_reads_the_store_once_per_bucket uses a store that counts its bucket_used_slots calls. Insertions that resize a bucket, then canonicalization, read the store's count once. Removing the line that keeps the count fails this test.
  • test_usage_count_does_not_keep_a_failed_read checks that a store that cannot serve the count is asked again by each insertion, and that a rehash returns the store's own error.
  • resize_block_replayed_on_the_witness_matches_the_store (needs test-bucket-resize) replays a block that resizes a bucket on its witness and gets the full store's state updates and root.
  • test_malformed_metadata_is_an_error covers a metadata slot witnessed as empty, undecodable metadata, and a capacity of u64::MAX, for metadata and for the used-slot scan. It replaces test_metadata_none_entry_panics.
  • test_empty_metadata_slot_fails_the_proof_check shows that a real witness whose metadata slot is emptied fails verify_proof.
  • test_witness_bucket_used_slots is restored from main.

Consumers

Nothing was compiled or run against either consumer.

  • stateless-validator derives a block's state updates by writing through EphemeralSaltState over a verified salt::Witness, so it needs the new salt revision and no code change. Its LightWitness is not affected: nothing in stateless-validator writes through it (it serves execution and tracing, which only read), and a read never asks for a bucket's used-slot count.
  • mega-reth needs no code change. The new field is pub, like the struct's other fields, so code outside the crate that builds the state from its fields keeps compiling.
  • A caller that keeps one state alive across many updates, such as a block builder, now reads each bucket's count once instead of once per insertion. That is the kind of caller the behaviour change above reaches, and it is worth a look from whoever owns such a path.

🤖 Generated with Claude Code

`Witness::bucket_used_slots` forwarded to `SaltWitness`, which looks up every
slot of the bucket in the witness on each call. `EphemeralSaltState` asks for
the count on every insertion into an empty slot, and a resize reinserts every
key of the bucket, so replaying a block that resizes a bucket on its witness
cost capacity x keys lookups: 5.2 s for a 16384-slot bucket and 106 s for a
65536-slot one (release build, Apple M4 Pro).

The witnessed slots do not change once the witness is built, and the state
adds its own insertions and deletions on top of the store's count, so every
call returned the same answer. Count it once when the witness is built, in one
pass over the witnessed keys, and serve it from a map: 20 ms and 99 ms for the
same two blocks.

The answers are those of the scan: a count only for a data bucket whose
metadata and every slot below its capacity are witnessed, an error otherwise,
and `Ok(0)` for metadata buckets and out-of-range ids. The pass runs on
witnesses that are not verified yet, so it never panics on malformed metadata.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Mutation testing - PASS

Nothing to test: no viable mutants were generated.

@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.80420% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
salt/src/state/state.rs 92.94% 6 Missing ⚠️
Files with missing lines Coverage Δ
salt/src/proof/salt_witness.rs 98.72% <100.00%> (+0.29%) ⬆️
salt/src/proof/witness.rs 94.39% <100.00%> (+0.36%) ⬆️
salt/src/state/state.rs 94.90% <92.94%> (-0.06%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Performance Benchmark Comparison

Compared 5 benchmark(s) against the latest main baseline.

Detailed Comparison
Benchmark Baseline Throughput (Kelem/s) New Throughput (Kelem/s) Change
update 10000 KVs/1 threads 83.52 76.43 -8.49%
update 10000 KVs/2 threads 160.32 136.45 -14.89%
update 10000 KVs/4 threads 299.94 249.20 -16.92%
update 10000 KVs/8 threads 490.69 432.26 -11.91%
update 10000 KVs/16 threads 617.42 577.19 -6.52%

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Mutation testing - PASS

Nothing to test: no viable mutants were generated (3 unviable, 0 timed out).

@shanchen97 shanchen97 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two notes on the precompute. The cache should stay; the scan it replaces should not come back.

Comment thread salt/src/proof/witness.rs Resolved
Comment thread salt/src/proof/witness.rs Outdated
Comment on lines 331 to 345
/// Returns the occupied-slot count of a bucket whose slots are all witnessed.
///
/// Same answers as [`SaltWitness::bucket_used_slots`] in constant time: resizing a bucket
/// reinserts every key, and each insertion asks for this count.
fn bucket_used_slots(&self, bucket_id: BucketId) -> Result<u64, Self::Error> {
self.salt_witness.bucket_used_slots(bucket_id)
if !is_valid_data_bucket(bucket_id) {
return Ok(0);
}
self.fully_witnessed
.get(&bucket_id)
.copied()
.ok_or(SaltError::NotInWitness {
what: "Bucket slots",
})
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the scan's answer. SaltWitness::bucket_used_slots returns NotInWitness { what: "Metadata" } when metadata is missing, NotInWitness { what: "Key" } when a slot is missing, InvalidFormat when metadata does not decode, and panics when metadata is witnessed as None. Every failure here is NotInWitness { what: "Bucket slots" }, and the panic becomes Err. The PR body records that difference; this doc says the opposite.

bucket_used_slots_matches_a_scan_of_the_witness compares .ok(), so two different errors pass. It locks the Ok counts, not "same answers". test_witness_bucket_used_slots still says this method delegates to SaltWitness; it no longer does, and that comment is what someone will follow when they delete the cache.

Write the contract this function actually has. A fully witnessed data bucket returns the occupied-slot count. A metadata bucket or an out-of-range id returns Ok(0). Everything else is NotInWitness { what: "Bucket slots" }, including metadata the scan would panic on. Ok matches the scan and the error variant does not, on purpose: shi_upsert only tests Ok, and shi_rehash forwards the error. Leave SaltWitness::bucket_used_slots as the scanning implementation. Its tests lock the "Metadata" and "Key" variants, and this PR should not change them.

fully_witnessed reads as a set and holds a count, and both From and Witness::create fill it by hand. A private constructor and a name that says count (used_slots) puts that derived field in one place.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d535f9c.

  • Contract. The doc on Witness::bucket_used_slots now says what the function returns. A data bucket whose metadata and slots 0..capacity are all witnessed gives the occupied-slot count. A metadata bucket or an out-of-range id gives Ok(0). Anything else is NotInWitness { what: "Bucket slots" }, including metadata the scan panics on. It states that only the Ok results match the scan and that the errors differ on purpose, with the shi_upsert / shi_rehash reason.
  • Tests. bucket_used_slots_matches_a_scan_of_the_witness still compares the Ok counts with the scan, and now asserts the exact error for every unknown count instead of .ok(). building_a_witness_from_malformed_metadata_does_not_panic asserts the same error for its three malformed-metadata cases.
  • Stale comments. test_witness_bucket_used_slots no longer says the method delegates to SaltWitness.
  • Field. It is used_slots, private, and filled only in Witness::new, which From<SaltWitness> and Witness::create both call.

SaltWitness::bucket_used_slots and its tests are untouched; the PR only changes proof/witness.rs. I also updated the PR body: it said the field was pub(crate) and that this method "gives the scan's answers".

… one place

Review follow-up for the precomputed used-slot count.

- Walk each bucket's run of witnessed keys with one cursor, and write down
  why `capacity` keys below the capacity are exactly the slots `0..capacity`.
- Document what `Witness::bucket_used_slots` returns: only its `Ok` results
  match the scan in `SaltWitness::bucket_used_slots`; every unknown count is
  `NotInWitness { what: "Bucket slots" }`. Tests now pin that error.
- Rename the field to `used_slots`, make it private and fill it in
  `Witness::new`, which both `From<SaltWitness>` and `Witness::create` use.
- Fix test comments that still said the method delegates to `SaltWitness`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@shanchen97 shanchen97 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

flyq and others added 4 commits November 13, 2026 19:53
Resolve the conflict in salt/src/proof/witness.rs: both sides appended
tests at the end of the test module; keep both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fully_witnessed_usage: look up a bucket's metadata only when its first
  witnessed slot is slot 0 (a decoded capacity is at least 1, so no other
  bucket can be fully witnessed), and group each bucket's run with
  peek + next_if instead of a hand-managed cursor.
- Trim the bucket_used_slots doc to the contract.
- Tests: fold the malformed-metadata cases into
  bucket_used_slots_matches_a_scan_of_the_witness and cross-check every
  case against the scan; drop test_witness_bucket_used_slots, which that
  test covers; use apply_block, mock_salt_value and the module imports;
  build the plain-key test witness through Witness::from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tate

EphemeralSaltState asked the store for a bucket's used-slot count on every
insertion into an empty slot. A witness answers by looking up every slot of
the bucket, and a block that resizes a bucket reinserts all of its keys, so
replaying such a block on its witness cost capacity x keys lookups.

Keep the store's count of each bucket after the first successful read, in
`base_usage_count`. The store does not change while a state borrows it,
which `usage_count_delta` already relies on, so the count is fixed for the
state's lifetime. A failed read is not kept: the store is asked again and
its error reaches the caller unchanged.

This replaces the count cache in `Witness`, whose code is main's again.
Every `StateReader` now gets the fix: `Witness`, a bare `SaltWitness`, and
readers outside this crate.

The field is `pub` like the others, because code outside the crate rebuilds
the state with struct-update syntax.

`test_usage_count` and `test_metadata` wrote to the store under a live
state and expected the count to follow. They now read each change to the
store through a new state.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`SaltWitness::metadata` hit `unreachable!` when a witness mapped a metadata
key to `None`. A witness builder never produces that, and the proof check
rejects it, but a witness can be read before it is verified. Return
`InvalidFormat` instead, so `metadata` and the used-slot scan that reads it
report a malformed witness without panicking.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@flyq flyq changed the title fix(proof): count a witnessed bucket's used slots once per witness fix(state): read a bucket's used-slot count from the store once per state Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants