Add experimental cycle-consistent replica selection and measured comparison

[aneubeck]

Summary

Add allocation-free cycle-consistent replica selection alongside the existing streaming ConsistentPermutation, preserving its API and mappings:

• VirtualPermutation: the stronger, one-bit-lift implementation, 24-byte state, 1..=u64::MAX, direct replica_at(slot) and non-replaying nth.
• BalancedVirtualPermutation: an explicitly experimental matched-network comparison requested during review. It calls the exact existing Feistel, with the same seed, mixer, round counts and key schedule, using even-width/two-bit lifts over 1..=2^30. It also has 24-byte state. The new inverse shares the round function and reverses that exact key schedule; known-answer vectors guard the existing mapping.

Different membership semantics, not drop-in replacements: the existing iterator preserves survivor list order. Both cycle variants preserve each old replica slot except a slot replaced by an appended node, or previously naming a removed last node. All provide distinct nodes and stable k prefixes. Membership must remain consecutive IDs 0..n.

The documentation derives uniformity and expected O(k) evaluation under independent uniform ideal permutations and constant-cost forward/inverse calls. Actual finite-key networks are noncryptographic approximations: no exact uniformity, cryptographic security or worst-case O(k) guarantee is claimed. No rings, permutation arrays, duplicate sets, recursive stack or mapping-changing retry limits are used.

Latest three-way measurements

Apple M4 Max, native release build, rustc 1.92.0; same seeded corpus and setup accounting for every method. All three were rerun together: 131 N/k cases each for fresh and prehashed queries, plus constructors, setup-excluded streaming, equal-width collection and single-slot workloads. All comparisons fit the existing domain. Each case has 20 samples; the committed report includes 95% intervals, variance, seeds, environment and commands. The runner produces 1,081 estimates.

Mean ns/query:

Workload	Existing	Stronger one-bit	Matched two-bit
Fresh N=1,000, k=3	32.83	101.72	27.72
Fresh N=65,536, k=3	33.12	83.63	28.43
Fresh N=65,537, k=3	65.87	262.49	171.81
Fresh N=1,000, k=1,000	9,808.45	83,713.94	22,289.60
Stream only N=1,000, k=3	14.45	89.51	20.95
Prehashed slot 999, N=1,000	9,604.66 by replay	77.43 direct	12.69 direct

Matching the primitive and layer stride removes much of the earlier slowdown: at N=1,000,k=3 the cycle implementation improves from 101.72 to 27.72 ns, about 16% faster than existing. But setup-excluded streaming still favors existing, consistent with the complete-query win coming from avoiding allocation rather than cheaper traversal. The matched mean is below existing in 31/131 fresh cases, including the degenerate N=1 case; small differences are not blanket significance claims. Its worst ratio is 3.45x at N=257,k=8, and full enumeration remains slower. This is not an isolation of the cost of odd halves alone.

Statistical tradeoff — do not treat the matched variant as equivalent

Both primary and disjoint held-out 200,000-key corpora were rerun without tuning the inherited network. The matched variant has repeatable small-domain bias:

Diagnostic	Degrees of freedom	Primary chi-square	Held-out chi-square
N=5, slot 4	4	82.80	84.94
N=8, ordered slots (0,1)	55	112.66	131.31
N=8, first-three subset	55	133.99	146.63

For comparison, the N=8 subset statistics are 53.47/59.83 for existing and 47.66/40.41 for the stronger variant. At N=5,slot=4 the matched most frequent node is about 4% above its expected count in both corpora. The same primitive can behave differently after list versus cycle projection. The matched variant remains explicitly experimental, not a silent replacement for the stronger implementation or a claim that the ideal randomness goal is met. Existing/stronger diagnostic output and mappings remain unchanged. Statistical diagnostics are not CI p-value gates or proofs of independence.

Documentation and reproduction

• Algorithm, semantics, ideal-model proof and matched two-bit construction
• Measured comparison, intervals, state costs, statistics and traversal tails
• cargo bench -p consistent-choose-k-benchmarks --bench replica_comparison -- --noplot
• python3 crates/consistent-choose-k/benchmarks/summarize_replica_comparison.py target/criterion
• cargo run --release -p consistent-choose-k --example permutation_diagnostics (repeat with -- --held-out)

Original two-way results at commit d51abfb remain documented as historical measurements. Wider u64 cases are explicitly unpaired operation-count diagnostics, not speed comparisons with the old API.

Validation

Debug and release crate suites pass: 34 unit tests + 4 doctests in each, plus the separately run operation-count diagnostic. Coverage includes full permutations, prefixes, append/delete consistency, power-of-two/four boundaries, explicit lift/cycle-projection oracles, ideal lift fibers, every supported forward/inverse width and extreme values, unchanged-network known answers, invalid inputs and long walks. All-target/all-feature builds, strict targeted Clippy, workspace formatting and diff checks pass. Rustdoc builds with the pre-existing private-link warning in existing documentation.

All work and benchmarks ran in the isolated worktree. Local commands used the already-installed Command Line Tools via DEVELOPER_DIR=/Library/Developer/CommandLineTools because selected Xcode's license was unaccepted; no license or system setting was changed. No merge requested.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The novel permutation construction and its probabilistic guarantees warrant final expert human validation despite strong tests and documentation.

Review effort: Balanced
Findings: None

What changed in this PR

Adds experimental cycle-consistent replica selection with direct slot lookup, constant iterator state, extensive validation, and measured comparison against the existing policy.

Changes:

• Adds and exports VirtualPermutation.
• Adds correctness and distribution diagnostics.
• Adds reproducible benchmarks and design/performance documentation.

File	Description
src/​virtual_permutation.rs	Implements and tests cycle-consistent selection.
src/​lib.rs	Exports the new API.
README.md	Explains differing membership semantics.
examples/​permutation_diagnostics.rs	Adds distribution diagnostics.
docs/​virtual-permutation.md	Documents algorithm and guarantees.
docs/​virtual-permutation-performance.md	Records methodology and results.
docs/​permutation-design.md	Distinguishes existing semantics.
benchmarks/​summarize_replica_comparison.py	Summarizes Criterion results.
benchmarks/​replica_comparison.rs	Adds paired benchmarks.
benchmarks/​Cargo.toml	Registers the benchmark.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.