Track mocker/replay correctness and resource-efficiency fixes

[PeaBrane]

Track ten mocker/replay findings from a read-only review of main at fbd465d9ac14d37a6e73371ba4736a9364371eb5. The scope covers the simulated engine, scheduling/cache behavior, replay/load generation, and replay reporting. AIC kernel collection and timing-model work are excluded.

Fifteen focused reviewers inspected the scope; the findings below were consolidated and checked against callers, contracts, and existing tests. No patches, regression tests, or benchmarks were run. These are high-confidence static findings; the triggers and regression checks below still need execution during implementation. Performance magnitudes are derived examples, not measurements.

The first eight candidates have small, localized patch directions. The last two are supported liveness bugs whose fixes need more careful behavioral validation. Check off an item when a linked fix and its validation are complete.

Small-fix candidates

• 1. SGLang prefill uses the configured chunk budget instead of the remaining budget.

  • Code: prefill.rs:103.
  • Trigger: block size 4, chunk budget 8, ample KV and request slots, and queued disjoint prompts of lengths 4 and 8. The second request is rejected for this pass instead of using the remaining four-token chunk. With block size 4, chunk budget 6, and prompt length 5, the page-rounded charge is 8 and admission never progresses.
  • Proposed fix: compare page-rounded input against the remaining chunk budget before selecting full versus partial prefill.
  • Regression check: the 4+8 case admits both with four materialized tokens each; the page-four/chunk-six/prompt-five case completes.

• 2. Missing timestamps collapse real Mooncake arrival gaps.

  • Code: trace.rs:863; public native path: python.rs:1049.
  • Trigger: one row omits its timestamp and another has timestamp 100 ms. Normalization excludes the omitted value when finding the origin, then the driver maps it to zero: both requests arrive at zero.
  • Proposed fix: include implicit-zero arrivals when choosing the normalization origin, consistent with the Python materializer.
  • Regression check: after native loading and normalization, only the first request is ready at zero and the next arrival is 100 ms; preserve all-explicit and all-missing cases.

• 3. Replay inflates accept length on ordinary mixed prefill/decode passes.

  • Code: components/engine.rs:1134.
  • Trigger: one existing decode and one new prefill both emit a token, but the denominator counts only the one FPM decode request. Replay reports accept length 2 with speculation disabled and propagates it to policy snapshots.
  • Proposed fix: count distinct non-rejected token-emitting request IDs for the denominator, with the same filter for the numerator. This matches the existing scheduler accept-length contract.
  • Regression check: the mixed ordinary pass reports 1.0; a multi-token speculative burst still reports the correct ratio.

• 4. SGLang advertises reusable KV when prefix caching is disabled.

  • Code: sglang_backend.rs:908; unconditional allocation publication at line 302.
  • Trigger: supported configuration enable_prefix_caching=false, emit_kv_events=true, with complete prompt blocks. Observers receive Stored events even though another request cannot reuse those blocks.
  • Proposed fix: suppress publishers when caching is disabled, matching vLLM. The event contract defines Stored as becoming prefix-cache visible.
  • Regression check: capture emits no Stored events, and an overlapping matching request still reports zero reuse.

• 5. Aggregated vLLM repeatedly scans all requests for nonexistent handoffs.

  • Code: core.rs:1105; admission call at line 1793, retry calculation before the pending-work check at line 1027.
  • Trigger: ordinary aggregated replay has no destination handoffs, but admission attempts and pass completions still scan the request map for a necessarily zero result. Queued traffic under KV pressure amplifies this per decode pass.
  • Proposed fix: return zero headroom when both destination-owner collections are empty; check pending destination work before calculating headroom.
  • Regression check: preserve admission/completion outcomes for aggregated and disaggregated runs; verify the no-handoff path avoids the scan.

• 6. Sparse traces incur quadratic completion checks.

  • Code: driver.rs:1937.
  • Trigger: ordered one-turn sessions whose arrivals are spaced beyond each request's completion. Each idle gap scans the growing completed-session prefix; repeated checks give quadratic session visits.
  • Proposed fix: return false early when the ready-queue head proves an unfinished session exists, retaining the original scan as a fallback for stale entries and agentic states.
  • Regression check: preserve timings, counts, and terminal outcomes; compare visit counts or profile N versus 2N sparse sessions to verify scaling.

• 7. Finished delta sessions retain dead cumulative prompt buffers.

  • Code: driver.rs:1875; history materialization at line 1474.
  • Trigger: mooncake-delta sessions finish, but their cumulative Vec<u32> buffers survive until the whole replay ends. For example, 10,000 histories of 64,000 tokens retain about 2.56 GB in this additional storage.
  • Proposed fix: release the allocation when session_ended is true; clear() alone retains capacity. Preserve live-session histories and reporting metadata.
  • Regression check: completed, rejected, canceled, and failed sessions release capacity; live sessions preserve exact subsequent token identities. Eager source materialization is outside this fix.

• 8. Radix splits retain oversized prefix-vector allocations.

  • Code: radix_cache.rs:813.
  • Trigger: split_off leaves the original edge's backing capacity in each new prefix node. Repeated ascending splits retain capacities P, P-d, P-2d, etc. A 4,096-page edge split at successive eight-page prefixes retains about 16 MiB of key/page vector storage instead of 64 KiB, without adding physical cache pages.
  • Proposed fix: shrink the two new prefix vectors before inserting the intermediate node.
  • Regression check: cache contents, locks, node identity, and eviction order stay unchanged; retained vector capacity follows stored elements. Account for extra reallocation cost at split time.

Liveness bugs requiring more careful patches

• 9. Speculative vLLM can endlessly preempt an exact-capacity request.

  • Code: reservation at core.rs:2726 and allocation at line 2835.
  • Trigger: block size 4, two GPU blocks, prompt length 8, output length 1, speculative decoding enabled (aic_nextn=Some(1)), and unset model-length cap. Speculative reservation demands a third block for the terminal, uncomputed output token. The request preempts itself, requeues unchanged, and repeats.
  • Intent evidence: the ordinary path already has an exact-capacity terminal-sample regression test explicitly prohibiting this livelock.
  • Proposed fix: reserve only through the final computed token and coordinate allocation with intermediate-token finalization. Changing reservation arithmetic alone is insufficient.
  • Regression check: extend the existing geometry to speculation; require one terminal token, zero preemptions, and released active KV. Preserve multi-token bursts crossing a block boundary.

• 10. Positive-duration G3 restores can alternate evictions forever when the prefix cannot fit in G2.

  • Code: g3_offload.rs:55; host lookup deferral.
  • Trigger: warm G3 contains prefix [A,B], G1 misses both, G2 holds one block, and reads take positive virtual time. Promoting B evicts A; promoting A then evicts B. Every advancing timestamp clears attempt tracking, and lookup keeps deferring before H2D or cold computation.
  • Proposed fix: detect structurally impossible full-prefix promotion and allow bounded partial-prefix/cold fallback. Preserve intentional ordinary eviction and retries after temporary pin pressure.
  • Regression check: drive a bounded number of virtual deadlines with sufficient GPU capacity; require request completion and finite reads. Existing zero-duration re-promotion coverage does not exercise this positive-duration cycle.