gap: TRT-LLM GPT-OSS TTFT omits eager host dispatch and rank waiting

[simone-chen]

This was generated by AI during triage.

Problem

• AISimulate underpredicts GPT-OSS-120B TTFT on TensorRT-LLM: matched unprofiled runs average 48.86 ms, versus 16.53 ms predicted.
• Reproduction: B200, TP2/EP1, ISL/OSL 1024, concurrency 4, length ratio 0.8, NumPy seed 0; 8 warmup and 40 measured requests per repetition. Serving uses TensorRT-LLM PyTorch backend 1.3.0rc14.
• The op-based model lacks an explicit graph/eager execution identity and host-dispatch timing boundary. Mixed/prefill traces show substantial CPU submission gaps and cross-rank waiting that the current GPU op composition does not represent.

Evidence

• Original reproduction: 240/240 measured requests match replay token vectors. Follow-up collection: 400/400 match across two-rank traces, lightweight CPU timing and same-allocation controls.
• For an 886-prefill + 3-decode step, MoE kernel sums are about 7.49 ms, close to the 8.28 ms prediction. Increasing the shared MoE table is not supported by this evidence.
• With nested module timing disabled, complete forward CPU calls still average about 30 ms. Rank-dependent fused all-reduce/norm durations include peer waiting; they are not standalone collective throughput measurements.
• Profiling affects timings. The sequential same-allocation control observes TTFT of 53.15/56.24 ms with module timing off/on; temporal drift is not isolated. Diagnostic runs are not replacement E2E baselines or publishable kernel rows.
• CPU/GPU overlap and cross-rank dependencies prevent simply adding CPU time, GPU kernel sums and collective waiting. The full client TTFT gap is not established as a host-only delta.

Proposed follow-up — deferred

• Define actual cuda_graph/eager identity and separate CPU-dispatch/GPU timing boundaries, including runtime, CPU/affinity, rank, capture success and raw-sample provenance.
• Coordinate Collector output, performance-data loading, SDK/Rust composition and Gym/config propagation. Preserve legacy rows without relabeling them, and keep missing eager coverage visible.
• Validate dependency-aware composition using paired measurements and independent unprofiled workloads; retain graph-decode TPOT accuracy. Do not fit an E2E offset or publish rc14 timing as rc20 data.
• Replay all 31 GPT-OSS/B200 points; rerun all 263 points if shared timing/config code changes, retaining every regression.
• Current decision: retain evidence and defer the data-contract extension. This issue tracks unresolved work; no implementation is approved and TTFT is not fixed.

References

• Related PR: fix: correct TRT-LLM MLA labels and GPT-OSS MXFP4 profiles #264. Its precision/data corrections do not resolve this gap.
• Trace report and measurement boundaries.
• Compact measurements and artifact hashes.
• Deferred measurement-contract proposal.

TTFT tracking slides

• Issue detail.
• TTFT issues summary.
• Shared graph/eager and CPU-dispatch contract proposal: feat: define graph/eager and CPU-dispatch timing contracts #286. This is the implementation follow-up to this measured case, not an independent error contribution.