Repository navigation
[Question/Performance] Full message history is re-tokenized on every coding-agent turn #2285
Description
Background
The current coding-agent adapters receive OpenAI/Anthropic messages from
Claude Code or Codex and render the full message history on every turn:
https://github.com/THUDM/slime/blob/main/slime/agent/adapters/common.py
translated, tools_schema = self._translate(body)
prompt_ids = _render_token_ids(
translated,
tokenizer,
tools=tools_schema,
add_generation_prompt=True,
)_render_token_ids() calls tokenizer.apply_chat_template(..., tokenize=True)
for the complete message list.
I understand that the exact output_ids and rollout logprobs returned by
SGLang are preserved by TrajectoryManager, and that token drift is handled by
realign/fork/masking. Therefore this question is about CPU scalability rather
than token provenance correctness.
Concern
For a long-horizon agent trajectory, every request normally contains the
complete and growing message history.
If the prompt lengths over N turns are:
L1, L2, ..., LN
the adapter performs approximately:
L1 + L2 + ... + LN
tokens of CPU tokenization work.
When the context grows approximately linearly with the number of turns, this
becomes close to O(N^2) cumulative tokenization work per trajectory.
This may become significant for:
-
128K to 1M-token coding-agent trajectories;
-
many concurrent agent sessions sharing one adapter;
-
tool observations containing large source files or compiler logs;
-
repeated tool schemas and system prompts;
-
SGLang prefix-cache hits that make GPU prefill cheaper while CPU still
re-renders and re-tokenizes the full prompt; -
auto-compaction, sub-agent and retry paths.
The tokenization call is also synchronous in the aiohttp request path. A long
tokenization operation may therefore block other sessions served by the same
adapter event loop.
Questions
-
Is full-history re-tokenization on every coding-agent turn the intended
design? -
Has its CPU time, event-loop blocking time or throughput been benchmarked
for long contexts and concurrent sessions? -
Are there existing metrics for:
- chat-template rendering latency;
- tokenizer latency;
- number of input tokens re-tokenized per turn;
- adapter queueing latency;
- tokenizer CPU utilization?
-
Is there a plan to support session-scoped incremental tokenization or an
append-only token checkpoint, similar to a session-server TITO path? -
Was the removed radix-tree middleware from [Feature] Support token in token out for multi turn tasks #242/[slime-router] support pd disaggregation and remove radix tree middleware #1735 considered for the
coding-agent adapter, or was it removed because its consistency, memory or
maintenance cost was too high?
Possible directions
Some possible approaches, with full rendering retained as a correctness
fallback:
-
Keep a per-session token checkpoint and normalized message prefix.
-
When the next request is append-only and the tokenizer/chat-template/tools
configuration is unchanged, tokenize only the appended messages and merge
them with the checkpoint. -
Validate the incremental result against template-specific invariants and
fall back to full rendering for retries, compaction, rewrites, tool-schema
changes or ambiguous token boundaries. -
Move unavoidable full tokenization to a bounded worker pool so it does not
block the adapter event loop. -
Add latency and processed-token metrics before selecting an optimization.
A naive concatenation of independently tokenized message fragments may be
incorrect because of tokenizer boundary behavior and chat-template suffixes,
so this likely requires a model/template-aware implementation rather than
simply caching tokenizer(message) results.
Related work
- Feature: Token-in/Token-out for RL Sampling #117: token-in/token-out proposal
- [Feature] Support token in token out for multi turn tasks #242: radix-tree based multi-turn TITO
- [slime-router] support pd disaggregation and remove radix tree middleware #1735: removal of the radix-tree middleware
- feat(examples): add strands-sglang integration for agentic RL with TITO support #1359: native TITO agent loop through strands-sglang
- [coding-agent-rl] Refactor coding-agent RL: turn-node TrajectoryManager + pluggable harness layer #2005: current coding-agent TrajectoryManager and retokenization-drift handling
Could the maintainers clarify whether this CPU cost is already considered
acceptable, has been measured, or is planned to be optimized?
What I've Tried
- I have read the relative dissusion and docs, but not find any performence issues
Environment (if relevant)
- slime version:
- Python version:
- PyTorch version:
- CUDA/ROCm version:
- GPU type and count:
- OS:
Additional Context
No response
Pre-submission Checklist
- I have read the CONTRIBUTING.md and understand the collaboration scope.
- I have read the documentation and FAQ and my question is not answered there.
- I have searched for existing issues and my question has not been asked before.
Activity
liuhao-labs commentedon Aug 23, 2026
Hi, I confirmed this is still present on current
main:BaseAdapter._run_turn()synchronously renders and tokenizes the complete translated history inside the aiohttp event loop.TrajectoryManagerpreserves token provenance for training after a turn, but it is not a prompt-construction cache.I would like to propose a conservative first step rather than a generic incremental token cache: keep full-history
apply_chat_template(..., tokenize=True)as the source of truth, move that blocking call to an adapter-owned bounded worker behind a cancellation-safe tokenizer guard, and add focused timing/token-count observability. This would remove event-loop blocking while preserving the exact prompt IDs and all current compaction, retry, sub-agent, tool-schema, and re-tokenization-drift behavior. It would not claim to remove the O(N^2) total tokenization cost.I would intentionally avoid suffix token concatenation or a mutable per-session token checkpoint here. Generic chat templates can rewrite earlier history, tokenizer boundaries are not compositional, and concurrent or disconnected requests make cache advancement and invalidation subtle. #2287 also has an active, Qwen-specific append-only design in this area, so I would keep this change orthogonal.
Would this full-render offload plus observability be a useful scoped PR, or would you prefer to wait for the Qwen-specific history work before changing the adapter hot path? I can cover it with CPU-only aiohttp tests that prove the event loop remains responsive, cancellation does not exceed the worker bound, and the exact input IDs sent to SGLang are unchanged.
zhuyijie88 commentedon Aug 26, 2026
Hi liuhao-labs, thanks for the careful proposal. I think this is a useful and appropriately scoped first step, and it can proceed independently of the Qwen-specific append-only work in #2287.
Please keep full-history apply_chat_template(..., tokenize=True) as the correctness source of truth, and move the blocking render/tokenization work behind a bounded adapter-owned worker mechanism. The implementation should preserve exactly the same prompt_ids and current trajectory semantics across compaction, retries, sub-agent calls, tool-schema changes, and token drift. Cancellation must not allow tokenizer work or worker concurrency to grow without bound, and a canceled request must not trigger a later SGLang generation or write state into the session.
Please also add focused observability for prompt length, render/tokenization latency, tokenizer queue wait, worker utilization, event-loop lag, cancellation/discard counts, and relevant P50/P95/P99 values. CPU-only tests comparing synchronous and offloaded paths should verify identical input IDs, bounded concurrency, responsive event-loop behavior, and cancellation safety.
This should be described as an event-loop isolation and observability improvement, not as a solution to the cumulative O(N²) tokenization cost. A later model/template-specific append-only TITO optimization can build on these measurements without changing this conservative fallback path.
liuhao-labs commentedon Aug 26, 2026
Thanks for confirming the scope. I implemented the proposed first step in #2328.
The PR keeps full-history
apply_chat_template(..., tokenize=True)as the correctness source of truth, moves prompt rendering to a bounded adapter-owned worker, and ensures canceled requests cannot later call SGLang or write session/trajectory state. It also exposes the requested prompt-length, render-latency, queue-wait, utilization, event-loop-lag, cancellation/discard, and P50/P95/P99 metrics atGET /debug/adapter_metrics.The CPU coverage checks exact prompt-ID equivalence, event-loop responsiveness, bounded admission, cancellation races, and cleanup behavior. The latest PR run is green with 63 successful checks and 9 conditional skips. As requested, the PR describes this as event-loop isolation and observability rather than a solution to the cumulative O(N^2) tokenization cost.
The PR is ready for review: #2328.
perf(agent): isolate prompt rendering from event loop
perf(agent): isolate prompt rendering from event loop