Make the SDK own the whole CLI process tree - #2448
Conversation
`Client::stop`, `force_stop`, and `Drop` only ever reached the root CLI
process (`kill`/`start_kill` on the tracked `Child`). Anything the CLI
spawned as a descendant (an MCP server, a shell tool, a subagent process)
was left running, still holding whatever it held, whenever teardown ran
solely at the root.
Give the SDK a small process-tree containment primitive and route all
three teardown paths through it:
- Unix: `Command::process_group(0)` puts the spawned CLI in its own
process group at fork, before `exec`; termination signals the whole
group with `SIGKILL` via `killpg`.
- Windows: a private Job Object carrying
`JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` is created and the root process is
assigned to it immediately after spawn; termination calls
`TerminateJobObject`, and the kernel also enforces cleanup if this
process itself exits uncleanly before ever running teardown code.
- Both the stdio and TCP spawn paths go through the same
`Client::build_command` / attach sequence, so they get identical
containment.
- Attaching containment failing (Windows Job Object setup) degrades to
the previous root-only teardown rather than turning it into a hard
start failure; attempting a real OS termination and it failing is
still surfaced (`StopErrors` for `stop`, a logged error for
`force_stop`/`Drop`).
- The root child is still reaped exclusively through Tokio's
`Child::{wait,try_wait,kill,start_kill}` — nothing here calls a raw
`waitpid` on the tracked process, avoiding the double-reap hazard that
comes from bypassing Tokio's sole ownership of that pid.
Adds `rust/src/process_tree.rs` with focused unit tests, plus two
`Client`-level tests, using a purpose-built descendant that holds an
OS-enforced exclusive file lock (kernel-released on any process death,
including a forced kill, without any cooperating cleanup code). The
tests record the root and descendant pids, assert both alive and the
lock held before teardown, then assert both gone and the lock released
after — and they fail if group/job termination is not actually wired in
(verified by temporarily disabling it and observing the same tests fail
for that reason, then restoring).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unix startup failures can still orphan descendants before process-tree ownership reaches ClientInner.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (4)
| Severity | Finding |
|---|---|
rust/src/lib.rs — The process-tree handle is not yet owned by ClientInner, and dropping it does not terminate a… |
|
rust/src/lib.rs — The new whole-tree behavior in Drop has no regression coverage. The existing drop test constructs… |
|
rust/src/errors.rs — This public shutdown contract omits runtime-shutdown errors, although Client::stop collects them… |
|
rust/src/lib.rs — This unconditionally promises whole-tree termination, but startup intentionally permits containment… |
What changed in this PR
Adds Rust SDK process-tree containment so CLI descendants are terminated during shutdown.
Changes:
- Adds Unix process groups and Windows Job Objects.
- Integrates containment into spawn and teardown paths.
- Adds process-level tests and updates shutdown documentation.
| File | Description |
|---|---|
rust/src/process_tree.rs |
Implements cross-platform containment and tests. |
rust/src/lib.rs |
Integrates containment into client lifecycle. |
rust/src/errors.rs |
Updates shutdown error documentation. |
rust/Cargo.toml |
Adds platform-specific dependencies. |
rust/Cargo.lock |
Locks new dependencies. |
Suppressed comments (2)
rust/src/errors.rs:413
- The documented ordering skips runtime shutdown, whose errors are actually inserted after session destroys and before process-tree termination. This makes the positional contract for
errors()inaccurate.
/// Borrow the collected errors as a slice, in the order they occurred
/// (per-session destroys first, then process-tree termination, then
/// the final child reap).
rust/src/lib.rs:2585
- This also overstates
force_stopwhen Job Object attachment failed: the implementation then has no tree handle and kills only the root child. Document that fallback in this public API contract.
/// process is wedged on I/O. Terminates the CLI's whole process tree
/// (the CLI itself and any descendant it spawned) and sends a kill
/// signal to the CLI child, without awaiting reaper completion, and
/// immediately drops all per-session router state so dependent tasks
/// observe a closed channel rather than a hang.
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…esiliency-p07-sdk
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The Windows tests use a nonexistent PROCESS_SYNCHRONIZE constant and will not compile.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
rust/src/process_tree.rs — windows-sys does not define PROCESS_SYNCHRONIZE; the standard access right accepted by… |
Issues resolved since last review (4)
| Severity | Finding |
|---|---|
rust/src/lib.rs — This unconditionally promises whole-tree termination, but startup intentionally permits containment… View resolved comment |
|
rust/src/errors.rs — This public shutdown contract omits runtime-shutdown errors, although Client::stop collects them… View resolved comment |
|
rust/src/lib.rs — The new whole-tree behavior in Drop has no regression coverage. The existing drop test constructs… View resolved comment |
|
rust/src/lib.rs — The process-tree handle is not yet owned by ClientInner, and dropping it does not terminate a… View resolved comment |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks for this, Dan — the process-tree containment mechanism itself ( I rebased this branch onto the latest I tried to validate the fix end-to-end against the real CLI, using the three scenarios named in the PR description:
So right now I can't confirm this fix changes real-world behavior for any of the three cases named in the PR description, even though the underlying mechanism is correct and would matter for some other kind of descendant (e.g., a plain child process that isn't reparented and doesn't rely on stdio EOF for its own lifecycle). Do you have a specific scenario in mind where you're confident this fix makes a real difference? If so, could you add an E2E test for that case (against the real CLI, not the synthetic self-re-exec test binary) so we can verify it and merge on that basis? Alternatively, if it's easier, we could close this for now until there's a more complete plan for which descendants this should actually cover. For tracking, I've moved this to draft — please mark as ready to review when appropriate. |
|
Closing this in favor of #2458, which has already merged and covers the path forward. |


Summary
primitive (
rust/src/process_tree.rs): a dedicated process group onUnix, a Job Object carrying
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSEonWindows.
Client::stop,Client::force_stop, andDrop for ClientInnerthrough it, so all three now terminate the CLI's whole process tree
(the CLI and any descendant it spawned: an MCP server, a shell tool,
a subagent process), not only the CLI itself.
Client::build_commandsequence, so they get identical ownership.teardown rather than turning a containment failure into a hard start
failure (relevant on Windows, where Job Object setup can fail; Unix
attachment is established at fork and cannot practically fail once
spawn()succeeds). An actual OS termination call failing is stillsurfaced: collected into
StopErrorsforstop, logged forforce_stop/Drop.Child::{wait,try_wait,kill,start_kill}; nothing added here calls araw
waitpidon that pid, so there is no risk of the SDK and Tokioracing to reap (and neither ends up with
ECHILD) the same process.Design notes (relative to an earlier attempt at this)
An earlier revision of #2292 explored the same shape and was reset to a
narrower fix before merge, for two kinds of reasons worth answering here
directly:
descendants) was a reference-cycle in the lifecycle dispatcher, already
fixed independently and unrelated to descendant containment. This PR is
about descendant containment specifically, a different, forward-looking
requirement (prove a descendant, and a resource it holds, actually gets
released), not a resubmission of that bug fix.
threads via
CreateToolhelp32Snapshot; it assigns the Job Objectimmediately after a normal spawn. This accepts a small window where a
child that spawns its own descendant extremely fast could escape
containment (documented in the module), in exchange for not adding
CREATE_SUSPENDED/ResumeThread/snapshot-scan complexity or turning atransient Job Object failure into a hard CLI start failure.
waitpid. The earlierversion's
waitpid(-pgid, ...)loop competed with Tokio's ownownership of the root child's pid. Descendant liveness here is checked
with a signal-0 existence probe, and the group/job is only ever
signalled (never waited on) directly.
Mutex<Option<_>>>slots the existing code already used for the root child, taken once;
nothing here re-signals a process group id after the slot has been
emptied, so a since-recycled pgid is never addressed.
ask. It does not attempt cross-SDK parity with
.NET'sKill(entireProcessTree: true)or the direct-child-only Node/Python/Go/Java SDKs. See Unknowns.
Validation
cd rust && cargo test --no-default-features --features test-support --lib: 220 passed, 0 failed (215 pre-existing + 5 new: 3 in
process_tree::tests, 2Client-level inlib.rs's existing testmodule).
cargo +nightly-2026-04-14 fmt --all -- --config-path .rustfmt.nightly.toml --check: clean.
cargo clippy --all-targets --features test-support -- --no-deps -D warnings -D clippy::unwrap_used -D clippy::disallowed_macros -D clippy::await_holding_invalid_type: clean. (Thebundled-in-processfeature variant of this exact CI command could not be exercised locally due
to an unrelated transient network failure downloading that feature's CLI
archive; this change does not touch that feature.)
cargo doc --no-deps --features test-supportwithRUSTDOCFLAGS=-D warnings: clean.
process_tree::attachto always returnNone(simulating the containment feature not existing at all): both new
Client-level tests (force_stop_...,stop_...) failed within~15s with "descendant survived Client::force_stop" /
"...Client::stop" respectively. The process-tree unit test failed
immediately on its own
attach(...).expect(...).ProcessTree::terminatea no-op while leavingattachment itself intact (simulating "group/job termination
disabled" specifically): the process-tree unit test failed at
"root did not exit within the timeout after process-tree
termination" after ~15s.
seconds (no injected delay).
Unknowns
explicit scope.
.NETalready does whole-tree termination; Node,Python, Go, and Java remain direct-child-only after this change.
teardown code to actually run; it provides no protection if this
process itself is killed or crashes before calling
terminate(). Thisis inherent to process groups (there is no Unix equivalent of Windows'
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE) and is documented in the module.own descendant faster than
AssignProcessToJobObjectcan run) isaccepted, not closed.
subagent process it spawns for real); coverage here uses a
purpose-built helper process holding a real OS-enforced lock,
standing in for "a descendant holding a resource."
Generated by Copilot, reviewed and adjusted by the author before commit.