[an1310]

What does the PR do?

When the stub misses a health update during an execution, ProcessRequests() returns "Stub process '...' is not healthy." It still sends the stub the message that ends the execution. The stub then frees its response as soon as it sends it, and the next execution takes that freed response as its own. The result is wrong responses, a corrupted shared memory allocator, and server crashes (boost::interprocess::lock_exception, std::bad_alloc, out-of-bounds and JSON parse errors).

The backend now keeps the shared memory of that execution and sends the error responses. It then waits until the stub has sent the response before it ends the execution and releases the requests. The stub also clears GPU tensors left by a response that was never loaded.

Response factory fixes:

• The backend deletes the response factories of those requests. Before, they were never deleted and kept the model from unloading.
• The cleanup of response factories after an error skips requests that the response sender already closed. Before, a model that sent a final response through the response sender and then raised an error had that factory deleted twice, which crashed the server.
• The stub initializes is_response_factory_deleted in the response batch, which was left uninitialized. Because of it, a model that raised an error in execute() could fail to unload, such as wrong_model in the lifecycle tests.

Checklist

• I have read the Contribution guidelines and signed the Contributor License
  Agreement
• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• I ran pre-commit locally (pre-commit install, pre-commit run --all)
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

• test: Cover a Python backend stub that stops responding during execute server#9063

Where should the reviewer start?

src/python_be.cc: ModelInstanceState::FinishAbandonedExecution(), ModelInstanceState::ReleaseAbandonedResponseFactories() and the error path in TRITONBACKEND_ModelInstanceExecute().

Test plan:

• Added test_execute_unresponsive and test_execute_error_after_final to the lifecycle tests in test: Cover a Python backend stub that stops responding during execute server#9063, and checks that their models unload when the server shuts down.
  • In test_execute_unresponsive, the model stops its stub during execute(), so the server stops waiting for the response.
  • In test_execute_error_after_final, the model sends its final response through the response sender and then raises an error, with or without stopping the stub first.
• Without this change:
  • In test_execute_unresponsive, the next request gets "Failed to retrieve the response error." and the server then aborts.
  • test_execute_error_after_final crashes the server.
  • wrong_model does not unload.
• With it, every request receives its own response, the stub's Python threads keep running, and all lifecycle models unload.
• The lifecycle tests pass in 5 iterations with nvcr.io/nvidia/tritonserver:26.04-py3, both with a CPU-only build on aarch64 and with a GPU build on x86_64 (RTX 5090).
• GPU outputs, checked by hand with a model that returns CuPy tensors on an RTX 5090: without the stub change, the first request after the stall returns a wrong value. With it, every request returns its own value.

Caveats:

• A stub that stays alive but never resumes now blocks its instance. Before, its requests failed.
• The GPU tensor cleanup is not in the QA test, because the lifecycle tests install CPU-only torch.
• If the stub exits before it sends the response, the response factories of those requests are still not deleted.
• If a decoupled model closes one request through the response sender and then raises an error, the factories of the other requests in the batch are still not deleted. The stub reports deleted factories for the whole batch, not per request.

Background

On Triton 26.04 (aarch64), the server crashed repeatedly while the host was paging heavily. A stub core showed a free-list link in the shared memory pool increased by one, at the spot where AllocatedShmOwnership::ref_count_ of a freed response lives. Before 27f04d1, the backend did not end an execution it had stopped waiting for, and it restarted the stub.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• None

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge.

Summary

The PR keeps an unfinished execution’s shared memory alive after a failed health check. It waits for the stub’s response before ending that execution and releasing its requests.

The latest changes:

• The backend waits for an unhealthy stub before reusing its request memory.
• The stub clears leftover GPU tensors and sets response cleanup flags.

No new actionable issues were established. an1310 acknowledges that a live stub that never resumes blocks its instance. an1310 also identifies retained response factories as an existing limitation; cleanup still preserves sender-owned factories.

_{Reviews (2) · Last reviewed commit: "fix: Release the response factories of f..." · Reviewed by Greptile}