Skip to content

fix(llm): retarget the deadline on the existing client instead of rebuilding it - #520

Open
FacaHD wants to merge 5 commits into
NVIDIA:mainfrom
FacaHD:fix/async-client-cleanup
Open

fix(llm): retarget the deadline on the existing client instead of rebuilding it#520
FacaHD wants to merge 5 commits into
NVIDIA:mainfrom
FacaHD:fix/async-client-cleanup

Conversation

@FacaHD

@FacaHD FacaHD commented Sep 10, 2026

Copy link
Copy Markdown

Fixes #519

Problem

Under a workflow deadline (SKILLSPECTOR_MAX_WORKFLOW_SECONDS), large scans emitted a stream of:

Task exception was never retrieved
RuntimeError: Event loop is closed

Root cause

Every batch has to be sent with whatever time is left, and _model_for_call() expressed that by building a fresh chat model per call. LangChain caches the underlying httpx client in an lru_cache keyed on (base_url, timeout, socket_options), so a deadline shrinking by a few milliseconds per call is a new cache key every call. A long scan opened one connection pool per LLM request and began evicting the oldest once it passed 128.

Eviction is what crashed. Each analyzer node is synchronous and reaches run_async()asyncio.run(), so it owns its event loop. A pool evicted while a later analyzer is running belongs to an earlier, already-closed loop, and the OpenAI SDK's finaliser reacts to garbage collection with asyncio.get_running_loop().create_task(self.aclose()) — closing the earlier analyzer's sockets from the current analyzer's loop. Nothing awaits that task, so it surfaced only as "Task exception was never retrieved", once per eviction.

Fix

Apply the deadline to the client that already exists instead of baking it into a new one.

  • New _retarget_request_timeout() in llm_analyzer_base.py mutates client.timeout on the live client for ChatOpenAI, ChatAnthropic, and AgentCLIChatModel. Both SDKs read client.timeout when building each request — the same in-place mutation _uses_native_connection_retries already relies on for max_retries.
  • _model_for_call() retargets first and only falls back to constructing a replacement model for transports that keep no mutable deadline, so behaviour is unchanged for them.
  • _StructuredAgentCLIModel now reads transport settings through its owner rather than copying them at construction, so a structured wrapper no longer pins the deadline it happened to be created with. with_structured_output wraps that same model instance.

Dynamic timeouts, the explicit retry loop, the global concurrency limiter, and the one-loop-per-analyzer execution model are all untouched.

Result: one pool per analyzer, one cache key, nothing to evict, and no cleanup that can outlive its loop.

Validation

  • A 200-call reproduction against the openai 3.11 / httpx 2 combination that reported the failure goes from 75 unretrieved Event loop is closed tasks and 220 httpx clients → 0 and 20 (one per analyzer).
  • Regression tests added in tests/nodes/test_llm_analyzer_base.py and tests/unit/test_llm_utils.py.
  • make test (unit): 4015 passed, 14 skipped, 4 xfailed.
  • make lint: clean. make format / ruff format --check: no changes (198 files already formatted).
  • A real scan over an internal skills root produced zero Event loop is closed / Task exception was never retrieved messages.

Note: tests/integration/test_graph.py::test_graph_surfaces_degraded_llm_stage fails in my local environment, but it fails identically on unmodified main (verified by reverting only the two source files) — it is pre-existing and unrelated to this change.

@FacaHD
FacaHD force-pushed the fix/async-client-cleanup branch 2 times, most recently from 9a4258a to 56eab3f Compare September 10, 2026 10:44
…uilding it

A workflow deadline makes every batch carry a slightly different timeout, and
`_model_for_call` expressed that by building a fresh chat model per call.
LangChain caches the underlying httpx client in an `lru_cache` keyed on
`(base_url, timeout, socket_options)`, so a continuously shrinking deadline is a
new cache key every call: a long scan opened one connection pool per LLM request
and started evicting the oldest once it passed 128.

Eviction is what crashed. Each analyzer node is synchronous and reaches
`run_async()` -> `asyncio.run()`, so it owns its event loop. A pool evicted while
a later analyzer is running belongs to an earlier, already-closed loop, and the
OpenAI SDK's finaliser reacts to collection with
`asyncio.get_running_loop().create_task(self.aclose())`. Nothing awaits that
task, so it surfaced only as "Task exception was never retrieved:
RuntimeError: Event loop is closed", once per eviction.

Apply the deadline to the client that already exists instead. Both SDKs read
`client.timeout` when building each request, which is the same in-place mutation
`_uses_native_connection_retries` already relies on for `max_retries`, and
`with_structured_output` wraps that same model instance. Transports without a
mutable deadline still fall back to constructing a replacement, so behaviour is
unchanged for them.

Dynamic timeouts, the explicit retry loop, and the global concurrency limiter are
untouched, as is the one-loop-per-analyzer execution model.

A 200-call reproduction against the openai 3.11 / httpx2 combination that
reported the failure goes from 75 unretrieved `Event loop is closed` tasks and
220 httpx clients to 0 and 20 (one per analyzer).

Signed-off-by: FacaHD <49478213+FacaHD@users.noreply.github.com>
@FacaHD
FacaHD force-pushed the fix/async-client-cleanup branch from 56eab3f to a352111 Compare September 10, 2026 10:48
@FacaHD
FacaHD marked this pull request as ready for review September 10, 2026 11:04

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Reviewed head a352111b38445618152237c5058587cec23eae81 — APPROVE.

The implementation retargets the request timeout on the existing OpenAI, Anthropic, and CLI transports, preserves the fallback reconstruction path for unknown transports, keeps native retries disabled for dynamic deadlines, and makes structured CLI wrappers read their owner's current timeout. The regression coverage verifies client reuse and structured-wrapper propagation. I found no required changes.

Required checks pass, but GitHub currently reports mergeStateStatus=BEHIND; update against current main and re-run required checks before merging.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed current head 9652aee4672d122bfd6cea87ab4d4e39324222b3 after the later main synchronizations. All four PR-owned blobs remain unchanged. The implementation still retargets the existing OpenAI, Anthropic, and CLI transports, preserves reconstruction for unknown transports, keeps dynamic-deadline retries bounded, and propagates timeout updates through structured CLI wrappers. I found no required change.

Merge gate: test-unit is still running on this head and GitHub reports mergeStateStatus=BLOCKED; wait for all required checks to pass before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dynamic workflow deadline leaks an httpx pool per LLM call and raises "Event loop is closed"

2 participants