fix(#4097): propagate sustained RAG 5xx/408 indexing failures to backoff gate

[aheritier]

🤖 Automated implementer agent — this PR was opened by the implementer bot from Docker Agentic Platform, not by a human developer

Fixes #4097 (follow-up to #4060/#4062): sustained RAG 5xx/408 indexing failures were never reaching the StartableToolSet backoff gate — only 429 did, because classifyModelCallError swallowed 5xx/408 per-file inside VectorStore.Initialize, so a backend sustaining 503s/408s re-triggered a full concurrent re-index on every agent turn.

Design (option (a) from the issue)

Initialize now tracks the first gate-arming transient error (429/408/retryable-5xx per startBackoffRetryable's own predicate, via new isGateArmingTransientError) seen across the run's goroutines, using atomic.Pointer[error].CompareAndSwap for race-free first-writer-wins. An isolated per-file failure with any successful indexing still returns nil — unchanged per-file-skip semantics, since indexed files persist and only the failures get retried next run. But if a run indexes zero files and at least one attempted file hit a gate-arming status, that error is now propagated (wrapped with %w throughout the chain) so it survives errors.As all the way to startBackoffRetryable.

429 is untouched — it already aborts the whole run on the first failure via the existing errIndexingAborted path.

Tests

• pkg/rag/strategy/vector_store_test.go: isolated-vs-sustained-failure distinction at the VectorStore.Initialize unit level (408/500/502/503/504, plus a FileIndexConcurrency>1 variant proving the atomic write is race-free under -race).
• pkg/tools/builtin/rag/rag_backoff_test.go: generalized the existing 429 gate-arming test to also cover 408/503, proving the full Manager→ToolSet.Start→StartableToolSet plumbing arms the gate for the wider status family.

Docs

Updated docs/tools/rag/index.md's retry-policy table/prose and the cross-reference in docs/tools/mcp/index.md to describe the new behaviour instead of the "429-only" gap.

Sequencing note

Issue #4073 (touches this same pkg/rag/strategy/vector_store.go seam) is intentionally held back until this merges.

Review

Reviewed by an independent reviewer sub-agent through two passes (request_changes → fixes → approve). No findings were rejected; all were addressed.

[aheritier]

🤖 Automated implementer agent — this comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Review-of-record (self-review is blocked for the PR author on this instance, so recording findings as a COMMENTED review rather than an approval).

Fix correctness

• pkg/rag/strategy/indexing_errors.go — new isGateArmingTransientError (L47-56) mirrors startBackoffRetryable's own predicate (pkg/tools/startable_backoff.go:51-59): pre-filters to *modelerrors.StatusError via errors.As, then modelerrors.RetryableHTTPStatus. Plain-text errors (timeouts, chunk counters) can't arm the gate here either, matching the existing 429-path guarantee.
• pkg/rag/strategy/vector_store.go Initialize (~L241-246, L333-341, L373-382): each errgroup goroutine records the first gate-arming transient error via atomic.Pointer[error].CompareAndSwap(nil, &err); after g.Wait(), if indexed == 0 and one was recorded, the error is propagated wrapped with %w so *modelerrors.StatusError survives the chain up through Manager.Initialize → rag.ToolSet.Start → StartableToolSet.startLocked/tryStartLocked. An isolated per-file failure with indexed > 0 still returns nil — unchanged.
• 429 abort-on-first-failure (errIndexingAborted) is untouched; filesToIndex == 0 and cancellation short-circuit before the new block, so neither interferes.
• CheckAndReindexChangedFiles/file-watcher path deliberately left untouched (out of scope; #4073 lands on this seam next).

Tests

• pkg/rag/strategy/vector_store_test.go: TestInitializeContinuesOnTransientModelError (isolated failure, 1-of-5 files) vs TestInitializeSurfacesSustainedTransientModelError (408/500/502/503/504, all-fail) pin the distinction; ..._ConcurrentFailures variant with FileIndexConcurrency: 5 exercises the atomic write under real concurrency (go test -race clean).
• pkg/tools/builtin/rag/rag_backoff_test.go: TestRAGStartableBackoff_StatusErrorEngagesGate generalized to 429/408/503 subtests, proving the full Manager→ToolSet.Start→StartableToolSet plumbing arms the gate for the wider status family, not just 429.

Validation (this sandbox needed sudo apt-get install -y gcc libc6-dev for CGO/treesitter): CGO_ENABLED=1 task build, task test (all packages green, including -race on the touched packages), task lint (0 golangci-lint issues, 0 custom-cop offenses, go mod tidy clean) all pass.

Process note: this PR went through two independent reviewer sub-agent passes (request_changes → doc-precision/comment/race-test fixes → approve). No findings were rejected as false positives; all were addressed.

[aheritier]

👋 This PR has merge conflicts with the base branch. Please rebase or merge the latest base branch and resolve them. I've moved it to draft and added status/needs-rebase; it'll be picked back up automatically once the conflicts are cleared.