feat(supervisor)!: apply streamed configuration snapshots

[pimlock]

Summary

Initialize supervisors and apply live configuration updates through ConnectSupervisor. Protocol revision 2 uses streamed snapshots and requires successful bootstrap before the gateway marks a supervisor initialized. Older protocol revisions retain polling compatibility. The supervisor must connect to every remote middleware required by the initial effective policy before starting the workload, even when fail_open is configured. A failed live registry reload retains the last-known-good registry and keeps the workload running.

Stage 2 architecture walkthrough

Related Issue

Part of #1731.

Stacked on #3244.

Changes

• Make the revision 2 supervisor stream authoritative for bootstrap and live configuration in combined and sidecar deployments.
• Add bounded, coalesced delivery with per-session sequencing, acknowledgements, and periodic repair.
• Record sanitized application outcomes, preserve policy fallback behavior, and update architecture and operational documentation.

Testing

• mise run pre-commit
• mise run ci
• mise run e2e:docker

Latest retention integration validation at 13f7f06b3: full mise run ci passed, including 6,186 Rust tests. mise run e2e with OPENSHELL_E2E_DOCKER_TEST=live_policy_update passed four live-policy Rust tests, 87 Python tests with 81 skips, and the MCP baseline checks, using rebuilt gateway and supervisor binaries.

Known reliability issue observed under load

A focused reproduction with 32 running sandboxes and 32 concurrent exec clients produced 6 false exit-code-1 results across 37,376 exec calls, including commands that explicitly ran exit 0. Diagnostic logs confirmed that the orphan reaper collected a child's exit status 0 before SSH's waiter received ECHILD and substituted exit code 1. No policy changes or gateway restart were needed.

The implicated spawn/register/reap code predates this PR. Existing fix commit c5cf4ec2b is included in #3142.

Retention follow-ups

Continuously dirty admitted keys can delay overflow recipients until a slot is released. Earlier stress probes also observed SQLite lock failures and growing gateway RSS; their attribution remains unresolved. These follow-ups are not fixed by this integration, and the earlier latency measurements have not been rerun against the combined head.

Checklist

• Conventional commits with DCO sign-off
• Architecture and user-facing documentation updated
• Related public skill reviewed and updated
• No generated TypeScript sources committed

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3265.docs.buildwithfern.com/openshell

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[pimlock]

Added the startup policy preparation exchange in fa58bddb8.

Before this change, the supervisor discovered or enriched the startup policy, called the unary UpdateConfig RPC, then reconnected so it could receive a fresh authoritative bootstrap. That extra mutation and reconnect made the startup ownership model harder to follow.

The initial ConnectSupervisor stream now carries the complete exchange:

1. SupervisorHello may offer the policy found in the sandbox image.
2. The gateway selects its existing policy when present. Otherwise it selects the image policy.
3. The gateway sends the selected full policy as StartupConfigCandidate.
4. The supervisor performs image-specific filesystem enrichment and replies with StartupConfigPrepared, using unchanged, a complete prepared policy, or a bounded failure.
5. The gateway treats a prepared policy as an untrusted proposal. It runs it through the normal sandbox policy update path, including validation, safety checks, persistence, and composition.
6. The gateway rebuilds the bootstrap and sends SessionAccepted. The supervisor initializes only from that bootstrap.

There is no reconnect after preparation. Later reconnects omit the image policy, skip startup preparation, and receive the current gateway bootstrap directly.

This keeps the gateway authoritative without requiring it to know what paths exist in the sandbox image. It also removes the old ambiguity around mutation responses: neither the candidate nor the preparation response becomes runtime state. SessionAccepted remains the single startup source of truth.

Failure behavior is fail-closed. Candidate mismatches, preparation failures, and invalid prepared policies produce SessionRejected before the supervisor can observe SessionAccepted. Session traffic is buffered until the remaining fallible acceptance work completes, so a config update cannot overtake acceptance.

The change includes protocol bindings, architecture documentation with a sequence diagram, positive tests for policy precedence and persistence, a negative invalid-policy test, and the focused Docker live-policy E2E coverage.

[pimlock]

gator-agent

PR Review Status

Thanks @pimlock. I reviewed the current stacked-base merge in critical-only mode against the durable feedback ledger. The merge resolution introduces no new Critical defect, and GATOR-c10f3724-01 through GATOR-c10f3724-04 remain resolved.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None; all four prior Gator findings remain resolved.

Gator metadata

• Validation: Project-valid stage 2 of accepted issue Push gateway-owned desired state to supervisors #1731, stacked on active stage-1 PR feat(supervisor): stage gateway configuration snapshot delivery #3244.
• Docs: The reviewed merge resolution does not introduce a new direct UX change; the PR's existing operator documentation remains present.
• Checks: Current-head Branch Checks and E2E are queued or running; completed required gates are green.
• E2E: test:e2e and test:e2e-kubernetes remain applied, and current-head workflows were dispatched without a rerun or /ok to test.
• Head SHA: 1dc9c881ea2b5ac75a93e6dca5e6ee65e1d0c165
• Base SHA: 2091c38a87979c98db51a73722e4e2a204906575
• Merge base SHA: 2091c38a87979c98db51a73722e4e2a204906575
• Patch ID: 8246581dd660c77df41e64fc0013fc444c069255
• Gator payload: 9
• Review mode: critical_only
• Previous reviewed SHA: bd0ad5ee590142391b8eda9cdbf9ed59ee772638
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[politerealism]

Hey @sjenning @mrunalp @derekwaynecarr — this one's been open a bit and is part of the #1731 staged rollout (this is stage 2 of 3). Any chance you have bandwidth for a review pass in the near term? Happy to help unblock if there's anything I can clarify in the meantime.