Quarantine unhealthy vGPU VFs via a persisted health store

[yummybomb]

Summary

Bottom half of the wedged-VF work (detection lands separately on top, in #435). This PR adds the quarantine machinery: a persisted VF health store plus the placement, admission, and API changes that act on it. Nothing in this PR reports failures automatically yet — the store is fed by the guest-side detection in #435, and by operators editing the state file per the runbook.

• VF init failures are tallied in <data-dir>/gpu/vf-health.json per instance assignment; a VF is quarantined once failures accumulate from gpu.vf_quarantine_threshold distinct assignments (default 2). An exact-assignment success report clears the match and every older tally. It rescinds the quarantine only when the matched failure is the most recent one recorded, i.e. the report that crossed the threshold.
• Quarantined VFs are excluded from placement and advertised profile availability. A card carrying any quarantined VF sorts behind every clean card regardless of load, so it only receives placements when no clean card can serve the profile. This is deliberate: one wedge is treated as a signal about the card, not just the VF. Selection among equivalent clean free VFs is randomized so one VF cannot capture every placement.
• GET /resources reports allocatable_slots and quarantined_slots; GPU admission gates on the allocatable count. While the health store is unavailable, allocatable_slots is 0 and placement_disabled_reason carries the load or write error, so a broken state file is distinguishable from a full host. The hypeman_resources_gpu_slots gauge gains kind=allocatable and kind=quarantined, and hypeman_resources_gpu_placement_disabled is 1 while the store is unavailable so the condition is alertable from metrics alone.
• Each persisted vendor VFIO claim records GPUClaimedAt. The health store keys reports on (instance, assignment), and this is the assignment identity Detect wedged vGPU VFs from the guest and report to the health store #435 will report.
• devices.InitVFHealth(path, threshold) takes the configured threshold directly, so loaded tallies are always evaluated against it without an ordering dependency on a separate setter.

Rebuilt on the claim-first allocator

This branch was rebuilt from main after #321 merged as the claim-first rewrite. The health store, config, paths, resources, API, and docs commits are the originally reviewed ones, replayed. Placement is the part that changed:

• Exclusion and the randomized tiebreak now live in selectVendorVFIOVF in lib/instances/vgpu.go, which is where placement lives on main. The devices-layer selectLeastLoadedVF and reconcile hunks from the earlier revision are gone with the code they patched.
• Placement reads the quarantine set under the allocation lock, but quarantine mutations happen under the devices lock. configure re-checks quarantine under that lock before writing to the VF, so a VF quarantined between selection and configuration fails the claim instead of being handed to a guest.
• The selector tests moved with the code; configure gained tests for the quarantine and store-unavailable refusals.

Safety and failure handling

• An unreadable or invalid state file fails closed: mutations are refused, placement and advertised availability are disabled, and loads are retried after repair.
• State writes fsync the temp file before the rename and the directory after, so a host crash cannot silently drop a quarantine.
• State write failures before rename roll back in memory. Failures after rename retain the visible state and keep placement closed until a later report durably re-persists it.
• Quarantine only removes capacity — it never touches a running instance.

lib/devices/GPU.md documents the store semantics, draining the parent GPU, the SR-IOV recovery cycle (including quiescing nvidia-persistenced alongside DCGM), and clearing quarantine state.

Testing

Passed locally:

go vet ./lib/... ./cmd/...
go test -race -count=2 ./lib/devices ./lib/paths ./lib/resources ./cmd/api/config
go test -race ./cmd/api/api -run 'Resource|GPU'
go test -race -count=5 ./lib/instances -run 'VGPU|VendorVFIO|Select|Quarantin|StoredVGPU|CleanupResets|CleanupPreserves|CleanupRemoves|ReleaseStored|Snapshot.*GPU|Fork.*GPU'

The full lib/instances and cmd/api/api suites were not run here; tests that boot real VM images or pull images need a host this environment does not provide. lib/oapi/oapi.go was regenerated with make oapi-generate, not merged.

---

Note

Medium Risk
Changes vGPU placement, admission, and fail-closed behavior when VF health state is corrupt or unwritable; incorrect quarantine or store outages can block all new vGPU allocations until recovery.

Overview
Adds persisted VF health (<data-dir>/gpu/vf-health.json) and the plumbing to act on it when guest driver init failures stack up. Config gpu.vf_quarantine_threshold (default 2) controls how many distinct instance assignments must fail before a VF is quarantined; tallies are re-evaluated on load when the threshold changes.

Placement and capacity: Quarantined VFs are dropped from vendor-VFIO profile counts and from selectVendorVFIOVF (GPUs with any quarantined VF sort behind clean cards; ties among equivalent free VFs are randomized). configure re-checks quarantine under the devices lock. Vendor VFIO claims persist GPUClaimedAt as the assignment key for future failure/success reports (fed by a follow-up PR or manual state edits).

API and admission: GET /resources GPU status gains allocatable_slots, quarantined_slots, and optional placement_disabled_reason. GPU admission and pending reservations gate on allocatable slots, not total_slots - used_slots. Invalid or unwritable health state fails closed (no new vGPU placement until load/write recovers).

Ops: Startup calls devices.InitVFHealth; metrics add allocatable/quarantined slot kinds and hypeman_resources_gpu_placement_disabled. Docs and example config describe the store and runbook.

^{Reviewed by Cursor Bugbot for commit 7cc1a45. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at b41a501 (pushed)

go get github.com/kernel/hypeman-go-staging@b41a50163af264946f4e773e9aee2369f7cbd53f

✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

232 files generated at 5359640 (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at 6e45f59 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note

Level	Code	Message	Targets

Build metadata

Build	bd_76GgDDPN-compact-oat
Timestamp	2026-09-02T20:08:04.597Z
stlc	8413509
Spec hash	a72fd0a7917b
Config hash	659c3687c3f0

---

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-09-02 20:08:33 UTC

[yummybomb]

Cleanup pass (deslop / test dedup), no behavior changes:

• b4f924f4 — the identical lock-order comment appeared verbatim in both ReportVFInitFailure and ReportVFInitSuccess; it now lives once on the vendorVFIOMu declaration.
• 94f8496e — collapsed redundant test cases in vf_health_test.go (595 → 559 lines, 22 → 19 tests):
  • TestVGPUAvailabilityExcludesOnlyQuarantinedVFs folded into TestVGPUAvailability (below-threshold tally asserted alongside the quarantined/allocated cases).
  • TestCheckedAddressesFailsClosedOnUnloadedState removed; its unique repaired-state recovery assertion moved into TestVGPUAvailabilityFailsWhenStoreUnavailable via the public API, and the invalid-record table already asserts checkedAddresses errors per case.
  • TestReportVFInitFailureRespectsConfiguredThreshold removed — subsumed by TestReportVFInitFailureQuarantinesAtThreshold (outcomes/counts/threshold field) and the two threshold re-evaluation tests.

The persist-failure/rollback/retry test family was left intact — each pins a distinct branch fixed by 2bea9f08/a69fae1b/ce05d4eb. Exported surface is untouched (ReportVFInit*, VFHealthStoreUnavailable, TotalQuarantinedVFs, result types), but the stacked #435 will need a rebase: its diff context overlaps the removed comments in vf_health.go, and it anchors new tests near TestReportVFInitSuccessWithoutMatchingFailureClearsNothing.

Verified: go vet + go test -race ./lib/devices ./lib/paths ./lib/resources ./cmd/api/config pass, and make oapi-generate reproduces lib/oapi/oapi.go byte-identical.

[yummybomb]

Follow-up pass on the review nits, no behavior change to placement or quarantine semantics:

• c7f6c00 — GetVGPUAvailability returns a VGPUAvailability snapshot that ListGPUProfilesWithVFs now takes, so a /resources read hits the VF health store once instead of twice (and retries a latched persist once, not twice). listProfiles takes the quarantined set instead of reading the store.
• Threshold re-evaluation (SetVFQuarantineThreshold, load) returns the persist error instead of logging it via slog. InitVFHealth now reports a failed load-time re-evaluation write, so the startup log fires for it; main.go treats a threshold error as a config failure. Added a comment on why re-evaluation keeps the in-memory quarantine on a failed write, unlike reports.
• Aligned the rescind wording (VFInitSuccessReport, reportSuccess, GPU.md) with the code: only the most recent recorded failure rescinds.
• Comment on why GPUClaimedAt is set beside setStoredVGPUDevice rather than inside it.
• New test: TestInitVFHealthFailsWhenReevaluatedQuarantineCannotPersist.

Verified: gofmt, go vet (linux; darwin build of devices/instances/resources), go test -race ./lib/devices ./lib/resources ./cmd/api/config, ./cmd/api/api -run 'Resource|GPU', and the lib/instances vGPU subset from the PR body all pass. #435 will need a rebase for the SetVFQuarantineThreshold and ListGPUProfilesWithVFs signature changes.

[yummybomb]

made some changes since pr 321 changed signfiicantly from the last review will request a re review

[sjmiller609]

re-reviewed. adjustments look good

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9aea9c4. Configure here.}