Integrate vendor VFIO vGPUs into the instance lifecycle

[yummybomb]

Summary

Top layer of the vendor VFIO vGPU stack (generalize-vgpu-device #322 ← hypervisor-liveness #363 ← vendor-vfio-backend #364 ← this). The backend itself is in #364; this layer wires it into the instance lifecycle using claim-first allocation: GPUDevicePath in instance metadata is the authoritative VF claim, persisted before any hardware is touched.

• Claim-first allocation — under a single allocation lock, the allocator lists instance metadata, selects an unclaimed VF (least-loaded placement preserved), and persists the claim; only then is the VF configured. Within a GPU, clean VFs are preferred over dirty ones, and a dirty VF that refuses its reset is skipped in favor of another candidate. Metadata presence reserves the VF, so a stale claim can never clear a VF out from under a live owner — the allocator simply never hands out a claimed VF.
• Idempotent configuration with dirty-VF repair — a dirty VF selected by the allocator is reset at claim time, before the claim is persisted and regardless of whether its leftover type matches the requested profile, gated on the open-VFIO-handle in-use check; a refused reset fails the allocation. ConfigureVGPU then no-ops when the VF already has the requested type and keeps a reset-and-recreate backstop for a type mismatch. When the leftover type consumes enough framebuffer that the requested profile disappears from every creatable list, the allocator resets unclaimed dirty VFs and retries selection once. This replaces the periodic vendor VFIO device sweep.
• Reset-before-unclaim everywhere — every release path (create cleanup, start rollback, stop, delete, reconciler) resets the VF while the metadata claim is still on disk and removes the claim only afterward. Failure cleanup confirms the VMM is dead (socket-ownership liveness from Harden hypervisor process liveness checks #363) before touching the device; ambiguous liveness preserves the claim. If a hardware reset fails, the claim is still removed and the next allocation repairs the dirty VF. When create cleanup cannot confirm the VMM is dead, the claim and instance data are retained with an error log; deleting the instance retries the release.
• Simplified reconciler — a periodic pass releases metadata claims whose VMM is confirmed dead, under the instance lock then the allocation lock. The device-level sweep remains for mdevs only. Removed entirely: GPUAssignedAt, GPURetainedForCleanup, delete-only retention records, vgpu_cleanup_pending errors, the assignment grace period, the in-memory owners map, and the live-claim release guard.
• Hypervisor scope — hypervisor selection remains a caller policy, preserving the framework layer's existing mdev behavior (mdevs are created fresh per assignment, so claim-first applies to vendor VFIO only). The production rollout continues to place vGPU instances on QEMU; the Cloud Hypervisor vendor-VFIO limitation (upstream cloud-hypervisor#7572) is documented in lib/devices/GPU.md (in the base PR Add the vendor VFIO vGPU device backend #364).
• API — the instance API exposes gpu.device_path, the assignment identity on vendor VFIO hosts where mdev_uuid is empty.

The key invariant: cleanup never resets a VF after removing its claim, and all recovery routes through the allocator, which rechecks current metadata and VFIO liveness.

Testing

• go build ./..., go vet ./... clean
• go test ./lib/devices/ and the lib/instances vGPU/reconcile suites pass; go test -race passes for the new concurrency and crash-recovery tests (concurrent claims select distinct VFs; a crash between claim and configure is reconciled at startup; dirty-VF repair with and without open VFIO handles, including a dirty VF that suppresses the profile from all creatable lists; release order under failed resets; clean-first VF selection with fallback when a dirty VF refuses reset; GPU ordering with unknown load). Claim-path tests are linux-tagged since they depend on sysfs paths; cleanup/release/selector tests remain cross-platform
• Broader lib/instances / cmd/api/api suites have pre-existing environment failures (image pulls, network privileges) that reproduce identically on the unmodified base

---

Note

High Risk
Changes GPU allocation, release ordering, and hypervisor kill semantics on production hosts; incorrect liveness or claim handling could leak VFs or wedge hardware.

Overview
Vendor VFIO vGPUs are wired into create/start/stop/delete using metadata-first VF claims: the allocator persists GPUDevicePath under a global lock, then configures the VF (mdev creation stays on the existing path). Least-loaded GPU selection, clean-VF preference, dirty-VF repair at claim time, and removal of the in-memory owner map and periodic vendor VFIO device sweep move recovery into allocation and reconcile.

Reconciliation replaces startup-only mdev cleanup with StartVGPUReconciler (initial pass + ~1 minute): stale metadata claims are released when the VMM is confirmed dead; mdev sweeps skip UUIDs owned by running instances. New metrics cover reconcile and stale-release failures.

VF safety: QEMU and instance teardown SIGTERM before SIGKILL when VFIO/vGPU is attached (VFIOTermGrace), with docs on SIGKILL-during-init wedge risk. Instance API responses add gpu.device_path when set; integration tests allow vendor VFIO hosts.

^{Reviewed by Cursor Bugbot for commit fb132cd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yummybomb]

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design:

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit fb132cd. Configure here.}