[trueoneplusone]

Summary

This PR adds an opt-in cuda_graph_image_encoder=True fast path to build_sam2_video_predictor for fixed-signature CUDA video inference.

It lazily captures forward_image into a native PyTorch CUDAGraph and replays it on subsequent frames to reduce CPU kernel-launch overhead. The existing eager path remains the default.

Refs #768.

Why this scope

The image encoder is a good CUDA Graph boundary because its tensor signature is fixed across video frames and it does not depend on SAM2's temporal memory bookkeeping.

I intentionally do not capture track_step or alter frame_idx/memory selection. Those paths use real temporal distances for memory and object-pointer positional encoding, so forcing a sentinel frame index or static memory structure could silently change tracking semantics.

Correctness details

CUDA Graph outputs live at fixed addresses and are overwritten on the next replay. SAM2 caches image features between interactions/frames, so exposing those buffers directly would create cross-frame aliasing.

This implementation therefore clones the captured output tree before returning it, preserving frame-owned feature tensors.

The graph is automatically re-captured when any relevant input signature component changes:

• shape
• dtype
• device
• CUDA autocast enabled/disabled
• CUDA autocast dtype

Training, gradient-enabled, and non-CUDA execution fall back to eager mode.

cuda_graph_image_encoder=True is currently mutually exclusive with vos_optimized=True / compile_image_encoder=True to avoid stacking two independent graph/compiler fast paths without validation.

Local validation

Environment:

• RTX 5090
• PyTorch 2.11.0 + CUDA 12.8
• SAM2.1 Hiera Small
• BF16 autocast
• official notebooks/videos/bedroom

End-to-end runs through the public builder API:

Input	Frames	Path	ms / frame	FPS	Speedup
512 px	21	eager	113.84	8.78	1.00×
512 px	21	CUDA Graph image encoder	72.85	13.73	1.56×
1024 px (default)	11	eager	116.91	8.55	1.00×
1024 px (default)	11	CUDA Graph image encoder	74.03	13.51	1.58×

For both resolutions, eager vs. CUDA Graph outputs had:

• minimum mask IoU: 1.0
• mean mask IoU: 1.0
• maximum absolute mask-logit difference: 0.0

I also checked multiple different input frames at the encoder boundary and observed exact feature parity.

Tests

Added coverage for:

• CPU eager fallback
• CUDA graph/eager parity
• output ownership across repeated replays
• recapture on input-signature changes
• rejection of simultaneous VOS compile and CUDA Graph fast paths

Local checks:

• pytest -q tests/test_cuda_graph_image_encoder.py → 3 passed
• ufmt check ... → passed
• git diff --check → passed

The new option is documented in the README.