feat: resolve oci:// model references via llmman serve

[ericcurtin]

Summary

Adds an oci:// scheme so a model published as a CNCF ModelPack OCI artifact can be used anywhere a HuggingFace repo id can:

aphrodite run oci://ghcr.io/org/model:tag

Model distribution is increasingly moving to OCI registries -- the same registries, credentials, mirroring and air-gap tooling a deployment already uses for container images.

How it works

ModelConfig.maybe_pull_model_tokenizer_for_runai is already the one rewrite hook that runs before anything else touches model=/tokenizer=, so the oci:// branch goes there. The two schemes are disjoint, so it dispatches and returns early -- the object-storage path is reached only when no oci:// reference is present, and is otherwise bit-for-bit unchanged.

Acquisition is delegated to a running llmman serve rather than hand-rolled: llmman already implements the ModelPack media types, registry auth, resumable blob download and a content-addressed store.

New aphrodite/transformers_utils/llmman.py is the client, stdlib-only (urllib), no new dependency:

• GET /api/version probes reachability and identity; a server answering without a version field is reported as "not an llmman daemon", worth distinguishing from nothing listening.
• POST /api/pull streams NDJSON so a multi-gigabyte fetch is not silent. An error arrives in-band at HTTP 200, and a stream that ends without success is also a failure -- both are errors, not a completed pull.
• llmman resolve --no-pull reports where the bytes landed. The daemon deliberately exposes no local path, so the CLI is the documented interface; --no-pull guarantees it only reports on what /api/pull already fetched, keeping the daemon the only thing that touches the network.
• LLMMAN_HOST is honoured with llmman's own parsing, including rewriting a wildcard bind (0.0.0.0, [::]) to loopback.

aphrodite/transformers_utils/oci_utils.py handles the scheme, deliberately mirroring the shape of the neighbouring runai_utils.py (including the str() cast so pathlib.Path inputs work).

A pull needs both the daemon reachable and the binary on PATH (or APHRODITE_LLMMAN_BIN); each missing piece has its own actionable error. Neither is required unless an oci:// reference is actually used.

Design notes

• Explicit scheme, no sniffing. A bare registry/name:tag is indistinguishable from a HuggingFace repo id (org/model); guessing would silently hijack existing --model org/model deployments.
• model_weights keeps the reference the user typed, matching what the runai path does with its URL.
• Mixing with S3. Model and tokenizer resolve independently, so oci:// and s3:// can be combined.
• One pull for a shared reference. A --tokenizer naming the same image reuses the resolved directory -- a ModelPack image is pulled whole, so the tokenizer is already there.

Limitations

llmman resolve reads the local store, not the daemon's, so the daemon must share the store (same host and user, or shared LLMMAN_MODELS).

Testing

Two new files under tests/transformers_utils/. test_llmman.py (8 cases) runs against a real HTTP server on a loopback port, not mocks, so the NDJSON streaming contract is genuinely exercised: /api/version accepted, a non-llmman server rejected, nothing-listening reported actionably; pull success with forwarded byte progress and the exact request body asserted; in-band error at HTTP 200; a stream ending without success; non-OK status; a non-JSON diagnostic tolerated.

test_oci_utils.py: scheme detection incl. case-insensitivity and pathlib.Path; that a HF repo id, a local path, and s3:// / gs:// / az:// are not claimed (the regression that matters most); strip_oci_scheme round-trips; the resolve contract plus eight malformed-output cases; every LLMMAN_HOST form incl. wildcard-to-loopback; binary default/override and the missing-binary error; empty reference rejected without touching the daemon; and the scheme stripped before hand-off with progress wired.

Verified honestly:

• 23 logic assertions executed standalone against a live loopback daemon (scheme handling incl. every non-claimed shape, delegation with the bare reference, endpoint parsing incl. wildcard rewriting, /api/version, pull success + progress + request body, in-band error, stream without success, resolve contract) -- all pass
• ruff format and ruff check clean on all touched files
• The pytest files were not executed in-tree: importing aphrodite.config.model needs torch and the full stack, unavailable here. Flagging rather than implying coverage I do not have.
• No end-to-end run against a live llmman serve backed by a real registry.

Disclosure: written with AI assistance.

[AlpinDale]

Thanks for the PR! Please resolve the merge conflicts and take a look at the review comments below.

Also as an aside, I took a look through the llmman repo, and it seems to download through the daemon selected by LLMMAN_HOST, then runs a local CLI to find the files, which seems to read its own local store instead of the remote daemon's...? Please let me know if that's intentional because that'd imply remote daemons (even even local ones running under another user) won't work without shared storage.

[AlpinDale]

Maybe a bit of a nit, but this seems to break when used together with S3, for example OCI model + S3 tokenizer, and S3 model + OCI tokenizer both fail.

[ericcurtin]

Thanks, fixed: model and tokenizer are now resolved independently, so oci:// and s3:// can be mixed. Rebased, with a test added in tests/test_config.py.

[ericcurtin]

@AlpinDale Thanks, good catch. Not intentional for remote daemons: llmman resolve is an in-process hook that reads the local store (LLMMAN_MODELS, default ~/.local/share/llmman/store) and never contacts LLMMAN_HOST. So the daemon must share that store with Sonar (same host and user, or LLMMAN_MODELS on shared storage). I documented this in llmman.py and in the resolve error message.

[ericcurtin]

@AlpinDale Rebased and addressed your comments, PTAL when you can. Thank you!