IP-Adapter broken since 7f986a9: "vision_model." in unused_tensors[] drops every tensor of the CLIP-Vision encoder

[mikolajpod]

Summary

Since commit 7f986a9 ("feat: add SenseNova U1.5 support", #1935), loading a
standalone CLIP-Vision encoder with --clip_vision fails: every one of its 519
tensors is reported as not in model metadata and new_sd_ctx returns null.
This makes IP-Adapter unusable for SD 1.5 and SDXL.

The commit added "vision_model." to the unused_tensors[] array in
src/model_loader.cpp. That array is applied to every file the loader opens,
and the tensors of h94/IP-Adapter's models/image_encoder/model.safetensors
are all named vision_model.*, so they are discarded before anything else
happens.

Still present on current master (44dd137).

Reproduction

Following docs/ip_adapter.md exactly, with the weights it recommends:

sd-cli -M img_gen \
  -m v1-5-pruned-emaonly.safetensors \
  --clip_vision image_encoder.safetensors \
  --ip-adapter ip-adapter_sd15.safetensors \
  --ip-adapter-image style.png \
  -p "a photo" --steps 4 -W 512 -H 512

Weights:

• runwayml/stable-diffusion-v1-5 -> v1-5-pruned-emaonly.safetensors
• h94/IP-Adapter -> models/ip-adapter_sd15.safetensors
• h94/IP-Adapter -> models/image_encoder/model.safetensors

Expected: an image.

Actual:

[ERROR] model_manager.cpp:759 - CLIP vision tensor
        'cond_stage_model.transformer.vision_model.embeddings.class_embedding'
        not in model metadata
... 519 lines ...
[ERROR] diffusion_engine.cpp:1159 - model metadata validation failed

Root cause

src/model_loader.cpp:

const char* unused_tensors[] = {
    ...
    "text_encoders.llm.lm_head.",
    "language_model.lm_head.",
    "vision_model.",            // <- added by 7f986a9
};

is_unused_tensor() matches with starts_with, and parse_file() applies it
before the loader prepends the per-file prefix:

for (auto& tensor_storage : tensor_storages) {
    if (is_unused_tensor(tensor_storage.name)) {
        continue;                                   // dropped here
    }
    if (!starts_with(tensor_storage.name, prefix)) {
        tensor_storage.name = prefix + tensor_storage.name;
    }
    ...
}

So for the CLIP-Vision file the names are still the raw vision_model.* when
the filter runs, and all of them match. The clip_vision. ->
cond_stage_model.transformer. mapping in name_conversion.cpp never gets a
chance to run, because there is nothing left to map.

Bisect

• last good: 50062a4 (2026-08-02) - loads the same file with 0 missing
  tensors and generates an image
• first bad: 7f986a9 (2026-09-01)

git bisect over the 51 commits in between, testing whether the file loads.

Suggested fix

The intent of the entry is clearly to skip the vision tower inside SenseNova
U1.5's language model, and the sibling entry "language_model.lm_head."
suggests those tensors live under language_model..

Two options, in the order I would consider them:

1. Apply is_unused_tensor() after the prefix has been prepended. The main
   model is the only file loaded with an empty prefix
   (init_from_file(sd_ctx_params->model_path)); every component file gets an
   explicit one. SenseNova's vision_model.* would therefore still be filtered,
   while clip_vision.vision_model.* would survive. This keeps the original
   intent and needs no change to the list.

2. Narrow the entry to whatever prefix SenseNova's vision tensors actually
   carry (language_model.vision_model. if they follow the sibling entry).

I have not sent a PR because I do not have SenseNova U1.5 weights to verify
that either option still skips what it was meant to skip.

Workaround

For anyone hitting this before a fix lands: renaming the tensors in the
CLIP-Vision file so they start with clip_vision. dodges the filter and lands
them where the loader expects, because names that already begin with the
file's prefix are not prefixed again. Only the JSON header has to be rewritten;
safetensors offsets are relative to the data buffer, so the payload is copied
through unchanged.

Environment

• stable-diffusion.cpp 7f410a3 (master-859) and current master 44dd137
• Vulkan backend, GCC 15.2 (MSYS2 MinGW-w64), Windows 10
• NVIDIA Quadro T2000, driver 580.92