[Bugfix][multi_modal] Fix pos_ids being unitialized for minicpmv2.6 in hf runner

[music-dino]

Purpose

#48413 introduced two separate failures in the Multi-Modal Models (Extended Generation 3) TG. - CI
The two types of failures are:

• NameError: name 'List' is not defined
• AttributeError: 'MiniCPMV' object has no attribute 'all_tied_weights_keys'

The second one has two parts; this PR addresses the second part of the second failure:

• The first issue, presented by the error message is circumvented by [Bugfix][CI/Build] Fix InternViT load crash on transformers v5 (missing all_tied_weights_keys) #49679, I didn't want to duplicate it in this PR.
  Either [Bugfix][CI/Build] Fix InternViT load crash on transformers v5 (missing all_tied_weights_keys) #49679 should be merged before this, or this PR can absorb its change (or something similar).
• After the AttributeError is fixed, a second issue presents itself: the tests are flaky. The flakiness is addressed by this PR.

The flakiness in the 6 minicpmv_26 presents itself in two different ways:

• (More common) - the hf reference produces all NaN outputs
• hf and vllm outputs do not agree

Temporary instrumentation was added to HfRunner that scanned every tensor in the loaded model for non-finite values and registered per-module forward hooks to locate where NaN first appeared. The post-load scan flagged resampler.pos_embed as non-finite before any forward pass had run, which ruled out the forward pass and pointed directly at model loading.

That was confirmed with a standalone script that loads the same checkpoint under both runners and dumps the buffer. Under vLLM it holds the expected sin/cos table; under HuggingFace it is all zeros; and re-running the model's own _set_2d_pos_cache on the HuggingFace model reproduces the vLLM values exactly.

pos_embed_check.py

import torch
from transformers import AutoModelForCausalLM

from tests.conftest import HfRunner, VllmRunner

MODEL = "openbmb/MiniCPM-V-2_6"


def show(label: str, buf: torch.Tensor, dtype=None, device=None) -> None:
    b = buf.float().cpu()
    print(f"\n=== {label} ===")
    print(f"shape={tuple(b.shape)} dtype={dtype or buf.dtype} device={device or buf.device}")
    print(
        f"min={b.min():.6f} max={b.max():.6f} mean={b.mean():.6f} "
        f"std={b.std():.6f} all_zero={bool((b == 0).all())}"
    )
    for h, w in ((0, 0), (5, 7), (30, 41)):
        print(f"  [{h},{w}, 0:6]    = {[round(v, 5) for v in b[h, w, 0:6].tolist()]}")
        print(f"  [{h},{w}, 896:902] = {[round(v, 5) for v in b[h, w, 896:902].tolist()]}")


with VllmRunner(
    MODEL,
    max_model_len=4096,
    max_num_seqs=2,
    dtype="auto",
    enforce_eager=True,
    limit_mm_per_prompt={"image": 1, "video": 0, "audio": 0},
    mm_processor_cache_gb=0,
) as vllm_model:
    (vllm_res,) = vllm_model.apply_model(
        lambda model: {
            "buf": model.resampler.pos_embed.float().cpu(),
            "dtype": model.resampler.pos_embed.dtype,
            "device": str(model.resampler.pos_embed.device),
        }
    )

hf_model = HfRunner(MODEL, dtype="auto", auto_cls=AutoModelForCausalLM)
resampler = hf_model.model.resampler
hf_buf = resampler.pos_embed.clone()

resampler._set_2d_pos_cache(resampler.max_size, resampler.pos_embed.device)
hf_fixed = resampler.pos_embed

show("vLLM", vllm_res["buf"], vllm_res["dtype"], vllm_res["device"])
show("HF from_pretrained (as-is)", hf_buf)
show("HF after recomputing the pos cache", hf_fixed)

The script was run with VLLM_ALLOW_INSECURE_SERIALIZATION=1 python pos_embed_check.py, the output being:

=== vLLM ===
shape=(96, 70, 3584) dtype=torch.float32 device=cuda:0
min=-1.000000 max=1.000000 mean=0.368915 std=0.603243 all_zero=False
  [0,0, 0:6]    = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  [0,0, 896:902] = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
  [5,7, 0:6]    = [0.65699, 0.60138, 0.54331, 0.48314, 0.42125, 0.358]
  [5,7, 896:902] = [0.7539, 0.79896, 0.83953, 0.87554, 0.90694, 0.93372]
  [30,41, 0:6]    = [-0.15862, 0.25708, 0.62491, 0.88467, 0.99692, 0.94799]
  [30,41, 896:902] = [-0.98734, -0.96639, -0.7807, -0.46623, -0.07838, 0.31831]

=== HF from_pretrained (as-is) ===
shape=(70, 70, 3584) dtype=torch.float32 device=cuda:0
min=0.000000 max=0.000000 mean=0.000000 std=0.000000 all_zero=True
  [0,0, 0:6]    = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  [0,0, 896:902] = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  [5,7, 0:6]    = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  [5,7, 896:902] = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  [30,41, 0:6]    = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  [30,41, 896:902] = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

=== HF after recomputing the pos cache ===
shape=(70, 70, 3584) dtype=torch.float32 device=cuda:0
min=-1.000000 max=1.000000 mean=0.377531 std=0.597889 all_zero=False
  [0,0, 0:6]    = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  [0,0, 896:902] = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
  [5,7, 0:6]    = [0.65699, 0.60138, 0.54331, 0.48314, 0.42125, 0.358]
  [5,7, 896:902] = [0.7539, 0.79896, 0.83953, 0.87554, 0.90694, 0.93372]
  [30,41, 0:6]    = [-0.15862, 0.25708, 0.62491, 0.88467, 0.99692, 0.94799]
  [30,41, 896:902] = [-0.98734, -0.96639, -0.7807, -0.46623, -0.07838, 0.31831]

Both are caused by the same hf defect - Resampler.pos_embed in MiniCPM-V's remote code.
MiniCPM-V's Resampler ends init by calling _set_2d_pos_cache, which computes a 2D sin/cos position table and registers it with persistent=False.
Transformers destroys them during load finalization. _move_missing_keys_from_meta_to_device ends with an unconditional
loop over named_non_persistent_buffers that replaces every entry with torch.empty_like. The _initialize_missing_keys
pass that follows does not restore it either, since _init_weights only rebuilds rotary embedding buffers.

Fix

minicpmv_26_patch_hf_runner now calls a small helper that walks the loaded model, finds modules exposing _set_2d_pos_cache, and re-runs it on the buffer's current device — restoring exactly what init computed. This is a test-only change

Test Plan

pytest -s -v   "models/multimodal/generation/test_common.py::test_multi_image_models[minicpmv_26-test_case77]" \ 
"models/multimodal/generation/test_common.py::test_multi_image_models[minicpmv_26-test_case78]" \
"models/multimodal/generation/test_common.py::test_multi_image_models[minicpmv_26-test_case79]" \
"models/multimodal/generation/test_common.py::test_single_image_models[minicpmv_26-test_case103]"  \
"models/multimodal/generation/test_common.py::test_single_image_models[minicpmv_26-test_case104]"  \
"models/multimodal/generation/test_common.py::test_single_image_models[minicpmv_26-test_case105]" 

Test Result

Previously failures like:

E                   AssertionError: Test0:
E                   Matched tokens:	[]
E                   hf:	'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'	{0: nan, 1: nan, 2: nan, 3: nan, 4: nan}
E                   vllm:	'Image-1 depicts a street scene in what appears to be a Chinatown area, characterized by the presence of a traditional Chinese archway with red and gold colors, which is a common architectural feature in such neighborhoods. The archway has Chinese characters on it, indicating its cultural significance. In the foreground, there is a stop sign, and a black SUV is driving on the street. The background shows various storefronts with signs, including one that reads "OPTUS," suggesting a commercial area. The overall atmosphere is that of a busy urban environment with cultural elements.\n\nImage-2 shows a close-up view of cherry blossoms in full bloom'	{1906: Logprob(logprob=-0.42546728253364563, rank=1, decoded_token='Image'), 785: Logprob(logprob=-1.9254672527313232, rank=2, decoded_token='The'), 27: Logprob(logprob=-2.1754672527313232, rank=3, decoded_token='<'), 5009: Logprob(logprob=-4.425467491149902, rank=4, decoded_token='Description'), 20629: Logprob(logprob=-4.425467491149902, rank=5, decoded_token='Both')}

Now all tests pass. The 6 tests were run in a loop for 20 iterations (120 test executions in total) to verify that the flakiness no longer occurs.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[AndreasKaratzas]

/ci run

[github-actions]

✅ Triggered Buildkite CI #82912 for commit a2fa2865f041.

[AndreasKaratzas]

LGTM

[DarkLight1337]

cc @tc-mb

[DarkLight1337]

Let's fix CI first and revert once the issue has been fixed upstream