fix(vllm): cap gpu_memory_utilization for encoder-only vision model load

[furionw]

TL;DR

Add gpu_memory_utilization=0.2 to the nested vllm.LLM(...) in load_vision_model(). The encode worker was hitting a 0.9-default memory precheck that mm_encoder_only=True does not skip, breaking three multimodal disagg tests on every amd64 nightly run.

Fixes OPS-4445. Unblocks the qwen3-vl-2b skip hunks in #8443.

The bug

components/src/dynamo/vllm/multimodal_utils/model.py:173 constructs a nested vLLM engine just to load the Qwen VL vision tower:

vllm_model = LLM(model=model_id, kv_cache_memory_bytes=64*MiB, max_model_len=1,
                 mm_encoder_only=True, enable_prefix_caching=False, ...)

It never sets gpu_memory_utilization, so vLLM falls back to its default 0.9. On CI's amd64 A10G (22.03 GiB) with the PD worker already holding some memory, the up-front free-memory check at vllm/v1/worker/utils.py:413 fails:

[rank0] File ".../dynamo/vllm/multimodal_utils/model.py", line 173, in load_vision_model
[rank0] File ".../vllm/v1/worker/gpu_worker.py", line 283, in init_device
[rank0] ValueError: Free memory on device cuda:0 (19.5/22.03 GiB) on startup is less
        than desired GPU memory utilization (0.9, 19.83 GiB).

That ValueError is what the test harness surfaces as the generic RuntimeError: Engine core initialization failed. See root cause above. — the "root cause above" only lives in the full per-test bash.log.txt (pulled from the allure-results artifact for job 72349014718), which is why the Linear ticket and #8443 never saw it.

Why only e_pd / epd / epd-video fail: only those topologies run a separate encode worker that calls load_vision_model(). p_d handles images inline via PIL; mm_agg_* is a single combined worker — neither exercises the nested LLM(), so neither hits the precheck.

Why it didn't reproduce on our dev workstation: on a 48 GiB card, 0.9 × 48 ≈ 43 GiB fits comfortably in the ~47 GiB free at startup. The bug is GPU-size-gated around ~24 GiB — exactly the CI class.

Why 0.2 and not something cleaner

mm_encoder_only=True already skips model-weight load (interfaces.py:234) and the encoder profile_run() pass (gpu_model_runner.py:5237). But the up-front request_memory() precheck at gpu_worker.py:283 runs before either of those — and it is architecture-blind. Alternatives considered:

Candidate	Skips request_memory()?	Notes
kv_cache_memory_bytes alone	No	Already set. Only skips later determine_available_memory().
skip_mm_profiling=True (v0.19.1 CLI flag)	No	Only skips encoder contribution to profile_run().
AutoModel.from_pretrained (the non-Qwen branch of load_vision_model)	N/A	Worse. Qwen3VLModel.__init__ bundles self.visual + self.language_model; HF loads both. Wastes ~2 GiB on Qwen3-VL-2B, ~27 GiB on Qwen3-VL-30B.
gpu_memory_utilization=0.2	Yes	0.2 × 24 GiB = 4.8 GiB ceiling — covers ViT weights (~1 GiB for Qwen3-VL-2B) + 64 MiB KV stub + activation peak on every GPU class we run on.

Upstream gap worth tracking

vllm/v1/worker/gpu_worker.py:request_memory() has no awareness of mm_encoder_only. When mm_encoder_only=True AND kv_cache_memory_bytes are both set, the precheck could be skipped (or effective utilization could be capped at kv_cache_memory_bytes / total_memory). Neither path exists in vLLM 0.19.1 — hence the explicit cap here.

Test plan

• amd64 post-merge vllm / Test cuda12.9 and cuda13.0 jobs pass mm_e_pd_qwen3-vl-2b, mm_epd_qwen3-vl-2b, and mm_epd_qwen3-vl-2b-video
• mm_agg_* and mm_p_d_* topologies (which don't go through load_vision_model) remain unaffected
• On a 48 GiB dev GPU, encode + PD single-GPU startup still succeeds end-to-end — the cap to 0.2 strictly lowers the ask; it never increases it

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Optimized GPU memory allocation settings for vision model initialization to improve resource efficiency and system stability during model loading operations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b63bf499-36a9-4788-8d22-24a8c83cc191

📥 Commits

Reviewing files that changed from the base of the PR and between cc583b2 and b7ea2dc.

📒 Files selected for processing (1)

• components/src/dynamo/vllm/multimodal_utils/model.py

---

Walkthrough

The vLLM LLM initialization for Qwen VL vision-model-only loading in load_vision_model now explicitly sets gpu_memory_utilization=0.2, adjusting the memory allocation behavior for this specific model configuration.

Changes

Cohort / File(s)	Summary
Memory Configuration components/src/dynamo/vllm/multimodal_utils/model.py	Added gpu_memory_utilization=0.2 parameter to vLLM LLM initialization for Qwen VL vision-model-only loading in load_vision_model.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: adding a gpu_memory_utilization cap for encoder-only vision model loading in vLLM.
Description check	✅ Passed	The description fully covers all required template sections: Overview (TL;DR), Details (comprehensive explanation of the bug and fix), Where to review (specific file and line), and Related Issues (OPS-4445 and PR #8443).
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[GuanLuo]

Is this regression introduced in newer vLLM version? Previously kv_cache_memory_bytes should affects the required memory determined by vLLM. I remember I was toggling good gpu_memory_utilization for encoder but then realized setting kv_cache_memory_bytes is sufficient