[reaganjlee]

Purpose

status: found bug in implementation

Resolves part of #21943. Previously, a change was made to implicitly make a model language-model only (#22299) when all modalities had limits set to 0. However, some users may want to input embeddings as an input while still saving on memory from the unused encoder. This changes behavior to allow this.

Test Plan

Running Qwen2.5VL-7B on L40S

Command: vllm serve Qwen/Qwen2.5-VL-7B-Instruct

(EngineCore_DP0 pid=4731) INFO 12-29 21:28:42 [default_loader.py:308] Loading weights took 2.66 seconds                                                                                        
(EngineCore_DP0 pid=4731) INFO 12-29 21:28:43 [gpu_model_runner.py:3734] Model loading took 15.6270 GiB memory and 142.865302 seconds                                                          
(EngineCore_DP0 pid=4731) INFO 12-29 21:28:43 [gpu_model_runner.py:4547] Encoder cache will be initialized with a budget of 114688 tokens, and profiled with 1 video items of the maximum featu
re size.                                                                                                                                                                                       
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:02 [backends.py:644] Using cache directory: /root/.cache/vllm/torch_compile_cache/9515dd7e93/rank_0_0/backbone for vLLM's torch.compile             
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:02 [backends.py:704] Dynamo bytecode transform time: 3.55 s                                                                                         
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:10 [backends.py:261] Cache the graph of compile range (1, 2048) for later use                                                                       
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:14 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 7.76 s                                                                     
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:14 [monitor.py:34] torch.compile takes 11.31 s in total                                                                                             
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:15 [gpu_worker.py:363] Available KV cache memory: 9.76 GiB                                                                                          
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:16 [kv_cache_utils.py:1291] GPU KV cache size: 182,720 tokens                                                                                       
(EngineCore_DP0 pid=4731) INFO 12-29 21:29:16 [kv_cache_utils.py:1296] Maximum concurrency for 128,000 tokens per request: 1.43x

Command: vllm serve Qwen/Qwen2.5-VL-7B-Instruct --limit-mm-per-prompt '{"image": 0, "video": 0}'

(EngineCore_DP0 pid=7327) INFO 12-29 21:31:37 [default_loader.py:308] Loading weights took 2.98 seconds                                                                                        
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:38 [gpu_model_runner.py:3734] Model loading took 14.3606 GiB memory and 3.536902 seconds                                                            
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:42 [backends.py:644] Using cache directory: /root/.cache/vllm/torch_compile_cache/06830b85c1/rank_0_0/backbone for vLLM's torch.compile             
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:42 [backends.py:704] Dynamo bytecode transform time: 3.65 s                                                                                         
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:47 [backends.py:261] Cache the graph of compile range (1, 2048) for later use                                                                       
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:49 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 3.66 s                                                                     
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:49 [monitor.py:34] torch.compile takes 7.30 s in total                                                                                              
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:50 [gpu_worker.py:363] Available KV cache memory: 26.85 GiB                                                                                         
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:51 [kv_cache_utils.py:1291] GPU KV cache size: 502,656 tokens                                                                                       
(EngineCore_DP0 pid=7327) INFO 12-29 21:31:51 [kv_cache_utils.py:1296] Maximum concurrency for 128,000 tokens per request: 3.93x

Command: vllm serve Qwen/Qwen2.5-VL-7B-Instruct --limit-mm-per-prompt '{"image": 0, "video": 0}' --enable-mm-embeds

(EngineCore_DP0 pid=9025) INFO 12-29 21:34:30 [default_loader.py:308] Loading weights took 2.38 seconds                                                                                        
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:31 [gpu_model_runner.py:3734] Model loading took 14.3606 GiB memory and 2.870875 seconds                                                            
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:31 [gpu_model_runner.py:4533] All modalities are disabled but enable_mm_embeds=True. Initializing encoder cache with None values for memory optimiza
tion.                                                                                                                                                                                          
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:35 [backends.py:644] Using cache directory: /root/.cache/vllm/torch_compile_cache/8536fc9e9c/rank_0_0/backbone for vLLM's torch.compile             
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:35 [backends.py:704] Dynamo bytecode transform time: 3.66 s                                                                                         
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:39 [backends.py:261] Cache the graph of compile range (1, 2048) for later use                                                                       
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:41 [backends.py:278] Compiling a graph for compile range (1, 2048) takes 2.01 s                                                                     
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:41 [monitor.py:34] torch.compile takes 5.67 s in total                                                                                              
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:42 [gpu_worker.py:363] Available KV cache memory: 26.84 GiB                                                                                         
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:43 [kv_cache_utils.py:1291] GPU KV cache size: 502,624 tokens                                                                                       
(EngineCore_DP0 pid=9025) INFO 12-29 21:34:43 [kv_cache_utils.py:1296] Maximum concurrency for 128,000 tokens per request: 3.93x

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request introduces functionality to allow providing pre-computed embeddings for a multimodal model even when the corresponding modality's encoder is disabled. This is a useful feature for saving memory. The changes span across the multimodal registry, processing logic, cache management, and model runners to correctly handle this new "embeddings-only" mode. The tests have also been updated to cover this new behavior.

My review found one critical issue in a test utility function that could lead to incorrect test behavior. Otherwise, the core logic and implementation appear sound.

[mergify]

Hi @reaganjlee, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/vllm-project/vllm/blob/66acef20fc59a3d50ef7ca811f463b571aff0dee/v1/core/encoder_cache_manager.py#L324-L337
Avoid returning a 1-token budget for embedding-only mode

When all modality limits are 0 but enable_mm_embeds=True, this returns an encoder compute budget/cache size of 1. The scheduler uses encoder_compute_budget to decide whether it can schedule a multimodal item based on the number of placeholder embeds (e.g., image/video embeds are typically far >1). That means any real embedding input will fail can_allocate, so the encoder step is never scheduled and the request can stall before the placeholder or proceed without inserting the provided embeddings. To support embedding-only mode, the budget needs to allow the actual number of embeds per item (or bypass the encoder budget check for embedding inputs) rather than forcing 1.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @reaganjlee.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[reaganjlee]

Migrating to #32493