[ywang96]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Similar to #22224, this PR enables an implicit language-model-only model for certain models that are released as a hybrid-only model when limit-mm-per-prompt is set 0 for all modalities. This PR also skips loading certain modality modules when they're set to 0 individually (See Qwen2.5-Omni as an example)

Some followup work:

• We should still add an explicit flag for models that support multimodal embeddings as input (e.g, Qwen2.5VL)
• In this case, the model is still treated as a multimodal model. However, memory profiling will be done by simply saturating the embedding cache with dummy multimodal embeddings (i.e, without activating multimodal encoder and projector since they're not loaded).

Co-authored by @sfeng33

Part of #21943

Test Plan

Running Qwen2.5VL-7B on L40S

Command: vllm serve Qwen/Qwen2.5-VL-7B-Instruct

(EngineCore_0 pid=3267325) INFO 08-06 21:05:33 [cuda.py:327] Using Flash Attention backend on V1 engine.
(EngineCore_0 pid=3267325) INFO 08-06 21:05:33 [weight_utils.py:296] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:00<00:00,  9.78it/s]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:00<00:00,  3.46it/s]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:01<00:00,  2.56it/s]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:01<00:00,  2.32it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00,  2.21it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00,  2.48it/s]
(EngineCore_0 pid=3267325) 
(EngineCore_0 pid=3267325) INFO 08-06 21:05:35 [default_loader.py:262] Loading weights took 2.12 seconds
(EngineCore_0 pid=3267325) INFO 08-06 21:05:35 [gpu_model_runner.py:1959] Model loading took 15.6264 GiB and 2.351742 seconds
(EngineCore_0 pid=3267325) INFO 08-06 21:05:36 [gpu_model_runner.py:2463] Encoder cache will be initialized with a budget of 114688 tokens, and profiled with 1 video items of the maximum feature size.
(EngineCore_0 pid=3267325) INFO 08-06 21:05:42 [backends.py:530] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/3ab54bc400/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_0 pid=3267325) INFO 08-06 21:05:42 [backends.py:541] Dynamo bytecode transform time: 3.57 s
(EngineCore_0 pid=3267325) INFO 08-06 21:05:45 [backends.py:161] Directly load the compiled graph(s) for dynamic shape from the cache, took 2.531 s
(EngineCore_0 pid=3267325) INFO 08-06 21:05:45 [monitor.py:34] torch.compile takes 3.57 s in total
(EngineCore_0 pid=3267325) INFO 08-06 21:05:46 [gpu_worker.py:275] Available KV cache memory: 21.89 GiB
(EngineCore_0 pid=3267325) INFO 08-06 21:05:47 [kv_cache_utils.py:829] GPU KV cache size: 409,920 tokens

Command: vllm serve Qwen/Qwen2.5-VL-7B-Instruct --limit-mm-per-prompt '{"image": 0, "video": 0}'

(EngineCore_0 pid=3265559) INFO 08-06 21:03:32 [parallel_state.py:1124] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_0 pid=3265559) INFO 08-06 21:03:33 [registry.py:129] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(EngineCore_0 pid=3265559) WARNING 08-06 21:03:33 [topk_topp_sampler.py:60] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(EngineCore_0 pid=3265559) INFO 08-06 21:03:33 [gpu_model_runner.py:1910] Starting to load model Qwen/Qwen2.5-VL-7B-Instruct...
(EngineCore_0 pid=3265559) INFO 08-06 21:03:33 [gpu_model_runner.py:1942] Loading model from scratch...
(EngineCore_0 pid=3265559) INFO 08-06 21:03:33 [cuda.py:327] Using Flash Attention backend on V1 engine.
(EngineCore_0 pid=3265559) INFO 08-06 21:03:33 [weight_utils.py:296] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:00<00:00,  9.61it/s]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:00<00:00,  3.34it/s]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:00<00:00,  3.07it/s]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:01<00:00,  2.63it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:01<00:00,  2.37it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:01<00:00,  2.69it/s]
(EngineCore_0 pid=3265559) 
(EngineCore_0 pid=3265559) INFO 08-06 21:03:35 [default_loader.py:262] Loading weights took 1.95 seconds
(EngineCore_0 pid=3265559) INFO 08-06 21:03:36 [gpu_model_runner.py:1959] Model loading took 14.3606 GiB and 2.169745 seconds
(EngineCore_0 pid=3265559) INFO 08-06 21:03:39 [backends.py:530] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/b75976a9c6/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_0 pid=3265559) INFO 08-06 21:03:39 [backends.py:541] Dynamo bytecode transform time: 3.64 s
(EngineCore_0 pid=3265559) INFO 08-06 21:03:42 [backends.py:161] Directly load the compiled graph(s) for dynamic shape from the cache, took 2.518 s
(EngineCore_0 pid=3265559) INFO 08-06 21:03:43 [monitor.py:34] torch.compile takes 3.64 s in total
(EngineCore_0 pid=3265559) INFO 08-06 21:03:43 [gpu_worker.py:275] Available KV cache memory: 24.20 GiB
(EngineCore_0 pid=3265559) INFO 08-06 21:03:44 [kv_cache_utils.py:829] GPU KV cache size: 453,200 tokens

Test Result

(Optional) Documentation Update

[gemini-code-assist]

Code Review

This pull request introduces a feature to disable multimodal capabilities by setting limit_mm_per_prompt to 0 for all modalities. The implementation correctly modifies the main configuration logic. However, this change introduces a critical bug in all modified model-specific files, where they might try to access methods on a None object, leading to an AttributeError. I've provided comments with suggestions to fix these issues by adding None checks before accessing the multimodal configuration.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ywang96]

@DarkLight1337 @Isotr0py If you think this is a good approach I will go ahead and add this functionality to all multimodal models.

[NickLucche]

Not a huge fan of wrapping all backbones initialization with

if multimodal_config.get_limit_per_prompt("image"):

I feel like it's incrementally making it harder to add new MM models from the ground up.

Can we perhaps take this chance to abstract that patter into a maybe_initialize_tower method or similar..?

[DarkLight1337]

Not a huge fan of wrapping all backbones initialization with

if multimodal_config.get_limit_per_prompt("image"):

Maybe we can have something like PPMissingLayer? But it instead wraps a module class

[NickLucche]

That would be great imo

[ywang96]

Not a huge fan of wrapping all backbones initialization with

if multimodal_config.get_limit_per_prompt("image"):

I feel like it's incrementally making it harder to add new MM models from the ground up.

Can we perhaps take this chance to abstract that patter into a maybe_initialize_tower method or similar..?

Yea I'm fine with doing that

[ywang96]

Not a huge fan of wrapping all backbones initialization with

if multimodal_config.get_limit_per_prompt("image"):

I feel like it's incrementally making it harder to add new MM models from the ground up.

Can we perhaps take this chance to abstract that patter into a maybe_initialize_tower method or similar..?

I'm going to merge this PR first since this change is only applied to a few models and not exposed to the users. Will follow up in the next PR to abstract this and apply to all LMMs.