[Core][MM] Add mechanism to configure multimodal fields which should stay on CPU

[lgeiger]

Purpose

This is a followup on #28141 and adds a multimodal_cpu_fields to the SupportsMultiModal protocol. This allows model implementers to define multimodal fields which shouldn't be transferred to the GPU.

I updated the Qwen VL model definitions to set image_grid_thw and video_grid_thw as CPU-only fields. These fields are transferred to the GPU by group_mm_kwargs_by_modality and copied back to the CPU in the model forward pass via .tolist(). The latter causes a cudaStreamSync which can hurt performance. Keeping them on CPU also allows to cleanup the model code a bit and removes the need to convert the list back to tensors.

I'm not sure whether SupportsMultiModal is the best place to define this, but it didn't require large code changes and still is backwards compatible so I think it's sensible to keep it close to merge_by_field_config.

Test Plan

vllm serve Qwen/Qwen3-VL-2B-Instruct-FP8 --limit-mm-per-prompt.video 0

vllm bench serve --backend openai-chat --model Qwen/Qwen3-VL-2B-Instruct-FP8 --endpoint /v1/chat/completions --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat --hf-split train --num-prompts 1000

Test Result

I generated a torch profile and verified that no more cudaStreamSync ops are present.

This also seems to slightly improve throughput by allowing more computation to be overlapped. Measured on a single L40s GPU though the benefit might be larger for faster GPUs.

Before:

============ Serving Benchmark Result ============
Successful requests:                     998
Failed requests:                         2
Benchmark duration (s):                  49.89
Total input tokens:                      94306
Total generated tokens:                  121122
Request throughput (req/s):              20.00
Output token throughput (tok/s):         2427.71
Peak output token throughput (tok/s):    14299.00
Peak concurrent requests:                998.00
Total Token throughput (tok/s):          4317.94
---------------Time to First Token----------------
Mean TTFT (ms):                          22076.45
Median TTFT (ms):                        20806.25
P99 TTFT (ms):                           45564.97
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          84.44
Median TPOT (ms):                        84.52
P99 TPOT (ms):                           126.30
---------------Inter-token Latency----------------
Mean ITL (ms):                           84.63
Median ITL (ms):                         73.15
P99 ITL (ms):                            514.58
==================================================

After:

============ Serving Benchmark Result ============
Successful requests:                     998
Failed requests:                         2
Benchmark duration (s):                  49.28
Total input tokens:                      94302
Total generated tokens:                  121141
Request throughput (req/s):              20.25
Output token throughput (tok/s):         2458.19
Peak output token throughput (tok/s):    15255.00
Peak concurrent requests:                998.00
Total Token throughput (tok/s):          4371.77
---------------Time to First Token----------------
Mean TTFT (ms):                          21838.22
Median TTFT (ms):                        20195.33
P99 TTFT (ms):                           44981.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          83.06
Median TPOT (ms):                        84.76
P99 TPOT (ms):                           128.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           83.61
Median ITL (ms):                         73.28
P99 ITL (ms):                            511.00
==================================================

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gemini-code-assist]

Code Review

This pull request introduces a valuable optimization by adding a multimodal_cpu_fields mechanism to prevent unnecessary GPU transfers of certain multimodal data. This is implemented cleanly by extending the SupportsMultiModal protocol and updating group_mm_kwargs_by_modality. The Qwen VL models are updated to use this feature, which simplifies the code and avoids performance-hurting CUDA synchronizations. However, there is a critical regression in the data-parallel path for qwen2_5_vl that needs to be addressed.

[gemini-code-assist]

Code Review

This pull request introduces a well-reasoned optimization to keep certain multimodal fields on the CPU, avoiding unnecessary GPU transfers and potential performance bottlenecks from CUDA synchronization. The changes are consistently applied across the relevant Qwen models and supporting utilities. The implementation is clean and the performance improvement, though modest in the provided benchmark, is a welcome enhancement. I've found one critical issue in qwen2_vl.py that needs to be addressed, but otherwise, the changes look solid.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to keep specified multimodal fields on the CPU, aiming to prevent unnecessary GPU-CPU data transfers and CUDA stream synchronizations for performance improvement. The changes are well-motivated and have been applied to the Qwen VL models. The implementation is largely correct, but I've identified a critical issue in qwen2_vl.py where a tensor is computed on the CPU but not moved to the GPU before being used in an operation with GPU tensors, which would cause a runtime error. I have provided a code suggestion to fix this. Apart from this issue, the changes are solid and align with the goal of enhancing performance and code clarity.

[DarkLight1337]

I am ok with putting this in the model definition.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lgeiger.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork