Skip to content

[Models][Qwen3VL] Fix MM encoder cuda graph input capture#42639

Closed
lgeiger wants to merge 1 commit into
vllm-project:mainfrom
lgeiger:qwen3vl-mm-cudagraph
Closed

[Models][Qwen3VL] Fix MM encoder cuda graph input capture#42639
lgeiger wants to merge 1 commit into
vllm-project:mainfrom
lgeiger:qwen3vl-mm-cudagraph

Conversation

@lgeiger

@lgeiger lgeiger commented May 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

Using a MM encoder CUDA graph currently fails with for Qwen3VL with:

File "/home/ubuntu/vllm/vllm/model_executor/models/qwen3_vl.py", line 371, in forward
        x = x.view(L, -1, self.temporal_patch_size, self.patch_size, self.patch_size)

RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 2, 16, 16] because the unspecified dimension size -1 can be any value and is ambiguous

Qwen2VL and Qwen2.5VL already use ceil to ensure that the capture capacity is never smaller than the token budget:

# Use ceil so captured capacity is never smaller than token_budget.
per_mm_item_output = (token_budget + max_batch_size - 1) // max_batch_size

# Use ceil here (not floor) so total captured capacity is never smaller
# than token_budget when token_budget is not divisible by max_batch_size
# (e.g., 324 budget with max_batch_size=8). Floor under-allocates
# input_buffer and can fail replay copy for valid single-item batches.
per_mm_item_output = (token_budget + max_batch_size - 1) // max_batch_size

This PR is a followup on #35963 and applies the same fix to Qwen3VL. /cc @Isotr0py @b-mu

Test Plan

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 --compilation-config '{"cudagraph_mm_encoder": true}'

Test Result

Server starts up properly

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
@lgeiger lgeiger force-pushed the qwen3vl-mm-cudagraph branch from eee7c70 to 289cedf Compare May 14, 2026 13:07

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the prepare_encoder_cudagraph_capture_inputs method in vllm/model_executor/models/qwen3_vl.py to use ceiling division when calculating per_mm_item_output, ensuring the captured capacity is never smaller than the token_budget. I have no feedback to provide as there were no review comments to evaluate.

@mergify mergify Bot added qwen Related to Qwen models nvidia labels May 14, 2026
@b-mu

b-mu commented May 14, 2026

Copy link
Copy Markdown
Contributor

This bug was fixed in #38040.

@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA May 14, 2026
@Isotr0py

Copy link
Copy Markdown
Member

Duplicate of #38040

@Isotr0py Isotr0py marked this as a duplicate of #38040 May 14, 2026
@Isotr0py Isotr0py closed this May 14, 2026
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nvidia qwen Related to Qwen models

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants