[Model] Use mm_position to compute mrope positions for Qwen2-VL/2.5-VL

[YunzhuLu]

Purpose

Followup to #28399 and #28730. This PR optimizes the M-RoPE position computation for both Qwen2-VL and Qwen2.5-VL.

• Leveraging the pre-calculated mm_features to compute M-RoPE positions.
• Simplifying index generation with NumPy-based vectorization.

Test Plan

VLLM_WORKER_MULTIPROC_METHOD=spawn lm_eval --model vllm-vlm --model_args "pretrained=Qwen/Qwen2-VL-7B-Instruct,max_model_len=8192" --tasks chartqa --batch_size 1 --apply_chat_template --seed 42

Test Result

Qwen2-VL test results:

Before:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
chartqa	0	none	0	anywhere_accuracy	↑	0.6860	±	0.0093
		none	0	exact_match	↑	0.3476	±	0.0095
		none	0	relaxed_accuracy	↑	0.4112	±	0.0098

After:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
chartqa	0	none	0	anywhere_accuracy	↑	0.6860	±	0.0093
		none	0	exact_match	↑	0.3476	±	0.0095
		none	0	relaxed_accuracy	↑	0.4112	±	0.0098

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

---

Note

Optimizes M-RoPE position generation by leveraging pre-parsed multimodal features and vectorized index math.

• Adds iter_mm_grid_thw to iterate mm_features by mm_position.offset, yielding (t,h,w) grids and temporal scale (t_factor) for images/videos
• Replaces token-scanning logic with offset-based segmentation and NumPy (np.indices, np.broadcast_to) to build 3D position IDs, then converts back to torch
• Applies spatial merging (spatial_merge_size) and temporal scaling using second_per_grid_ts * tokens_per_second for video
• Minor imports: introduce Iterator, add NumPy; updates both qwen2_vl.py and qwen2_5_vl.py consistently

^{Written by Cursor Bugbot for commit c370ddc7807818b64d98d78873931f675ab66d60. This will update automatically on new commits. Configure here.}

---

Note

Optimizes and simplifies M-RoPE position computation by leveraging pre-parsed multimodal features and NumPy vectorization.

• Adds iter_mm_grid_thw to iterate mm_features by mm_position.offset, yielding (t,h,w) grids and temporal scaling via second_per_grid_ts * tokens_per_second
• Rewrites get_mrope_input_positions to segment by offsets and build positions with np.broadcast_to/np.indices, then converts to torch
• Applies spatial merging (spatial_merge_size) and temporal scaling (t_factor) uniformly; removes token-id scanning and per-step torch index construction
• Minor imports: introduce Iterator and numpy; identical updates in qwen2_vl.py and qwen2_5_vl.py

^{Written by Cursor Bugbot for commit 76c65502e40d29d100c94ed01a6daecb8e76a79c. This will update automatically on new commits. Configure here.}

---

Note

Optimizes and simplifies M-RoPE position computation for Qwen2-VL and Qwen2.5-VL.

• Adds iter_mm_grid_thw to iterate mm_features by mm_position.offset, yielding (t,h,w) and temporal scale (t_factor)
• Replaces token scanning in get_mrope_input_positions with offset-based segmentation and NumPy (np.indices, np.broadcast_to), then converts back to torch
• Applies spatial merging via spatial_merge_size and temporal scaling using second_per_grid_ts * tokens_per_second for video
• Minor imports/typing: introduce Iterator and numpy; updates mirrored in qwen2_vl.py and qwen2_5_vl.py

^{Written by Cursor Bugbot for commit 23cc4d3365298890fc41111c1fac0a3a3bd40e24. This will update automatically on new commits. Configure here.}

---

Note

^{Cursor Bugbot is generating a summary for commit 95645a5. Configure here.}

---

Note

Optimizes and simplifies M-RoPE position id generation by leveraging pre-parsed multimodal features and NumPy.

• Adds iter_mm_grid_thw to traverse mm_features by mm_position.offset, yielding (t,h,w) grids and temporal scale t_factor
• Rewrites get_mrope_input_positions to segment by offsets and build 3D IDs with np.indices/np.broadcast_to, applying spatial merge and second_per_grid_ts * tokens_per_second, then converts back to torch
• Minor imports/typing updates (Iterator, numpy); identical changes in qwen2_vl.py and qwen2_5_vl.py

^{Written by Cursor Bugbot for commit b38131b. This will update automatically on new commits. Configure here.}

---

Note

Optimizes and simplifies M-RoPE position computation using pre-parsed multimodal features and NumPy.

• Adds iter_mm_grid_thw to iterate mm_features by mm_position.offset, yielding (t,h,w) and temporal scale t_factor
• Rewrites get_mrope_input_positions to segment by offsets and build 3D IDs via np.indices/np.broadcast_to, then converts to torch
• Applies spatial merging (spatial_merge_size) and temporal scaling (second_per_grid_ts * tokens_per_second) for video
• Minor imports/typing updates (Iterator, numpy); identical changes in qwen2_vl.py and qwen2_5_vl.py

^{Written by Cursor Bugbot for commit b862112. This will update automatically on new commits. Configure here.}

---

Note

Optimizes and simplifies M-RoPE position id generation by leveraging pre-parsed multimodal features and vectorized index math.

• Adds iter_mm_grid_thw to iterate mm_features by mm_position.offset, yielding (t,h,w) grids and temporal scale t_factor
• Rewrites get_mrope_input_positions to segment by offsets and build 3D IDs via np.indices/np.broadcast_to, then converts to torch; removes token-id scanning and per-step torch index construction
• Applies spatial merging (spatial_merge_size) and temporal scaling using second_per_grid_ts * tokens_per_second for video
• Minor updates: import Iterator and numpy; identical changes in qwen2_vl.py and qwen2_5_vl.py

^{Written by Cursor Bugbot for commit b3a0317. This will update automatically on new commits. Configure here.}

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request refactors the M-RoPE position computation for Qwen2-VL and Qwen2.5-VL models. The changes significantly simplify the logic by introducing a helper method iter_mm_grid_thw to iterate over multimodal features and by using NumPy for vectorized position calculations. This is a great improvement in terms of both readability and performance. I've found one minor issue with a type hint that should be corrected for static analysis correctness.

[gemini-code-assist]

Code Review

This pull request introduces a significant optimization for M-RoPE position computation in Qwen2-VL and Qwen2.5-VL models. By leveraging pre-calculated multimodal features and vectorized NumPy operations, the new implementation is cleaner, more readable, and more efficient than the previous token-scanning approach. The introduction of the iter_mm_grid_thw helper function is a good abstraction. I've identified a potential crash when handling empty inputs, which was also present in the previous implementation, and have provided suggestions to address this edge case.

[gemini-code-assist]

np.concatenate will raise a ValueError if llm_pos_ids_list is empty (e.g., for an empty prompt), which will cause a crash. This should be handled to avoid unexpected failures on valid edge cases.

Suggested change

	llm_positions = np.concatenate(llm_pos_ids_list, axis=1).reshape(3, -1)
	llm_positions = np.concatenate(llm_pos_ids_list, axis=1).reshape(3, -1) if llm_pos_ids_list else np.empty((3, 0), dtype=np.int64)

[gemini-code-assist]

If llm_positions becomes an empty array after the change in the previous line, calling .max() on it will raise a ValueError. This also needs to be handled to prevent a crash.

Suggested change

	mrope_position_delta = (llm_positions.max() + 1 - len(input_tokens)).item()
	mrope_position_delta = (llm_positions.max() + 1 - len(input_tokens)).item() if llm_positions.size > 0 else 0

[gemini-code-assist]

np.concatenate will raise a ValueError if llm_pos_ids_list is empty (e.g., for an empty prompt), which will cause a crash. This should be handled to avoid unexpected failures on valid edge cases.

Suggested change

	llm_positions = np.concatenate(llm_pos_ids_list, axis=1).reshape(3, -1)
	llm_positions = np.concatenate(llm_pos_ids_list, axis=1).reshape(3, -1) if llm_pos_ids_list else np.empty((3, 0), dtype=np.int64)

[gemini-code-assist]

If llm_positions becomes an empty array after the change in the previous line, calling .max() on it will raise a ValueError. This also needs to be handled to prevent a crash.

Suggested change

	mrope_position_delta = (llm_positions.max() + 1 - len(input_tokens)).item()
	mrope_position_delta = (llm_positions.max() + 1 - len(input_tokens)).item() if llm_positions.size > 0 else 0

[YunzhuLu]

/CC @DarkLight1337, thanks for taking a look!

[YunzhuLu]

Hi @Isotr0py,
The CI is stuck on a test_mcp_tools.py failure (due to insufficient GPU memory on the CI node), which is unrelated to M-RoPE changes.
Could you please help bypass this or manually merge? Thanks!

[ywang96]

Out of curiosity - have you done profiling to quantify the perf gain from this PR (similar to #28730 (comment))? Thanks

[YunzhuLu]

@ywang96 Here are the trace results for Qwen2-VL:

Image

Before

### After

Video

Before

### After