Skip to content

[Model Runner V2] Support multi-modal embeddings for spec decode model - #36097

Merged
WoosukKwon merged 3 commits into
vllm-project:mainfrom
TheEpicDolphin:gdelfin/mrv2-spec-decode-mm-embeddings
Mar 22, 2026
Merged

[Model Runner V2] Support multi-modal embeddings for spec decode model#36097
WoosukKwon merged 3 commits into
vllm-project:mainfrom
TheEpicDolphin:gdelfin/mrv2-spec-decode-mm-embeddings

Conversation

@TheEpicDolphin

@TheEpicDolphin TheEpicDolphin commented Mar 5, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Currently the multimodal inputs from the target model vision encoder are not passed to the draft model. Draft models trained on vision embeddings need to merge these inputs with the input token ids to function properly. This behavior is supported in MRV1, but not yet in MRV2. This PR addresses that gap.

Manual Testing

Verified no regression in acceptance rate for the following vision models:

1. VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen2.5-VL-7B-Instruct --no-enable-prefix-caching --tensor-parallel-size=1 --data-parallel-size=1 --speculative-config '{"method": "eagle3", "model": "Rayzl/qwen2.5-vl-7b-eagle3-sgl", "num_speculative_tokens": 3}'

2. VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3-VL-4B-Instruct --no-enable-prefix-caching --tensor-parallel-size=1 --data-parallel-size=1 --speculative-config '{"method": "eagle3", "model": "AngelSlim/Qwen3-VL-4B-Instruct_eagle3", "num_speculative_tokens": 3}'

Unfortunately, I couldn't find any target/draft model pairs that are both trained on vision embeddings and also currently work on MRV2 to test on and observe an increase in acceptance rate. So I instead tested the scenario where the draft model receives vision embeddings by forcing self.supports_mm_inputs to True, and checking that the correct number of embeddings are passed and merged with the input ids during draft model prefill.

@mergify mergify Bot added the v1 label Mar 5, 2026
@mergify

mergify Bot commented Mar 5, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @TheEpicDolphin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Mar 5, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for multi-modal embeddings in the speculative decoding model runner. The changes involve propagating scheduler_output to access encoder inputs and preparing multi-modal embeddings for the speculative step. The logic appears correct, but I've identified an opportunity to improve code maintainability by refactoring duplicated code.

Comment thread vllm/v1/worker/gpu/model_runner.py Outdated
@WoosukKwon

Copy link
Copy Markdown
Collaborator

Can you please rebase?

@TheEpicDolphin
TheEpicDolphin force-pushed the gdelfin/mrv2-spec-decode-mm-embeddings branch 2 times, most recently from df1a75e to 244b038 Compare March 9, 2026 22:11
@mergify mergify Bot added llama Related to Llama models speculative-decoding labels Mar 9, 2026
@TheEpicDolphin
TheEpicDolphin force-pushed the gdelfin/mrv2-spec-decode-mm-embeddings branch 3 times, most recently from bc2e42e to 8ffc9b2 Compare March 10, 2026 05:01
@mergify mergify Bot removed the needs-rebase label Mar 10, 2026
@TheEpicDolphin
TheEpicDolphin force-pushed the gdelfin/mrv2-spec-decode-mm-embeddings branch 2 times, most recently from e9dadcd to e95f68f Compare March 10, 2026 20:57
@mergify

mergify Bot commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @TheEpicDolphin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Mar 10, 2026
@TheEpicDolphin
TheEpicDolphin force-pushed the gdelfin/mrv2-spec-decode-mm-embeddings branch 7 times, most recently from b3fdf54 to 245e44f Compare March 10, 2026 23:47
@mergify mergify Bot removed the needs-rebase label Mar 10, 2026
@TheEpicDolphin
TheEpicDolphin force-pushed the gdelfin/mrv2-spec-decode-mm-embeddings branch 2 times, most recently from 4e6dfc4 to c251158 Compare March 11, 2026 00:33
@TheEpicDolphin
TheEpicDolphin marked this pull request as ready for review March 11, 2026 00:52
@mergify

mergify Bot commented Mar 20, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @TheEpicDolphin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Mar 20, 2026
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
@WoosukKwon WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 22, 2026
@mergify mergify Bot removed the needs-rebase label Mar 22, 2026

@WoosukKwon WoosukKwon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the PR! I've made small edits in the PR for minor code cleanup.

@WoosukKwon
WoosukKwon enabled auto-merge (squash) March 22, 2026 08:47
@WoosukKwon
WoosukKwon disabled auto-merge March 22, 2026 09:48
@WoosukKwon
WoosukKwon merged commit b3e8460 into vllm-project:main Mar 22, 2026
59 checks passed
RhizoNymph pushed a commit to RhizoNymph/vllm that referenced this pull request Mar 26, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
SouthWest7 pushed a commit to SouthWest7/vllm that referenced this pull request Mar 27, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm that referenced this pull request Mar 27, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
nithinvc pushed a commit to nithinvc/vllm that referenced this pull request Mar 27, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>

Signed-off-by: Nithin Chalapathi <nithin.ch10@gmail.com>
JiantaoXu pushed a commit to JiantaoXu/vllm that referenced this pull request Mar 28, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
mtparet pushed a commit to blackfuel-ai/vllm that referenced this pull request Apr 9, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
vllm-project#36097)

Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

llama Related to Llama models ready ONLY add when PR is ready to merge/full CI is needed speculative-decoding v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants