Support multimodal speculative decoding in draft_model mode

[EanWang211123]

Purpose

This PR follows the needs discussed in the following issues and adds support for multimodal speculative decoding in vLLM’s draft_model (both parallel and non-parallel draft mode) mode:

• [Tracking Issue][Performance]: Speculative decoding performance/QoL improvements #28947
• [Feature][Speculative Decoding]: Multi Modal Draft Model Support #33458

Currently, vLLM does not support speculative decoding for multimodal models in draft_model mode. This PR adds support for the following scenarios:

1. Target model + draft model are both multimodal models (VLMs)
   Example: Qwen2.5-VL-32B-Instruct + Qwen2.5-VL-3B

2. Cross-modal speculative decoding (target model is VLM, draft model is text-only)
   Example: Qwen2.5-VL-32B-Instruct + Qwen3-0.6B Qwen3-VL-8B-Instruct + amdPARD-Qwen3-0.6B

The cross-modal adaptation is implemented with reference to the speculative decoding design in SGLang standalone mode.

---

Background and Problem Statement

There are several key issues in the current vLLM draft_model path for multimodal scenarios:

• Multimodal support is determined using the target model config, which causes misclassification in the “VLM target + text draft” case

• M-RoPE / positional encoding cache and kernel position handling are incorrect in some paths, which can lead to:

  • startup failures (NotImplementedError / AttributeError)
  • position corruption under concurrency
  • reduced acceptance rate

• When both target and draft are VLMs (both using M-RoPE), the 3D M-RoPE positions are not preserved correctly

• Heterogeneous vocab-size combinations are overly restricted, blocking some practically safe combinations (e.g., using Qwen3-0.6B as draft for Qwen2.5-VL-32B)

This PR focuses on correctness and functionality enablement, so that multimodal / cross-modal speculative decoding can run correctly in draft_model mode.

---

Changes

1. eagle.py (6 changes)

• Use draft model config (instead of target model config) to determine supports_mm_inputs
  Avoids misclassifying the draft model as multimodal in the VLM target + text draft scenario and entering the wrong code path.

• Always initialize self.positions
  Fixes potential AttributeError when the Triton kernel writes position buffers in the M-RoPE draft-model path.

• Fix incorrect broadcasting in _set_positions under concurrency
  Prevents 1D positions from being accidentally reduced to a scalar and broadcasting request-0’s position to the entire batch.

• Fix kernel_positions computation in set_inputs_first_pass for cross-modal cases
  In the target(M-RoPE) + draft(1D RoPE) scenario, computes the real sequential positions instead of using compressed M-RoPE time positions, preventing draft position / KV-cache misalignment.

• Fix mrope_positions filling for dual-VLM (dual M-RoPE) scenarios
  Directly copies the target’s 3D M-RoPE positions for input tokens, avoiding corruption of image-token H/W dimensions caused by kernel-written 1D positions.

• Complete positional-encoding compatibility paths for multimodal / cross-modal speculative decoding
  Unifies position propagation and cache alignment logic across M-RoPE and 1D RoPE combinations for target/draft pairs.

2. speculative.py (1 change)

• Relax heterogeneous vocab validation
  Only raises when draft_vocab > target_vocab (possible out-of-bounds risk); allows safe combinations where draft_vocab < target_vocab (may slightly reduce acceptance rate).

---

Summary: Supported Scenarios

Scenario	Before	After
VLM target + text draft	Startup error NotImplementedError	✅ Works, positional encoding aligned correctly
VLM target + VLM draft	Startup error AttributeError, incorrect M-RoPE positions	✅ Works, full 3D M-RoPE preserved
Text target + text draft	Already worked	✅ Unaffected

---

Test Plan

Dataset: MMStar
Environment: 4090D
Target model: Qwen2.5-VL-32B-Instruct
Draft models: Qwen3-0.6B, Qwen2.5-VL-3B

Commands

CUDA_VISIBLE_DEVICES=4,5,6,7 \
vllm serve /models/Qwen2.5-VL-32B-Instruct/ \
--host 0.0.0.0 \
--port 30000 \
-tp 4 --max-model-len 10240 \
--gpu-memory-utilization 0.9  --max-cudagraph-capture-size 32 \
--speculative_config '{"model": "/models/Qwen3-0.6B/", "num_speculative_tokens": 3, "method": "draft_model"}'

CUDA_VISIBLE_DEVICES=4,5,6,7 \
vllm serve /models/Qwen2.5-VL-32B-Instruct/ \
--host 0.0.0.0 \
--port 30000 \
-tp 4 --max-model-len 10240 \
--gpu-memory-utilization 0.9  --max-cudagraph-capture-size 32 \
--speculative_config '{"model": "/models/Qwen2.5-VL-3B/", "num_speculative_tokens": 3, "method": "draft_model"}'

vllm bench serve \
--backend openai-chat \
--port 30000 \
--host 127.0.0.1 \
--endpoint /v1/chat/completions \
--model /models/Qwen2.5-VL-32B-Instruct/ \
--dataset-name hf \
--dataset-path /workspace/datasets/mmstar/ \ 
--hf-name Lin-Chen/MMStar \
--hf-split val \
--num-prompts 16 \
--max-concurrency 1 \
--temperature 0 \
--top-p 1.0 \
--seed 123

---

Test Result

baseline

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Maximum request concurrency:             1
Benchmark duration (s):                  38.06
Total input tokens:                      281
Total generated tokens:                  1521
Request throughput (req/s):              0.42
Output token throughput (tok/s):         39.96
Peak output token throughput (tok/s):    47.00
Peak concurrent requests:                2.00
Total token throughput (tok/s):          47.35
---------------Time to First Token----------------
Mean TTFT (ms):                          330.66
Median TTFT (ms):                        149.12
P99 TTFT (ms):                           2602.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          21.80
Median TPOT (ms):                        21.71
P99 TPOT (ms):                           23.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           21.54
Median ITL (ms):                         21.70
P99 ITL (ms):                            22.24
==================================================

draft model qwen3-0.6B

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Maximum request concurrency:             1
Benchmark duration (s):                  48.68
Total input tokens:                      281
Total generated tokens:                  1482
Request throughput (req/s):              0.33
Output token throughput (tok/s):         30.44
Peak output token throughput (tok/s):    18.00
Peak concurrent requests:                2.00
Total token throughput (tok/s):          36.21
---------------Time to First Token----------------
Mean TTFT (ms):                          363.07
Median TTFT (ms):                        165.07
P99 TTFT (ms):                           2761.91
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.91
Median TPOT (ms):                        30.90
P99 TPOT (ms):                           35.21
---------------Inter-token Latency----------------
Mean ITL (ms):                           57.15
Median ITL (ms):                         58.06
P99 ITL (ms):                            64.59
---------------Speculative Decoding---------------
Acceptance rate (%):                     33.65
Acceptance length:                       2.01
Drafts:                                  734
Draft tokens:                            2202
Accepted tokens:                         741
Per-position acceptance (%):
  Position 0:                            54.90
  Position 1:                            30.25
  Position 2:                            15.80
==================================================

draft: qwen2.5-vl-3b

============ Serving Benchmark Result ============
Successful requests:                     16
Failed requests:                         0
Maximum request concurrency:             1
Benchmark duration (s):                  69.35
Total input tokens:                      281
Total generated tokens:                  1506
Request throughput (req/s):              0.23
Output token throughput (tok/s):         21.72
Peak output token throughput (tok/s):    12.00
Peak concurrent requests:                2.00
Total token throughput (tok/s):          25.77
---------------Time to First Token----------------
Mean TTFT (ms):                          396.88
Median TTFT (ms):                        224.33
P99 TTFT (ms):                           2599.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          42.83
Median TPOT (ms):                        42.15
P99 TPOT (ms):                           53.87
---------------Inter-token Latency----------------
Mean ITL (ms):                           93.19
Median ITL (ms):                         94.94
P99 ITL (ms):                            106.86
---------------Speculative Decoding---------------
Acceptance rate (%):                     43.08
Acceptance length:                       2.29
Drafts:                                  660
Draft tokens:                            1980
Accepted tokens:                         853
Per-position acceptance (%):
  Position 0:                            62.12
  Position 1:                            41.21
  Position 2:                            25.91
==================================================

• Target model: Qwen3-VL-32B-Instruct
• Draft model: Qwen3-0.6B

Key Serving Benchmark Results

Metric	Baseline	Speculative Decoding
Benchmark duration (s)	26.24	19.93
Request throughput (req/s)	0.38	0.50
Output token throughput (tok/s)	44.63	58.77
Total token throughput (tok/s)	132.40	174.34
Mean TTFT (ms)	306.50	182.05
Mean TPOT (ms)	19.96	15.59
Acceptance rate (%)	N/A	49.17
Acceptance length	N/A	1.49

This shows around 30% throughput improvement when using speculative decoding on H20 TP1.

---

Result Summary and Notes

• qwen2.5-32b-vl + qwen3-0.6B average acceptance length: 2.01
• qwen2.5-32b-vl + qwen2.5-vl-3b average acceptance length: 2.29

The actual throughput improvement is limited because ITL increases significantly (from ~20 ms to ~60–90+ ms).

Based on analysis, this large increase does not appear in SGLang, and the same behavior is also observed in text-only speculative decoding tests (e.g., Qwen3-32B + Qwen3-0.6B). Therefore, this is likely not caused by the multimodal logic introduced in this PR, but instead may be related to the draft model path not yet supporting full CUDA Graph.

The focus of this PR is functionality support and positional-encoding correctness fixes. Draft model FULL CUDA GRAPH FEATURE performance optimization can be addressed in follow-up PRs.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Hi @EanWang211123, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[gemini-code-assist]

Code Review

This pull request adds support for multimodal speculative decoding in draft_model mode, covering both dual-VLM and cross-modal (VLM target + text-only draft) scenarios. The changes primarily focus on correctness by fixing issues related to positional encoding (M-RoPE), input handling for draft models, and relaxing vocabulary size validation. My review found the changes to be well-implemented and thoroughly explained. I have one suggestion to improve the clarity of an error message related to vocabulary size validation.

[mergify]

Hi @EanWang211123, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

Hi @EanWang211123, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[benchislett]

Please try to simplify your comments. Many of them are diluting the usefulness by repeating what the code is clearly doing

[mergify]

Hi @EanWang211123, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @EanWang211123.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[tomasruizt]

Why is this change necessary? How is passing the hidden_states related to multimodal support?

[EanWang211123]

After this PR, self.supports_mm_inputs is derived from the draft model config, so it's True whenever the draft itself is multimodal. But only EAGLE-style drafts (which share the target's architecture) actually receive target-precomputed mm_embeds via self.inputs_embeds — a standalone DraftModelProposer wrapping a VLM does not (it treats image placeholders as regular input_ids, and self.inputs_embeds is never populated on that path).

pass_hidden_states_to_model is the existing discriminator: True for EAGLE/DFlash, False for DraftModelProposer. Without the extra guard, a standalone multimodal draft would read uninitialised self.inputs_embeds and the dummy_run-captured cudagraph would diverge from the real propose path (input_ids=None vs tensor).

[tomasruizt]

@EanWang211123

1. Could you add unit test(s) to prove the code is working as intended? You can see examples in tests/v1/e2e/spec_decode/test_spec_decode.py
2. You mentioned that text-based SD is also slow (Qwen3-32B + Qwen3-0.6B), which is very unexpected. Do you still have the bench results? Are you using TP=4 here as well? This is worth tracking / reporting in an issue by itself, since that combination should definitely give speedups.
3. It only makes sense to complicate the code if we achieve speedups with it (e.g. reduce TPOT), which atm is not happening. I understand you focused first on correctness rather than speed. However, I would expect your SD extension to yield speedups today even without full CUDA graph. This result makes me suspicious that something is off atm. Addressing point (2) would clarify whether it is reasonable to expect speedups from your PR or not.

Let me know if there is anything I'm missing 👍

[tomasruizt]

Could we use this structure? This way the reader knows the function is not even triggered in text-only SD.

if self.uses_mrope:
    self._populate_mrope_...()

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @EanWang211123.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[EanWang211123]

@tomasruizt @benchislett

---

Multimodal Speculative Decoding Update

The relevant code has been refactored to support draft-model mode for multimodal speculative decoding, and additional tests have been conducted on H20.

• Target model: Qwen3-VL-32B-Instruct
• Draft model: Qwen3-0.6B

Key Serving Benchmark Results

Metric	Baseline	Speculative Decoding
Benchmark duration (s)	26.24	19.93
Request throughput (req/s)	0.38	0.50
Output token throughput (tok/s)	44.63	58.77
Total token throughput (tok/s)	132.40	174.34
Mean TTFT (ms)	306.50	182.05
Mean TPOT (ms)	19.96	15.59
Acceptance rate (%)	N/A	49.17
Acceptance length	N/A	1.49

This shows around 30% throughput improvement when using speculative decoding on H20 TP1.

---

Reason for Limited Acceleration in Previous Experiments

Regarding the previously observed limited acceleration with the Qwen3-32B + Qwen3-0.6B setup, I believe the main reason was multi-GPU communication overhead.

The previous experiments were conducted on 4 × RTX 4090 with TP4, without NVLink. In contrast, the current experiments use H20 TP1, which avoids the same level of inter-GPU communication overhead and achieves around 30% throughput improvement.

[tomasruizt]

@EanWang211123 thanks for this PR! I tried to reproduce the benchmark with multimodal draft_model speculative decoding and the server crashes on the first request.

Setup: Qwen/Qwen2.5-VL-3B-Instruct as both target and draft model, num_speculative_tokens=3, served via vllm bench serve against the Lin-Chen/MMStar (val) dataset.

Symptom: the benchmark request returns 500 Internal Server Error; the EngineCore dies with an IndexError raised from the draft proposer's first-pass embedding step (llm_base_proposer.build_model_inputs_first_pass). The multimodal mask and the embeds buffer disagree by one token (mask [282] vs tensor [283, 2048]):

Traceback (most recent call last):
  File ".../vllm/v1/engine/core.py", line 463, in step
    model_output = self.model_executor.sample_tokens(grammar_output)
  ...
  File ".../vllm/v1/worker/gpu_model_runner.py", line 5050, in propose_draft_token_ids
    draft_token_ids = self.drafter.propose(
  File ".../vllm/v1/spec_decode/llm_base_proposer.py", line 575, in propose
    model_kwargs, slot_mapping_size = self.build_model_inputs_first_pass(
  File ".../vllm/v1/spec_decode/llm_base_proposer.py", line 960, in build_model_inputs_first_pass
    self.inputs_embeds[:num_tokens] = self.model.embed_input_ids(
  File ".../vllm/model_executor/models/interfaces.py", line 405, in embed_input_ids
    return _merge_multimodal_embeddings(
  File ".../vllm/model_executor/models/utils.py", line 492, in _merge_multimodal_embeddings
    inputs_embeds[is_multimodal] = mm_embeds_flat.to(dtype=input_dtype)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^
IndexError: The shape of the mask [282] at index 0 does not match the shape of the indexed tensor [283, 2048] at index 0

This looks like an off-by-one between num_tokens used to slice self.inputs_embeds and the is_multimodal mask length in the draft model's first pass. (Full server log attached.)

Environment: engine self-reports v0.22.1rc1.dev330+g6deb05e0e (this PR branch, multimodal-draft-support). Single GPU, CUDA 13.0, torch 2.11.0+cu130.

Attached shell scripts and logs:
bench.log
bench.sh
serve.log
serve.sh

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!