[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP draft models by chaunceyjiang · Pull Request #46786 · vllm-project/vllm

chaunceyjiang · 2026-06-26T03:28:21Z

Some MTP models declare a single-tensor contract but return (logits_hidden, feedback_hidden) for final-norm correctness.

Purpose

Fix #46798
[Spec Decode] Handle tuple hidden states from MTP draft models

Test Plan

VLLM_USE_V2_MODEL_RUNNER=1 VLLM_NIXL_SIDE_CHANNEL_HOST=172.16.1.247 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 VLLM_ENGINE_READY_TIMEOUT_S=1800 vllm serve /mnt/data1/models/zai-org/GLM-5.2-FP8 \
            --served-model-name public/glm-52 \
            --trust-remote-code \
            --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_connector_extra_config": {"enforce_handshake_compat": false}}' \
            --chat-template-content-format=string \
            --tensor-parallel-size 8 \
            --tool-call-parser glm47 \
            --enable-auto-tool-choice \
            --reasoning-parser glm45 \
            --gpu-memory-utilization 0.92 \
            --enable-prompt-tokens-details \
            --speculative-config='{"method":"mtp","num_speculative_tokens":1}' \
            --shutdown-timeout 300 \
            --fingerprint-mode=none --max-model-len 1000
...
....
Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/eplb_utils.py", line 37, in wrapper
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     result = fn(self, *args, **kwargs)
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self.model_runner.profile_run()
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     result = fn(self, *args, **kwargs)
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000] Traceback (most recent call last):
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     result = fn(self, *args, **kwargs)
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/eplb_utils.py", line 37, in wrapper
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/executor/multiproc_executor.py", line 992, in worker_busy_loop
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/model_runner.py", line 607, in _dummy_run
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     result = fn(self, *args, **kwargs)
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/model_runner.py", line 607, in _dummy_run
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     output = func(*args, **kwargs)
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/model_runner.py", line 607, in _dummy_run
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self.speculator.propose(
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self.speculator.propose(
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self.speculator.propose(
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/model_runner.py", line 654, in profile_run
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/model_runner.py", line 607, in _dummy_run
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     hidden_states, sample_hidden_states = self._dummy_run(
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self.speculator.propose(
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]                                           ^^^^^^^^^^^^^^^^
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 234, in propose
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 234, in propose
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu_worker.py", line 438, in determine_available_memory
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 234, in propose
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self._prefill(
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self._prefill(
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self.model_runner.profile_run()
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self._prefill(
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 360, in _prefill
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 234, in propose
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 360, in _prefill
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 360, in _prefill
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     sample_hidden_states = last_hidden_states[last_token_indices]
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/eplb_utils.py", line 37, in wrapper
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     self._prefill(
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     sample_hidden_states = last_hidden_states[last_token_indices]
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     return func(*args, **kwargs)
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     sample_hidden_states = last_hidden_states[last_token_indices]
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]                            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     result = fn(self, *args, **kwargs)
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py", line 360, in _prefill
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]                            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]                            ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
(Worker_TP6 pid=3384772) ERROR 06-26 11:22:08 [multiproc_executor.py:1000] TypeError: only integer tensors of a single element can be converted to an index
(Worker_TP0 pid=3384741) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=3384743) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]     sample_hidden_states = last_hidden_states[last_token_indices]
(Worker_TP7 pid=3384775) ERROR 06-26 11:22:08 [multiproc_executor.py:1000] TypeError: only integer tensors of a single element can be converted to an index
(Worker_TP3 pid=3384746) ERROR 06-26 11:22:08 [multiproc_executor.py:1000]   File "/mnt/data4/jxy/vllm/vllm/v1/worker/gpu/model_runner.py", line 654, in profile_run
(Worker_TP5 pid=3384751) ERROR 06-26 11:22:08 [multiproc_executor.py:1000] TypeError: only integer tensors of a single element can be converted to an index

this pr:

rver pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=885189) INFO 06-26 11:19:45 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=885189) INFO:     Started server process [885189]
(APIServer pid=885189) INFO:     Waiting for application startup.
(APIServer pid=885189) INFO:     Application startup complete.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

TheEpicDolphin

Thanks for the fix! Now that model_returns_tuple is no longer used, could you please remove it from AutoRegressiveSpeculator and the classes that derive from it?

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

TheEpicDolphin

LGTM

…aft models (vllm-project#46786)

…aft models (vllm-project#46786) Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

…aft models (vllm-project#46786)

[Spec Decode] Handle tuple hidden states from MTP draft models

0895685

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

chaunceyjiang requested review from WoosukKwon, njhill and yewentao256 as code owners June 26, 2026 03:28

claude Bot reviewed Jun 26, 2026

View reviewed changes

mergify Bot added the v1 label Jun 26, 2026

chaunceyjiang changed the title ~~[Spec Decode] Handle tuple hidden states from MTP draft models~~ [Model Runner V2][Spec Decode] Handle tuple hidden states from MTP draft models Jun 26, 2026

chaunceyjiang mentioned this pull request Jun 26, 2026

[Model Runner V2][Spec Decode] Add Gemma4 MTP support #43241

Merged

njhill requested review from TheEpicDolphin and benchislett June 26, 2026 04:47

TheEpicDolphin suggested changes Jun 26, 2026

View reviewed changes

[Spec Decode] Handle tuple hidden states from MTP draft models

c36f9b0

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

chaunceyjiang requested a review from TheEpicDolphin June 27, 2026 08:02

TheEpicDolphin approved these changes Jun 28, 2026

View reviewed changes

ywang96 approved these changes Jun 28, 2026

View reviewed changes

ywang96 merged commit 11a1230 into vllm-project:main Jun 28, 2026
6 checks passed

This was referenced Jun 29, 2026

[Spec][V2] Support MTP speculative decoding under pipeline parallelism #46994

Open

[RFC]: MTP speculative decoding under pipeline parallelism (PP>1) #44697

Open

wincent8 pushed a commit to wincent8/vllm that referenced this pull request Jun 29, 2026

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP dr…

6a813f0

…aft models (vllm-project#46786)

rjrock pushed a commit to rjrock/vllm that referenced this pull request Jul 1, 2026

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP dr…

4f0676d

…aft models (vllm-project#46786)

noooop pushed a commit to noooop/vllm that referenced this pull request Jul 9, 2026

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP dr…

fe1e756

…aft models (vllm-project#46786) Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

Dao007forever pushed a commit to Dao007forever/vllm that referenced this pull request Jul 18, 2026

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP dr…

0360feb

…aft models (vllm-project#46786)

philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP dr…

c82a584

…aft models (vllm-project#46786)

plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP dr…

7e7e732

…aft models (vllm-project#46786)

Navigation Menu

Uh oh!

Uh oh!

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP draft models - #46786

[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP draft models#46786
ywang96 merged 2 commits into
vllm-project:mainfrom
chaunceyjiang:fix-mtp-tuple-hidden-states

chaunceyjiang commented Jun 26, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

TheEpicDolphin left a comment

Uh oh!

TheEpicDolphin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

chaunceyjiang commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

TheEpicDolphin left a comment

Choose a reason for hiding this comment

Uh oh!

TheEpicDolphin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chaunceyjiang commented Jun 26, 2026 •

edited

Loading