Skip to content

[Quant][MTP] Reuse packed FP8 logits inputs - #51947

Draft
WoosukKwon wants to merge 1 commit into
vllm-project:mainfrom
WoosukKwon:agent/deepseek-v32-mtp-fp8
Draft

WoosukKwon wants to merge 1 commit into
vllm-project:mainfrom
WoosukKwon:agent/deepseek-v32-mtp-fp8

Conversation

@WoosukKwon

Copy link
Copy Markdown
Collaborator

Purpose

Fuse the sparse DeepSeek V3.2 / GLM-5.2 MTP layer's final all-reduce and RMSNorm
with packed FP8 quantization for its shared LM head.

The draft layer returns two deliberately different values:

  • a packed QuantizedActivation for logits projection; and
  • the post-final-norm BF16 hidden state recycled into the next MTP position.

The logits processor and both autoregressive speculators preserve and index this
contract without unpacking and requantizing it.

Dependencies

The branch contains only the seven-file MTP integration diff against main.
Until #51942 lands, isolated mypy reports the expected missing collective helper.

Duplicate-work check

No issue number was provided. I searched open PRs for DeepSeek V3.2 fused norm quant MTP. The only exact prior implementation was the superseded draft
#51936.

Tests

The portions independent of the missing prerequisite symbol pass directly:

.venv/bin/python -m pytest \
  tests/fusion/test_quant_activation_contract.py \
  tests/v1/worker/test_gpu_autoregressive_speculator.py -q
# 23 passed

On the integrated branch:

.venv/bin/python -m pytest tests/models/deepseek_v32/test_sequence_parallel.py -q
# 5 passed

All hooks other than the expected prerequisite-related mypy error pass.

Model evaluation

Two-node GB200, TP=8, C1, random 8K input / 1K output, MTP=3, ten measured
prompts, temperature 0:

Model Overall Position 0 Position 1 Position 2
zai-org/GLM-5.2-FP8 83.47% 90.83% 83.00% 76.57%
nvidia/GLM-5.2-NVFP4 89.82% 93.79% 90.15% 85.53%

Both runs completed all ten prompts without failures.

AI assistance disclosure

This change was developed with OpenAI Codex assistance. This is a blocked draft
PR; the human submitter must review every changed line and confirm they
understand and can defend the change before marking it ready.

Fuse the final draft all-reduce/RMSNorm with LM-head quantization while preserving BF16 feedback hidden states across MTP positions.

Co-authored-by: OpenAI Codex <noreply@openai.com>

Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
@mergify mergify Bot added deepseek Related to DeepSeek models quantization mrv2 Model Runner V2 specific labels Aug 12, 2026
@mergify

mergify Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @WoosukKwon.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models mrv2 Model Runner V2 specific needs-rebase quantization speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant