Repository navigation
[Model][Spec Decode] Refactor #41834 with useful improvements - #63996
Merged
Merged
Conversation
…caches - mHC: DeepseekV4Model's layer loop lets the unfused mHC post/pre (T > 32) write the next residual into the dead input residual instead of allocating a new [T, hc_mult, hidden] bf16 buffer. Outputs are bit-identical; the registered custom op is unchanged. - DeepSeek V4/V4.1: skip the pre-hc_head _mtp_hidden_buffer for DSpark, which drafts from the target's aux hidden states. - MRV1 drafter: drop the cached token->request mapping and num_computed_tokens after the in-place layout/seq_lens rewrites and in extend_all_queries_by_N, so builders inside the draft loop see the current layout. Salvaged from vllm-project#41834. Co-authored-by: alexbi29 <alexbi29@users.noreply.github.com> Co-authored-by: jasl <jasl9187@hotmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: lazymio <mio@lazym.io>
wtdcode
requested review from
AndreasKaratzas,
MatthewBonanni,
WoosukKwon,
alexm-redhat,
benchislett,
dllehr-amd,
hongxiayang,
mgoin,
njhill,
tjtanaa,
tlrmchlsmth,
yewentao256,
youkaichao,
zhuohan123 and
zyongye
as code owners
October 27, 2026 08:16
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
mergify
Bot
added
deepseek
|
✅ @wtdcode, CI is now available for this PR.
|
Member
|
/ci run |
|
✅ Triggered Buildkite CI #97559 for commit |
Harry-Chen
approved these changes
Oct 27, 2026
Harry-Chen
left a comment
Harry-Chen
left a comment
Member
There was a problem hiding this comment.
Thanks for the improvement!
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
#41834 has been stale for a while, and many key contributions have been superseded by other PRs. This PR extracts the useful parts of that huge PR and conducts end-to-end experiments on two Sparks to ensure it works well. The authorship is also acknowledged.
Claims
In general, this PR comprises:
8192token prefill setup, this saves 256 MiB VRAM._mtp_hidden_bufferunder DSpark. DSpark does not need and access these memories and this saves another 256 MiB VRAM per card.Validation
Reproduced on two Sparks. I did several end-to-end experiments using DeepSeek-v4-Flash and ensured that the tokens output is the same.
No performance regression is observed, and I confirm the VRAM savings and draft acceptance rate benefits.
Details
See above.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.