[Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target - #48787
Merged
Merged
Conversation
Signed-off-by: mgoin <mgoin64@gmail.com>
mgoin
requested review from
MatthewBonanni,
ProExpertProg,
WoosukKwon,
benchislett,
houseroad,
luccafong,
njhill,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256 and
youkaichao
as code owners
July 15, 2026 21:19
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
mgoin
changed the title
MatthewBonanni
approved these changes
Jul 15, 2026
MatthewBonanni
left a comment
MatthewBonanni
left a comment
Member
There was a problem hiding this comment.
LGTM, please just add a gsm8k with the mixed KV cache dtypes to make sure everything works correctly
Apply the drafter kv_cache_dtype override in the MRV2 autoregressive proposer (eagle/mtp via load_eagle_model), which builds its draft config independently of _create_draft_vllm_config. Normalize '-' to '_' in --speculative-config dotted keys so --speculative-config.kv-cache-dtype resolves to the kv_cache_dtype field instead of failing pydantic. Scoped to spec config, whose fields are all identifiers; free-form dict args (e.g. --hf-overrides) are untouched. Verified end-to-end: GLM-5.2-NVFP4-FP8 TP4 + dspark, target kv-cache fp8 with draft bfloat16, GSM8K 95.0% (200 q, 0 invalid). Signed-off-by: mgoin <mgoin64@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closed
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Currently running
vllm serve RedHatAI/GLM-5.2-NVFP4-FP8 --tensor-parallel-size 4 --spec-model GLM-5.2-speculator.dspark --spec-method dspark --spec-tokens 7 --kv-cache-dtype fp8will fail on Blackwell since the global--kv-cache-dtype fp8will apply to both the GLM 5.2 target and the DSpark drafter. We don't currently have an attention backend that simultaneously supports non-causal attention and FP8 kv cache, so the drafter fails to find a valid attention backend.With this PR, at least we have an escape hatch by adding
--speculative_config.kv_cache_dtype bfloat16so that the drafter is kept in BF16 while the target (majority of kv cache space) is running in FP8Test Plan
Test Result
Manually verified with
vllm serve RedHatAI/GLM-5.2-NVFP4-FP8 --tensor-parallel-size 4 --spec-model GLM-5.2-speculator.dspark --spec-method dspark --spec-tokens 7 --kv-cache-dtype fp8 --speculative_config.kv_cache_dtype bfloat16on B300 to get 95% on GSM8kEssential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.