Repository navigation
Conversation
…nfig _check_and_update_cudagraph_mode passed the runner's vllm_config (the target model) into get_cudagraph_support for every KV group. For a draft-side group that divides the target's query-head count by the draft's num_kv_heads, so spec decode lost all CUDA graphs whenever the two models' head ratios differed. Resolve the config per group: draft-side groups (is_eagle_group) are judged against speculative_config.draft_model_config, following the existing dataclasses.replace precedent in the eagle loader. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
…t model's config Model Runner V2 asks each attention group's metadata builder for its CUDA-graph support with the runner's vllm_config, which holds the target model. For a draft layer that pairs the target's query-head count with the draft layer's KV cache spec. With a drafter of 40 query and 40 KV heads under a target with 24 query heads per rank, FlashInfer finds 24 % 40 != 0, reports UNIFORM_SINGLE_TOKEN_DECODE, and the runner drops FULL_AND_PIECEWISE to PIECEWISE for every layer, the target's included. Apply the V1 runner's rule from the previous commits to MRV2: draft layers are judged with speculative_config.draft_model_config. MRV2 already tags draft layers (is_draft_layer, set by both its EAGLE-style and draft_model speculators). Attention grouping copies the tag onto the group's KV cache spec, because merged group specs drop it, and both CUDA-graph checks (get_attn_cg_support and the varlen check used by adaptive verification) pick the config from it. Assisted-by: Claude Signed-off-by: Seyfal Sultanov <seyfal.s@outlook.com>
The V1 runner resolves its CUDA-graph mode twice: first for the minimal KV cache config of the CUDA-graph memory profiling pass, then for the real config. Only get_global_kv_cache_groups tagged is_eagle_group, so in the first pass the draft group was untagged and _config_for_kv_group judged it with the target's config. That pass already dropped FULL_AND_PIECEWISE to PIECEWISE, and resolve_cudagraph_mode_and_sizes keeps the downgrade. Tag the groups in get_kv_cache_groups_from_builder, which both configs are built from. Assisted-by: Claude Signed-off-by: Seyfal Sultanov <seyfal.s@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First PR of the eagle/v0.1.5 stack.
eagle/v0.1.5starts atspec-round-contract(upstream b5755c3 plus vllm-project#63205 and vllm-project#64844, the same tree as the upstream PR).This PR carries vllm-project#62964 as it stands upstream (4 commits; Yufeng He's commits keep his authorship):
The net diff equals the upstream PR's diff against b5755c3 (checked with
git diff, only hunk offsets differ).Stack: this PR -> 2 (vllm-project#62800, vllm-project#63279) -> 3 (fork patches) -> 4 (release version).
Validation
Gates for the whole stack are recorded on the release PR (4): fork CPU tests, wheel build, GPU smoke on Qwen3.8-27B and the chain A/B against v0.1.4.