[Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags by andakai · Pull Request #61455 · vllm-project/vllm

andakai · 2026-10-13T16:41:48Z

Overview

is_eagle_group decides which KV cache groups take the EAGLE last-block drop. The planner used to guess these groups from non_causal_multi_token_decode and from "the group holding the last registered layer" (MTP, DeepSeek-V4), and only on some grouping paths. This PR replaces the guesses with an explicit per-layer tag that the runners report, so a group is flagged iff it holds a draft layer. Supersedes #41640 (closed; its compared per-model field changed grouping) and #56026 (adds another guessing rule).

Claims

DFlash / DSpark drafters on hybrid models no longer make every group take the drop, so Mamba groups keep their prefix-cache reuse ([Spec Decode][dflash] Drafter KV group is never identified, so prefix-cache reuse is silently disabled for the Mamba groups #57032, [Bug]: Speculative decoding (mtp and dflash) silently disables prefix-cache hits for hybrid GDN models on nightly; worked on v0.24.0 #54360).
A drafter spanning several groups (e.g. DeepSeek-V4 / V4.1 DSpark) gets every one of its groups flagged, not only the group holding the last layer.
Gemma4 MTP no longer flags an unrelated target group. Its drafter shares the target's KV, so no group holds draft-owned KV and the coordinator's existing "flag all" fallback applies.
The flags survive pipeline parallelism. Before, the scheduler config (worker 0's projection) cleared the flag of every group with no layers on worker 0, so with PP >= 2 the draft groups on the last stage were never flagged and the coordinator flagged every group.
New draft models need no planner code to be identified. Out-of-tree runners must tag draft specs to benefit; untagged runners keep the conservative "flag all" fallback.

Validation

Unit tests (CPU; the only failures need a GPU and fail the same way without this PR):

pytest tests/v1/core/test_kv_cache_planning.py tests/v1/core/test_kv_cache_utils.py tests/v1/worker/test_attn_utils.py tests/v1/spec_decode/test_mtp.py

test_mtp.py and test_attn_utils.py also pass in full on one GB200 (84 passed). tests/v1/kv_connector/unit gives identical results with and without this PR (1843 passed, 10 skipped; the same 97 GPU-only failures on CPU).

test_eagle_groups_are_the_groups_holding_draft_layers checks, through get_kv_cache_configs and generate_scheduler_kv_cache_config, that the scheduler flags exactly the groups holding draft layers for a uniform model, a Qwen3.5-like GDN hybrid with MTP, and an MLA + SWA-MLA drafter spanning two groups, with pp=1 and pp=2. Reverting the PP projection fix makes the pp=2 case fail.

End-to-end, Qwen/Qwen3.5-4B with {"method": "mtp", "num_speculative_tokens": 3}, prefix caching on, one GB200, main (ae8f41238c) vs this PR. The workload is 8 prompts sharing a ~2.2k-token prefix, sent in two waves, greedy, 128 output tokens:

Metric	main	PR	PR repeat
Prefix hit rate, wave 1 / wave 2	0.4409 / 0.5039	0.4409 / 0.5039	0.4409 / 0.5039
Mean acceptance length	3.3065	3.3065	3.3065
Wave-2 outputs equal wave-1 outputs	yes	yes	yes
GSM8K-250 flexible-extract	0.760	0.764	0.752

The GSM8K delta (1/250) is below the PR repeat spread (3/250). Commands: vllm serve Qwen/Qwen3.5-4B --enable-prefix-caching --max-model-len 8192 --speculative-config '{"method":"mtp","num_speculative_tokens":3}', then lm_eval --model local-completions --tasks gsm8k --limit 250 --gen_kwargs temperature=0.0.

The DFlash-on-hybrid case could not be run end to end: the local Ornith-1.5-35B-A3B DFlash2 draft sets sample_from_anchor: true, which DFlash rejects at startup on main and on this PR alike.

Details

Speculators set AttentionLayerBase.is_draft_layer on their layers at load time (V2 already did; V1 now does too). Both runners copy it onto KVCacheSpec.is_draft.
KVCacheSpec.is_draft describes ownership, not cache format, so it uses compare=False: it never changes equality, hashing, grouping, or the cross-worker spec check. The merge assertions skip compare=False fields, because a group can mix target and draft layers (Qwen3.5 MTP). The field is only meaningful on per-layer specs; groups expose KVCacheGroupSpec.is_eagle_group.
get_kv_cache_configs sets is_eagle_group after get_kv_cache_groups_from_builder, so it also covers groupings from a model's SupportsKVCacheGrouping or a platform builder. Flags are only set when use_eagle_block_drop() is true.
_project_kv_cache_groups_to_worker now keeps the global flag. Every is_eagle_group consumer (coordinator, OffloadingConnector scheduler, Mooncake store coordinator) runs on the scheduler side.
Behavior change: on uniform models the OffloadingConnector now drops the tail chunk like the GPU coordinator. Newly flagged groups recompute one tail chunk on external hits; this is a performance cost, not a correctness one.
HiSparse resident/hot specs are not tagged; their groups are not prefix-cacheable, so no hit lookup reads the flag.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance (Claude) was used to prepare this change. The human submitters reviewed every changed line and ran the tests above.

🤖 Generated with Claude Code

… guessing EAGLE groups The KV cache planner guessed which groups hold drafter layers: it borrowed MLAAttentionSpec.non_causal_multi_token_decode, flagged the group holding the last registered layer for method == "mtp" or DeepSeek-V4/V4.1, and only did so on the packed and uniform-page-size paths. DFlash drafters were never found (so every group, Mamba included, took the EAGLE drop), Gemma4's KV-sharing MTP flagged an unrelated target group, and a drafter spanning several groups had only one of them flagged. The speculator already knows its layers. Mark them with is_draft_layer at load time (V2 already did; V1 now does too), carry the mark on the new KVCacheSpec.is_draft_layer field (excluded from equality so grouping is unchanged), and flag a group as EAGLE in get_kv_cache_configs iff it holds a draft layer and EAGLE block drop is enabled. All grouping-path heuristics are removed. Signed-off-by: Dakai An <dakaian108@gmail.com>

…pec-from-runner Signed-off-by: Dakai An <dakaian108@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

JaredforReal

LGTM~

mergify · 2026-10-16T23:36:24Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @andakai.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

ivanium

Nice work! I made some minor tweaks on variable naming and a small change to fix PP projection.

github-actions · 2026-10-17T07:08:38Z

✅ @andakai, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

ivanium · 2026-10-17T07:08:41Z

/ci run

github-actions · 2026-10-17T07:09:00Z

✅ Triggered Buildkite CI #95560 for commit 693983d65654.

Resolve the conflict in tests/v1/core/test_kv_cache_planning.py: take main's spec-decode grouping tests (vllm-project#61455, explicit draft-layer tags, which removed _spec_decode_grouping_config) and keep this branch's parallel_config in _grouping_config, which the QSA DCP grouping reads. Signed-off-by: kimseunghyun <hellonypcteam@gmail.com>

Step5 support added a second SWA retention scheme for EAGLE/MTP prefix hits: a larger retained window (window + proof block + scheduler-block drift), matching admission caps and pool sizing, and a coordinator set of proof groups. Main already covers the same hits with eagle_proof_margin, the multi-module MTP extra_retained_tokens and replay boundaries, and vllm-project#61455 now flags every drafter group, so restore main's SWA retention, sizing and coordinator. The external-hit null padding below the attended window stays. Step-5-Preview, TP4 + EP, MTP k=3, Mooncake Store, same harness for both: Store vs GPU-cache replay is 64/64 bit-identical, stress-phase local hits are identical (332,928 / 610,640 tokens), GSM8K 0.935 vs 0.933. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFMbNwZXyJArMwo7GA7wxm Signed-off-by: Jared Wen <jaredwen@inferact.ai>

Resolves the overlap with vllm-project#53558 (pluggable KV cache config builder) and vllm-project#61455 (explicit draft-layer EAGLE tags): the groups now come from the builder, main tags is_eagle_group from the draft layers first, and this branch's _retain_eagle_hit_blocks_below_window runs on the tagged groups. The planning test tags the draft layer with is_draft instead of relying on the removed MTP trailing-layer heuristic. Signed-off-by: Stu Cao <stu@primeintellect.ai>

…ipped Since vllm-project#61455 the planning test checks that draft-layer tags do not change the grouping. The tags now also decide which sliding-window groups retain the EAGLE hit blocks below the window, so strip extra_retained_tokens before comparing the tagged and untagged groupings. Signed-off-by: Stu Cao <stu@primeintellect.ai>

Step5 support added a second SWA retention scheme for EAGLE/MTP prefix hits: a larger retained window (window + proof block + scheduler-block drift), matching admission caps and pool sizing, and a coordinator set of proof groups. Main already covers the same hits with eagle_proof_margin, the multi-module MTP extra_retained_tokens and replay boundaries, and vllm-project#61455 now flags every drafter group, so restore main's SWA retention, sizing and coordinator. The external-hit null padding below the attended window stays. Step-5-Preview, TP4 + EP, MTP k=3, Mooncake Store, same harness for both: Store vs GPU-cache replay is 64/64 bit-identical, stress-phase local hits are identical (332,928 / 610,640 tokens), GSM8K 0.935 vs 0.933. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jared Wen <jaredwen@inferact.ai>

Step5 support added a second SWA retention scheme for EAGLE/MTP prefix hits: a larger retained window (window + proof block + scheduler-block drift), matching admission caps and pool sizing, and a coordinator set of proof groups. Main already covers the same hits with eagle_proof_margin, the multi-module MTP extra_retained_tokens and replay boundaries, and vllm-project#61455 now flags every drafter group, so restore main's SWA retention, sizing and coordinator. The external-hit null padding below the attended window stays. Step-5-Preview, TP4 + EP, MTP k=3, Mooncake Store, same harness for both: Store vs GPU-cache replay is 64/64 bit-identical, stress-phase local hits are identical (332,928 / 610,640 tokens), GSM8K 0.935 vs 0.933. Signed-off-by: Jared Wen <jaredwen@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…t-layer tags (vllm-project#61455) Signed-off-by: Dakai An <dakaian108@gmail.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

### What this PR does / why we need it? Port `vllm-ascend` to vLLM main `c0b6f74` (bumped from `a98247a`) while keeping the `v0.31.0` release lane green. The upstream window moved the KV-cache planning helpers into a new module, made EAGLE draft groups explicit, typed DeepSeek-V4 configs through upstream Transformers, removed Medusa/MLP speculator support, and changed several spec-decode/KV-cache/EPLB signatures. Every gate below is either a `vllm_version_is("0.31.0")` split or an attribute/argument probe where two main dev builds share one version string. | Files | Upstream vLLM change (why it breaks vllm-ascend) | vllm-ascend adaptation | |-------|--------------------------------------------------|------------------------| | `.github/vllm-main-verified.commit` | — (main-lane pin only) | Verified main commit bumped `a98247a` -> `c0b6f74` | | `tests/e2e/pull_request/four_card/test_minimax_m3.py` | The main-lane pin re-JITs the static kernels, so the first generate on a cold cache shifts the near-uniform dummy logprobs (~3e-3) enough to flip a greedy token | On main run a throwaway `generate_greedy_logprobs` before the baseline/replay pair so both measure the warmed steady state and share the cached prefix; v0.31.0 keeps the single-shot comparison | | `tests/ut/core/test_circular_buffer.py` | [vllm#61126](vllm-project/vllm#61126) (cascade-attention removal) moved the prefix-block query up to `KVCacheManager` and dropped it from single-type managers | On main assert only the manager's own `get_num_skipped_tokens(65536)`; keep the combined `get_num_common_prefix_blocks(...)` assert on v0.31.0 | | `tests/ut/core/test_kv_cache_interface.py` | [vllm#61183](vllm-project/vllm#61183) moved `is_kv_cache_spec_uniform` into `vllm.v1.core.kv_cache_planning` | Version-aware import (`kv_cache_utils` on v0.31.0 / `kv_cache_planning` on main) | | `tests/ut/distributed/ascend_store/_mock_deps.py` | [vllm#61183](vllm-project/vllm#61183) added the `vllm.v1.core.kv_cache_planning` module | Add it to the mocked-vLLM module list | | `tests/ut/distributed/ascend_store/test_request_lifecycle.py` | [vllm#58668](vllm-project/vllm#58668) trims `num_computed_tokens` to the tokens actually kept when a request stops mid token-list | On main subtract `(accepted + 1) - <newly appended output tokens>` from the expected computed count; keep the old expectation on v0.31.0 | | `tests/ut/distributed/kv_transfer/kv_p2p/test_mooncake_connector.py` | [vllm#61183](vllm-project/vllm#61183) moved `get_kv_cache_config_from_groups` / `is_kv_cache_spec_uniform` | Version-aware import | | `tests/ut/models/test_deepseek_v41_cache.py` | [vllm#61183](vllm-project/vllm#61183) relocated the planning helpers; [vllm#53558](vllm-project/vllm#53558) gave `_max_memory_usage_bytes_from_groups` a leading `builder` argument | Import from the live planning module and probe the upstream signature to call the 2-arg (v0.31.0) vs 3-arg (main) form, feeding an Ascend `_get_pool_bytes_per_block` builder | | `tests/ut/ops/test_gdn_attn_builder.py` | [vllm#58737](vllm-project/vllm#58737) derives `num_reqs` from the block table and asserts its row count equals `seq_lens`; Ascend's `unpadded()` keeps the graph-padded rows | New UT: a padded 4-row block table with 2 real requests must still yield a 2-row `cache_indices` | | `tests/ut/patch/platform/test_packed_kv_cache_compat.py` | [vllm#61183](vllm-project/vllm#61183) relocated the patched module; [vllm#61455](vllm-project/vllm#61455) removed `_annotate_eagle_groups` from main | Patch `cache_patch._kv_cache_planning`; new UT asserts the packed hook tolerates an upstream module without the EAGLE-annotation API | | `tests/ut/patch/platform/test_patch_eplb.py` | [vllm#57549](vllm-project/vllm#57549) emits the async EPLB cycle-committed log from the async worker wrapper on main | Expect the workspace-move log only on v0.31.0 | | `tests/ut/patch/platform/test_patch_kv_cache_coordinator.py` | [vllm#61183](vllm-project/vllm#61183) relocated the patch entry point | Assert against `kcu_patch._kv_cache_planning` instead of `vllm.v1.core.kv_cache_utils` | | `tests/ut/patch/platform/test_patch_pp_mtp.py` | [vllm#56322](vllm-project/vllm#56322) makes `ModelConfig.verify_with_parallel_config` read `parallel_config.enable_hc_sp` | Add `enable_hc_sp=False` to the constructed parallel config (ignored on v0.31.0) | | `tests/ut/patch/platform/test_prefix_cache_cp_patches.py` | [vllm#60043](vllm-project/vllm#60043) removed the `mamba_cache_mode="all"` leftover and requires `"align"` with prefix caching; [vllm#53558](vllm-project/vllm#53558) added the `builder` argument | Version-select the mamba cache mode; probe the upstream admission signature for the `builder` argument | | `tests/ut/spec_decode/test_mla_dspark.py` | [vllm#60692](vllm-project/vllm#60692) renamed the capture hook `precompute_context_kv(num_reqs)` -> `prepare_forward(num_reqs, attn_metadata)` | Version-split the expected capture args/kwargs | | `tests/ut/test_compressed_prefix_cache.py` | [vllm#61183](vllm-project/vllm#61183) moved `is_kv_cache_spec_uniform` | Version-aware import | | `tests/ut/worker/test_attn_utils_v2.py` | [vllm#61455](vllm-project/vllm#61455) annotates EAGLE groups from `spec.is_draft`; [vllm#60766](vllm-project/vllm#60766) resolves pin memory lazily via `is_pin_memory_available()` | New UT that draft-layer specs get tagged; on main monkeypatch `is_pin_memory_available` to `False` | | `tests/ut/worker/test_model_runner_v2.py` | [vllm#56716](vllm-project/vllm#56716) suppresses EPLB even for `skip_eplb` dummy runs; [vllm#60946](vllm-project/vllm#60946) renamed the PCP staging hook | Add a `runner.eplb` stub; call the version-appropriate staging method | | `tests/ut/worker/test_mtp_pcp_speculator_v2.py` | [vllm#60885](vllm-project/vllm#60885) rejects a mismatched explicit float16/bfloat16 KV-cache dtype | Build the model config with `bfloat16` instead of `float32` | | `tests/ut/worker/test_pcp_manager_v2.py` | [vllm#60946](vllm-project/vllm#60946) renamed `prepare_inputs_to_capture` -> `prepare_inputs_for_capture_or_profile` | Patch and call the live method name per lane | | `tests/ut/worker/test_v2_dummy_padding.py` | [vllm#61767](vllm-project/vllm#61767) collects DP votes through `get_dp_group().cpu_all_reduce_()` instead of `dist.all_reduce(..., group=cpu_group)` | Version-split the mocked collective | | `tests/ut/worker/v2/test_eplb_controller.py` | [vllm#57549](vllm-project/vllm#57549) gives `add_model` an optional `model_name` and names the draft entry | Expect the 3-arg call (`model_name="draft (draft)"`) on main, 2-arg on v0.31.0 | | `vllm_ascend/_310p/worker_310p.py` | [vllm#61183](vllm-project/vllm#61183) moved `get_kv_cache_groups` | Version-aware import | | `vllm_ascend/distributed/ec_transfer/ec_memcache_connector.py` | [vllm#57904](vllm-project/vllm#57904) dropped `_get_encoder_cache_hidden_dim` from `cpu.common` (encoder widths are measured at startup) | On main size the receive buffer as `out_hidden_size * (1 + len(deepstack_visual_indexes))` (Qwen3-VL DeepStack) else `get_inputs_embeds_size()`; keep the helper on v0.31.0 | | `vllm_ascend/models/deepseek_v4/compressor.py` | [vllm#61392](vllm-project/vllm#61392) switched `DeepseekV4Config` to the upstream Transformers class | Import from `transformers` on main, from `vllm.transformers_utils.configs.deepseek_v4` on v0.31.0 | | `vllm_ascend/models/deepseek_v4/indexer.py` | Same [vllm#61392](vllm-project/vllm#61392) | Same version-aware import | | `vllm_ascend/models/deepseek_v4/mm_preprocess.py` | Same [vllm#61392](vllm-project/vllm#61392) | Same version-aware import | | `vllm_ascend/models/deepseek_v4/model.py` | [vllm#61392](vllm-project/vllm#61392) moves the config to Transformers, folds `compress_ratios`/`num_hash_layers` into `layer_types`/`mlp_layer_types`, and keys YaRN under the compressor (`compress`) group | Restore the legacy fields on main from the new lists, read the `compress`-scoped rope params with safe defaults, and keep the legacy path on v0.31.0 | | `vllm_ascend/models/deepseek_v41/cache_config.py` | [vllm#61183](vllm-project/vllm#61183) moved `may_override_num_blocks` | Version-aware import | | `vllm_ascend/models/glm5next/cache_config.py` | [vllm#61183](vllm-project/vllm#61183) moved `create_kv_cache_group_specs` / `may_override_num_blocks` | Version-aware import | | `vllm_ascend/ops/gdn_attn_builder.py` | [vllm#58737](vllm-project/vllm#58737) derives `num_reqs` from the block table and asserts row parity with `seq_lens` | On main trim the padded block table to `seq_lens.shape[0]` before `mamba_get_block_table_tensor` when the sizes differ | | `vllm_ascend/patch/__init__.py` | [vllm#61183](vllm-project/vllm#61183) moved `get_kv_cache_config_from_groups` to `kv_cache_planning` | Update the patch-registry doc | | `vllm_ascend/patch/platform/__init__.py` | [vllm#61183](vllm-project/vllm#61183) moved the KV-cache planning/patcher module | Update the patch-registry doc | | `vllm_ascend/patch/platform/patch_eplb.py` | [vllm#57549](vllm-project/vllm#57549) moves the async EPLB cycle-committed log onto the async worker path on main | Report completion from the wrapped async worker (rank 0) on main; keep the workspace-move log on v0.31.0 | | `vllm_ascend/patch/platform/patch_kv_cache_utils.py` | [vllm#61183](vllm-project/vllm#61183) moved the planning helpers; [vllm#53558](vllm-project/vllm#53558) added the `builder` argument; [vllm#61455](vllm-project/vllm#61455) dropped `_annotate_eagle_groups` | Resolve the live planning module by version, patch its symbols, gate the EAGLE annotation on attribute presence, and dispatch the admission helper by argument count | | `vllm_ascend/quantization/methods/kv_cache/turboquant/cache.py` | [vllm#61183](vllm-project/vllm#61183) moved `may_override_num_blocks` | Version-aware import | | `vllm_ascend/spec_decode/__init__.py` | [vllm#61701](vllm-project/vllm#61701) removed Medusa/MLP speculator support | Import `AscendMedusaProposer` only on v0.31.0; raise a clear error for `method="medusa"` on main | | `vllm_ascend/spec_decode/medusa_proposer.py` | [vllm#61701](vllm-project/vllm#61701) removed `vllm.v1.spec_decode.medusa` | Add `# type: ignore[import-not-found]`; the module only loads on the lane that still ships it | | `vllm_ascend/worker/v1/model_runner.py` | Same [vllm#61701](vllm-project/vllm#61701) | Gate the Medusa import and the `isinstance(self.drafter, AscendMedusaProposer)` branch on v0.31.0 | | `vllm_ascend/worker/v2/attn_utils.py` | [vllm#61455](vllm-project/vllm#61455) annotates EAGLE groups from `spec.is_draft` | On main tag the spec with `is_draft` from `attn_module.is_draft_layer` | | `vllm_ascend/worker/v2/pcp_manager.py` | [vllm#60946](vllm-project/vllm#60946) added the `moe_gathers_tokens` argument and renamed `prepare_inputs_to_capture` -> `prepare_inputs_for_capture_or_profile` | Forward the new argument; version-split the staging hook | | `vllm_ascend/worker/v2/sample/apply_top_k_top_p.py` | [vllm#61996](vllm-project/vllm#61996) added `has_topp_only_rows` to the split top-p pipeline | Accept and ignore the extra argument on main (the PyTorch fallback handles top-p-only rows) | | `vllm_ascend/worker/v2/spec_decode/dflash/aclgraph.py` | [vllm#60692](vllm-project/vllm#60692) renamed `precompute_context_kv` -> `prepare_forward` | Pass the capture hook the installed vLLM expects | | `vllm_ascend/worker/v2/spec_decode/lmhead_tp_utils.py` | [vllm#59554](vllm-project/vllm#59554) added `num_steps` to `Speculator.sample_draft` | Version-split the override signature and forward `num_steps` on main | | `vllm_ascend/worker/v2/spec_decode/rejection_sampler_utils.py` | [vllm#61451](vllm-project/vllm#61451) renamed the logits-stats kernel argument `num_speculative_steps` -> `num_logits` and moved bonus-token detection into `_is_bonus_logit` | Pass `num_logits` on main; keep `num_speculative_steps` on v0.31.0 | | `vllm_ascend/worker/worker.py` | [vllm#61183](vllm-project/vllm#61183) moved `get_kv_cache_groups`/`unify_hybrid_kv_cache_specs`; [vllm#57516](vllm-project/vllm#57516) reclaims cached memory via `torch.accelerator.empty_cache()` inside `load_model` | Version-aware imports; on main neutralize `torch.npu.empty_cache` while loading weights under the sleep-mode pluggable memory pool | ### Does this PR introduce _any_ user-facing change? No. This is an internal main2main compatibility update; on the `v0.31.0` release lane behavior is unchanged, and on main the only behavioral note is that Medusa speculative decoding now raises a clear error (upstream removed it in vllm#61701). ### How was this patch tested? - Unit tests: the changed UT files above (KV-cache planning/coordinator, EPLB, PCP/spec-decode, DeepSeek-V4/V4.1 cache, mamba align, `worker/worker.py`). - E2E: `tests/e2e/pull_request/four_card/test_minimax_m3.py`. - Run the main lane against the pinned `c0b6f74` and the `v0.31.0` lane via `vllm_version_is`. --- ### Checklist #### Testing - [ ] **Boundary / edge-case coverage**: I reviewed whether this change involves boundary conditions (e.g. empty input, single/zero element, min/max values, off-by-one, sequence-length limits, integer overflow). Where applicable, I added unit tests covering these edge cases in `tests/ut/` or `tests/e2e/`. - [ ] **Regression test**: If this PR fixes a bug, I added a regression test that would fail without the fix. - [ ] **Error / failure path coverage**: I covered abnormal paths (exceptions, timeouts, out-of-range, division by zero) in addition to the happy path. - [ ] **Meaningful & reliable tests**: New tests assert real behavior and are not written merely to satisfy CI. - [ ] **Performance impact**: If this PR touches a performance-sensitive path, I attached benchmark results and added performance regression tests. - vLLM main: vllm-project/vllm@a98247a Signed-off-by: zhangxinyuehfad <starmoon_zhang@163.com>

Upstream vLLM vllm-project#61455 tags draft groups on the global KV cache groups and keeps is_eagle_group when it projects them onto a worker that owns none of a group's layers, so every worker's config already carries the flag merge_eagle_group_flags restored. Remove it, its call, its test and its seam pin. The DeepSeek-V4.1 encoder fix for tool results (upstream vLLM vllm-project#62199) is now upstream's own text: drop the stale modification notice from that file.

andakai added 2 commits October 14, 2026 00:11

Merge remote-tracking branch 'origin/main' into refactor/drafter-kv-s…

690912d

…pec-from-runner Signed-off-by: Dakai An <dakaian108@gmail.com>

mergify Bot added speculative-decoding mrv2 Model Runner V2 specific kv-connector kv-cache-manager labels Oct 13, 2026

wei-yubo mentioned this pull request Oct 14, 2026

[BugFix][KV Cache] Adapt positional MTP draft-group annotation for Ascend hybrid layouts vllm-project/vllm-ascend#17656

Closed

ZJY0516 reviewed Oct 14, 2026

View reviewed changes

Comment thread vllm/v1/spec_decode/llm_base_proposer.py

ZJY0516 mentioned this pull request Oct 14, 2026

[Bugfix] Annotate DFlash draft KV cache groups via the trailing-layer rule #61545

Closed

6 tasks done

untrall mentioned this pull request Oct 14, 2026

[Test][Spec Decode] Regression test for #61443 (GLM-5.3-Flash DFlash shape) on draft-layer tags andakai/vllm#8

Closed

6 tasks done

andakai marked this pull request as ready for review October 16, 2026 17:10

andakai requested review from ApostaC, MatthewBonanni, NickLucche, WoosukKwon, alexm-redhat, benchislett, heheda12345, ivanium, njhill, orozery, robertgshaw2-redhat, wzhao18, xuechendi, yewentao256 and ywang96 as code owners October 16, 2026 17:10

claude Bot reviewed Oct 16, 2026

View reviewed changes

Merge branch 'main' into refactor/drafter-kv-spec-from-runner

bae418b

JaredforReal approved these changes Oct 16, 2026

View reviewed changes

ivanium approved these changes Oct 17, 2026

View reviewed changes

ivanium added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 17, 2026

ivanium enabled auto-merge (squash) October 17, 2026 07:08

Merge branch 'main' into refactor/drafter-kv-spec-from-runner

693983d

ivanium merged commit c3de0b3 into vllm-project:main Oct 17, 2026
202 checks passed

JaredforReal mentioned this pull request Oct 17, 2026

[Step5] Keep the CSA indexer state in KV cache layers and drop KV side storage jeejeelee/vllm#16

Merged

6 tasks done

untrall mentioned this pull request Oct 17, 2026

[Test][Spec Decode] Regression test for #61443 (GLM-5.3-Flash DFlash shape) on draft-layer tags #62180

Open

JaredforReal mentioned this pull request Oct 17, 2026

[Core] Drop the Step5 SWA EAGLE-proof retention jeejeelee/vllm#19

Merged

6 tasks done

JaredforReal mentioned this pull request Oct 17, 2026

[Bugfix][Core] Keep the EAGLE-reachable SWA tail at a shared-prefix junction #62206

Merged

6 tasks done

ivanium mentioned this pull request Oct 18, 2026

[Bugfix][Core] Treat a shared-prefix junction as a set of positions under EAGLE #62278

Draft

6 tasks done

ogidog mentioned this pull request Oct 19, 2026

[Bug][Spec Decode] DFlash2 (SWA) draft group is never annotated is_eagle_group under the generic packed KV layout (GLM-5.3-Flash shape) #61443

Closed

This was referenced Oct 20, 2026

[GLM5][Spec Decode] Group sliding-window draft layers separately in the GLM-5.3 KV builder (DFlash2 on A100) wtdcode/vllm-backport#120

Open

[Feature]: SM8x (Ampere A100/A800) support for DeepSeek-V4-Flash / DeepSeek-V4-Flash-0731 DSpark #50576

Open

zhangxinyuehfad mentioned this pull request Oct 21, 2026

[Misc] vllm main2main 1017 vllm-project/vllm-ascend#19302

Merged

5 tasks

ogidog mentioned this pull request Oct 23, 2026

qwen4_exp: DFlash/DFlash2 speculative decoding for Qwen3.8-Flash-Next wtdcode/vllm-backport#128

Open

andakai mentioned this pull request Oct 27, 2026

[Refactor][KV Cache] Remove EAGLE block dropping and default to lookahead hashes andakai/vllm#14

Draft

5 of 6 tasks

Navigation Menu

Repository navigation

[Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags - #61455

[Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags#61455
ivanium merged 6 commits into
vllm-project:mainfrom
andakai:refactor/drafter-kv-spec-from-runner

andakai commented Oct 13, 2026 •

edited by ivanium

Loading

Uh oh!

Uh oh!

claude Bot left a comment

Uh oh!

JaredforReal left a comment

Uh oh!

mergify Bot commented Oct 16, 2026

Uh oh!

ivanium left a comment

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

ivanium commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

andakai commented Oct 13, 2026 • edited by ivanium Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

Details

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

JaredforReal left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Oct 16, 2026

Uh oh!

ivanium left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

ivanium commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andakai commented Oct 13, 2026 •

edited by ivanium

Loading