[Bugfix][Spec Decode] Fix EAGLE and dense draft startup with EP - #56930
Merged
WoosukKwon merged 3 commits intoSep 17, 2026
Merged
Conversation
kyleliang-nv
force-pushed
the
kylliang/fix-dense-spec-draft-ep
branch
from
September 16, 2026 15:50
de41bed to
e679fb5
Compare
kyleliang-nv
marked this pull request as ready for review
September 16, 2026 15:50
kyleliang-nv
requested review from
ProExpertProg,
WoosukKwon,
houseroad,
mgoin,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256 and
youkaichao
as code owners
September 16, 2026 15:50
|
✅ @kyleliang-nv, CI is now available for this PR.
|
Collaborator
|
LGTM thanks for the fix! |
Collaborator
|
/ci run |
|
❌ This PR is 6 commits behind upstream |
kyleliang-nv
force-pushed
the
kylliang/fix-dense-spec-draft-ep
branch
from
September 16, 2026 18:09
e679fb5 to
9be3744
Compare
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Kyle Liang <kyleliang-nv@users.noreply.github.com>
Replace mocked validation with local model configs covering EAGLE, EAGLE3, independent drafts, and DeepSeek V4 MTP/DSpark. Use matching target and draft TP sizes and document the final-config EP requirement. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Kyle Liang <kyleliang-nv@users.noreply.github.com>
Accept an optional final draft ModelConfig and preserve EP inheritance when model information is omitted. Keep real configuration regressions and add omitted/None compatibility coverage. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Kyle Liang <kyleliang-nv@users.noreply.github.com>
kyleliang-nv
force-pushed
the
kylliang/fix-dense-spec-draft-ep
branch
from
September 16, 2026 18:10
9be3744 to
a2e2e90
Compare
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #89395 for commit |
WoosukKwon
approved these changes
Sep 16, 2026
WoosukKwon
enabled auto-merge (squash)
September 16, 2026 18:35
Collaborator
|
/ci retry |
|
✅ Queued 2 failed job(s) for retry in Buildkite CI #89395. |
WoosukKwon
disabled auto-merge
September 17, 2026 00:03
This was referenced Sep 17, 2026
6 tasks
maeehart
added a commit
to maeehart/vllm
that referenced
this pull request
Sep 17, 2026
Take vllm-project#56930's draft_model_config expert-parallel handling in place of draft_is_moe. Rename is_fused_moe_situv2_a8w4_enabled to is_fused_moe_situv2_enabled after vllm-project#53940. Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Sep 19, 2026
Tflowers-0129
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Sep 23, 2026
### What this PR does / why we need it?
#### Change Summary
The PR advances the main2main lane to vLLM v0.30.0 (commit
`ced6857afa0ea7b2e3f0846a62e1394e90f15607`), adapting vllm-ascend to
every upstream change in the `84030bbe` -> `4991f97` -> `ced6857` range.
Because both CI lanes now install vLLM v0.30.0, all
`vllm_version_is("0.29.0")` forks are permanently false and are
collapsed to the main behavior.
| Files | Upstream vLLM change | vllm-ascend adaptation |
|-------|---------------------|------------------------|
| `.github/vllm-main-verified.commit` | — | Updated verified main commit
hash `84030bbe` -> `4991f97` -> `ced6857` (v0.30.0 tag commit) |
| `.github/vllm-release-tag.commit` | — | Bumped the release boundary to
`v0.30.0` |
| `.github/workflows/pr_test.yaml` | — | Added a dual-version cpu-ut
matrix (`vllm_versions`) for the main2main lane; temporarily commented
out pre-commit/mypy and set `cpu-ut` to `if: false`; dropped the
`needs.cpu-ut` requirement from the ready gate; added a
main2main-specific cpu-ut failure hint |
| `Dockerfile` | — | `VLLM_TAG` default -> `v0.30.0` |
| `Dockerfile.310p` | — | `VLLM_TAG` default -> `v0.30.0` |
| `Dockerfile.310p.openEuler` | — | `VLLM_TAG` default -> `v0.30.0` |
| `Dockerfile.a3` | — | `VLLM_TAG` default -> `v0.30.0` |
| `Dockerfile.a3.openEuler` | — | `VLLM_TAG` default -> `v0.30.0` |
| `Dockerfile.a5` | — | `VLLM_TAG` default -> `v0.30.0` |
| `Dockerfile.a5.openEuler` | — | `VLLM_TAG` default -> `v0.30.0` |
| `Dockerfile.openEuler` | — | `VLLM_TAG` default -> `v0.30.0` |
| `README.md` | — | CI notes updated to v0.30.0 |
| `README.zh.md` | — | CI notes updated to v0.30.0 |
|
`tests/e2e/nightly/single_node/ops/singlecard_ops/triton/test_num_nans.py`
| — | `VLLM_VERSION` default 0.29.0 -> 0.30.0 |
| `tests/ut/_310p/test_model_runner_v2_310p.py` | v0.30.0 boundary |
Dropped the `vllm_version_is` gate test; `_needs_kv_cache_zeroing_310p`
always uses `spec_config.use_eagle_block_drop()` |
| `tests/ut/core/test_dyntra_lb_scheduler.py` | — | Removed the 0.29
`KVConnectorBlockState.block_ids` assertions; assert the `req_ids` form
only |
| `tests/ut/core/test_scheduler_connector_block_state.py` | — | Removed
the 0.29 `block_ids` snapshot branch |
| `tests/ut/kv_offload/test_native_cpu_offload.py` | — | Block -> chunk
API: assert `spec.num_chunks` |
| `tests/ut/kv_offload/test_npu_offload_spec.py` | — |
`num_blocks`/`kv_bytes_per_block` -> `num_chunks`/`kv_bytes_per_chunk` |
| `tests/ut/models/test_deepseek_v41_registration.py` |
[vllm#56741](vllm-project/vllm#56741) | Removed
the V4.1 0.29 skip; registration is always active |
| `tests/ut/patch/platform/test_patch_mamba_block_aligned_split.py` | —
| Removed the 0.29 gate for `mamba_fine_grained_prefix_cache` |
| `tests/ut/patch/platform/test_patch_speculative_config_dspark.py` |
[vllm#55914](vllm-project/vllm#55914) /
[vllm#56930](vllm-project/vllm#56930) | Removed
the DeepSeek V4.1 0.29 `pytest.skip` |
| `tests/ut/patch/platform/test_patch_use_v2_model_runner.py` |
[vllm#54523](vllm-project/vllm#54523) | Deleted
the 0.28.0 PCP+DP validator workaround tests; reworked the
version-routing fixtures |
| `tests/ut/patch/worker/test_patch_dspark_pp.py` | — | Parametrize
`legacy: bool` instead of version strings |
| `tests/ut/quantization/configs/test_modelslim_config.py` |
[vllm#56741](vllm-project/vllm#56741) | Removed
the V4.1 0.29 skip |
| `tests/ut/spec_decode/test_dspark_proposer.py` |
[vllm#56741](vllm-project/vllm#56741) | Removed
the V4.1 0.29 skip marker |
| `tests/ut/spec_decode/test_eagle_proposer.py` | — | `uses_xdrope_dim`
-> `mrope_num_dims` |
| `tests/ut/test_compressed_prefix_cache.py` | — | `replay_boundaries`
now unconditional |
| `tests/ut/worker/test_encoder_acl_graph.py` | — | `axis_keys=()` now
unconditional |
| `tests/ut/worker/test_model_runner_v1.py` |
[vllm#56741](vllm-project/vllm#56741) | Removed
the V4.1 0.29 skips and the `vllm_version_is` patches |
| `tests/ut/worker/test_model_runner_v2.py` | v0.30.0 boundary |
Version-routing fixture rework |
| `tests/ut/worker/test_pcp_manager_v2.py` |
[vllm#56107](vllm-project/vllm#56107) /
[vllm#53867](vllm-project/vllm#53867) | Dropped
the 0.29 `req_states` coverage / added `padded_num_reqs` coverage;
removed the 0.28/0.29 branches |
| `tests/ut/worker/v2/test_pp_utils.py` | — | Replaced the
version-routing matrix with `use_legacy_spec_pp() is False` |
| `vllm_ascend/_310p/model_runner_310p.py` | — | Removed the 0.29
xdrope-position branch |
| `vllm_ascend/_310p/spec_decode/llm_base_proposer_310.py` | — | Removed
the 0.29 xdrope `target_positions[0]` squeeze |
| `vllm_ascend/_310p/worker/v2/model_runner.py` |
[vllm#57270](vllm-project/vllm#57270) | Removed
the 0.29 `max_seq_len_np` kwarg and the 0.29 eagle-block-drop branch |
| `vllm_ascend/_310p/worker/v2/rope.py` | — | mrope `num_dims =
model_config.mrope_num_dims` (dropped the 0.29 constant 3) |
| `vllm_ascend/_310p/worker/v2/states.py` |
[vllm#56908](vllm-project/vllm#56908) |
`UvaBuffer.uva` property -> method; final v0.30.0 form |
| `vllm_ascend/attention/mla_v1.py` |
[vllm#56181](vllm-project/vllm#56181) | Draft
TND_NTD layout forced via `_EXTRA_CTX.is_draft_model`; dropped the 0.29
gate |
| `vllm_ascend/attention/utils.py` |
[vllm#55353](vllm-project/vllm#55353) /
[vllm#56157](vllm-project/vllm#56157) |
Ascend-owned
`_seq_lens_cpu`/`_num_computed_tokens_cpu`/`dcp_local_seq_lens_cpu` are
now unconditional |
| `vllm_ascend/batch_invariant.py` | — | `reduce_sum` accepts
NumPy-style `axis` + `dtype`, rejects `dim`+`axis` together, forwards
`dtype` to the native fallback |
| `vllm_ascend/core/dyntra_lb_scheduler.py` | — |
`KVConnectorBlockState` always uses `req_ids`+`resolve_block_ids`
(dropped the 0.29 `block_ids` snapshot) |
| `vllm_ascend/core/kv_cache_interface.py` |
[vllm#53906](vllm-project/vllm#53906) | MLA
`get_storage_block_size` override unconditional; dropped the 0.29
`storage_block_size` property |
| `vllm_ascend/core/recompute_scheduler.py` | — | Dropped the xdrope
kwarg and the 0.29 `block_ids` snapshot |
| `vllm_ascend/core/scheduler_profiling_chunk.py` | — | Dropped the 0.29
`block_ids` snapshot |
| `vllm_ascend/distributed/kv_transfer/kv_pool/kv_offload/native/npu.py`
| — | Block -> chunk API (`num_chunks`, `kv_bytes_per_chunk`)
unconditional |
| `vllm_ascend/lora/punica_npu.py` |
[vllm#53555](vllm-project/vllm#53555) + boundary
| `add_lora_logits` per-adapter matmul fallback for heads smaller than
the rank; final state drops the `apply_lora_full_linear` binding (both
supported targets predate #53555) |
| `vllm_ascend/models/__init__.py` |
[vllm#56741](vllm-project/vllm#56741) | V4.1
registration (`DeepseekV41ForCausalLM`/`DSparkModel`) unconditional |
| `vllm_ascend/models/deepseek_v41/engram/embedding.py` |
[vllm#56741](vllm-project/vllm#56741) |
`deepseek_v4_1` -> `deepseek_v41` import unconditional |
| `vllm_ascend/models/deepseek_v41/engram/hash_state.py` |
[vllm#56741](vllm-project/vllm#56741) |
`deepseek_v4_1` -> `deepseek_v41` import unconditional |
| `vllm_ascend/models/deepseek_v41/engram/parallel.py` |
[vllm#56741](vllm-project/vllm#56741) |
`deepseek_v4_1` -> `deepseek_v41` import unconditional |
| `vllm_ascend/models/deepseek_v41/model.py` |
[vllm#56741](vllm-project/vllm#56741) |
`deepseek_v4_1` -> `deepseek_v41` import unconditional |
| `vllm_ascend/models/deepseek_v41/vl_model.py` |
[vllm#56741](vllm-project/vllm#56741) /
[vllm#56554](vllm-project/vllm#56554) |
`deepseek_v41` imports; drop `IMAGE_PAD_ID`/alignment-pad handling on
main |
| `vllm_ascend/ops/mla.py` |
[vllm#56157](vllm-project/vllm#56157) |
`MLAAttention.supports_pcp_dcp = True` set on the class, unconditional |
| `vllm_ascend/ops/rotary_embedding.py` |
[vllm#56446](vllm-project/vllm#56446) | YaRN
mscale signature adaptation; v0.29 branch collapsed |
| `vllm_ascend/patch/__init__.py` |
[vllm#55914](vllm-project/vllm#55914) /
[vllm#56930](vllm-project/vllm#56930) | Registry
entry for the new draft-EP patch; dropped version gates |
| `vllm_ascend/patch/platform/__init__.py` |
[vllm#56741](vllm-project/vllm#56741 Engram
| `patch_engram_config` imported unconditionally |
| `vllm_ascend/patch/platform/patch_balance_schedule.py` | — | Dropped
the 0.29 `block_ids` snapshot |
| `vllm_ascend/patch/platform/patch_kv_cache_coordinator.py` |
[vllm#54736](vllm-project/vllm#54736) + boundary
| Accept/forward `allow_partial_hash_hits`; collapsed the 0.29 gate |
| `vllm_ascend/patch/platform/patch_parallel_config.py` |
[vllm#54523](vllm-project/vllm#54523) | Removed
the 0.29 `_validate_parallel_config` PCP+DP workaround; keeps
`use_sequence_parallel_moe` |
| `vllm_ascend/patch/platform/patch_speculative_config.py` |
[vllm#55914](vllm-project/vllm#55914) without
[vllm#56930](vllm-project/vllm#56930) | Skip
`_verify_with_expert_parallelism` for non-MoE draft
(`runner_type=="draft"`) |
| `vllm_ascend/patch/platform/patch_use_v2_model_runner.py` |
[vllm#54523](vllm-project/vllm#54523) | Removed
the 0.28.0 PCP+DP validation workaround |
| `vllm_ascend/patch/worker/patch_bind_kv_cache.py` |
[vllm#53781](vllm-project/vllm#53781) | Ascend
`bind_kv_cache_to_layers` (assign the raw allocation); collapsed the
0.29 gate |
| `vllm_ascend/patch/worker/patch_deepseek_v2.py` |
[vllm#53781](vllm-project/vllm#53781) |
Accept/ignore `index_group_builder`; `SparseMLAIndexGroupBuilder` import
collapse |
| `vllm_ascend/patch/worker/patch_mamba_utils.py` |
[vllm#56898](vllm-project/vllm#56898) |
`GPUInputBatch` import source version-gated, then collapsed to
`gpu_input_batch.InputBatch` |
| `vllm_ascend/patch/worker/patch_v2/patch_attn_utils.py` |
[vllm#53781](vllm-project/vllm#53781) | Register
Ascend `bind_kv_cache_to_layers`; expose tuple element 0 for the device
filter; collapsed the 0.29 gate |
| `vllm_ascend/patch/worker/patch_v2/patch_dspark.py` | — | Legacy
Spec+PP bypass comment (inactive on v0.30.0) |
| `vllm_ascend/patch/worker/patch_v2/patch_spec_pp.py` |
[vllm#56888](vllm-project/vllm#56888) | Alias
`async_tensor_h2d as async_copy_to_gpu`; collapsed the 0.29 gate |
| `vllm_ascend/patch/worker/patch_v2/patch_uva.py` |
[vllm#56908](vllm-project/vllm#56908) | `uva`
property vs method; final v0.30.0 method form |
| `vllm_ascend/spec_decode/llm_base_proposer.py` |
[vllm#56254](vllm-project/vllm#56254) +
[vllm#55914](vllm-project/vllm#55914) /
[vllm#56930](vllm-project/vllm#56930) + boundary
| Gate the V4.1 DSpark import by `HAS_TRITON`; static
`_draft_embed_accepts_mm` check instead of the runtime `embed_input_ids`
probe; collapsed version gates |
| `vllm_ascend/utils.py` | — | `vllm_version_is` docstring 0.29 -> 0.30;
Kimi MLA custom-op registration unconditional |
| `vllm_ascend/worker/model_runner_v1.py` |
[vllm#56741](vllm-project/vllm#56741) | V4.1 dsa
metadata/cache imports unconditional; removed xdrope position handling |
| `vllm_ascend/worker/v2/aclgraph_utils.py` |
[vllm#51700](vllm-project/vllm#51700) |
`ModelAclGraphManager.__init__` accepts/forwards `ubatch_runner`;
`UBatchRunner` import collapse |
| `vllm_ascend/worker/v2/model_runner.py` |
[vllm#56888](vllm-project/vllm#56888) /
[vllm#53867](vllm-project/vllm#53867) /
[vllm#51700](vllm-project/vllm#51700) /
[vllm#57270](vllm-project/vllm#57270) +
[vllm#56181](vllm-project/vllm#56181) + boundary
| Async-copy alias; pass `BatchExecutionDescriptor` to
`maybe_partition_pcp_batch`; `ubatch_runner`; `make_dummy(is_padding)`;
keep the replicated PCP draft on the global batch; collapsed version
gates. Also restores the `_check_oproj_tp_graph_step` guard and the PD
decode-recompute `gather_batch_req_state` override accidentally removed
by the v0.28.0-boundary cleanup (`e30adae80`) |
| `vllm_ascend/worker/v2/pcp_manager.py` |
[vllm#56888](vllm-project/vllm#56888) /
[vllm#56107](vllm-project/vllm#56107) /
[vllm#53867](vllm-project/vllm#53867) +
[vllm#56181](vllm-project/vllm#56181) + boundary
| Async alias; drop `req_states`; forward `padded_num_reqs`;
`prepare_draft_prefill` no-op / `restore_for_sampling` skip; collapsed
version gates |
| `vllm_ascend/worker/v2/pp_utils.py` | — | `use_legacy_spec_pp()`
returns `False` |
| `vllm_ascend/worker/v2/spec_decode/autoregressive/speculator.py` |
[vllm#56181](vllm-project/vllm#56181) | Route
through `_build_uniform_attn_metadata`/`_build_attn_metadata`; re-hook
the Ascend rotary-positions injection onto the new methods; dropped the
0.29 gate |
| `vllm_ascend/worker/v2/spec_decode/dflash/speculator.py` |
[vllm#56181](vllm-project/vllm#56181) | Same
`BatchExecutionDescriptor` routing; dropped the 0.29 gate |
| `vllm_ascend/worker/v2/spec_decode/dspark/speculator.py` |
[vllm#56181](vllm-project/vllm#56181) | Same
routing; dropped the 0.29 gate |
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
- vLLM main:
vllm-project/vllm@84030bb
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Co-authored-by: zhao-stack <80399320+zhao-stack@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
After #55914, pairing an EP-enabled MoE target with a dense EAGLE, EAGLE3, or independent draft model fails during configuration validation:
Pass the final draft
ModelConfig, after draft architecture overrides, tocreate_draft_parallel_config. Disable inherited EP only when that config reports a dense draft throughModelConfig.is_moe, the same classification used by the existing validator. MoE drafts, including DeepSeek V4 MTP and DSpark, continue to inherit target EP. Target EP and target/draft TP settings remain unchanged.The new
draft_model_configargument is optional. Existing two-argument calls, or calls that explicitly passNone, retain the previous EP inheritance. The normal configuration-construction path always supplies the final draft config. This preserves helper-call compatibility; it does not infer draft density or fix existing validation failures for callers that omit model information.Test Plan
The regression constructs real
ModelConfigandSpeculativeConfigobjects from small local configuration files, exercising architecture conversion and production validation without mocks or model weights. Its 24 cases cover dense independent/EAGLE/EAGLE3 drafts and MoE independent/MTP/DSpark drafts, target EP on/off, and matching target/draft TP sizes of 1 and 2.Four additional compatibility cases cover omitted and explicit-
Nonedraft model configs, each with target EP on/off.pre-commit run --files vllm/config/speculative.py tests/test_config.py HF_HUB_OFFLINE=1 .venv/bin/python -m pytest tests/test_config.py -k 'draft_inherits_ep_only_for_moe or draft_parallel_config_preserves_ep_without_model' -vTest Result
Rebased without conflicts onto public
mainatfc8132a5e523294ea69decb5e7cbf50e9d3e135eon 2026-09-16. The three existing patches are unchanged by the rebase.a2e2e9035de0e81252cda7164198c4d2f5230a9d: 28 passed.TypeError.e679fb51c48b62b83920a24ea733bd4b3efb6cbe, repeated the negative control using an in-process wrapper that deliberately omits the draft model config when calling the real helper: the same 6 dense-draft/target-EP cases failed with expert-count validation errors, and 22 passed. The on-disk source was unchanged.Related work
This PR overlaps #56933, which addresses the same regression. Following review, this PR adopts the same backward-compatible optional-model-config interface, with an explicit legacy fallback and real configuration-construction regression coverage. These are alternative fixes, not dependencies; maintainers can select or consolidate the overlapping changes. #35143 addresses a separate dense-draft DP synchronization optimization.
AI assistance
AI assistance was used to investigate the regression, implement the change, and prepare tests. Test commands, results, and validation limits are documented above.