[Misc][Main2Main] Upgrade vLLM to 0429(DSV4/v0.20.0)#8856
[Misc][Main2Main] Upgrade vLLM to 0429(DSV4/v0.20.0)#8856gcanlin wants to merge 29 commits intovllm-project:mainfrom
Conversation
Signed-off-by: wxsIcey <1790571317@qq.com>
Signed-off-by: wxsIcey <1790571317@qq.com>
gcanlin
requested review from
LCAIZJ,
MengqingCao,
Yikun,
realliujiaxu,
wangxiyuan,
whx-sjtu and
zzzzwwjj
as code owners
github-actions
Bot
added
documentation
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
gcanlin
added
ready
gcanlin
changed the title
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request updates the vLLM integration to a newer version, necessitating significant refactoring of the Mixture-of-Experts (MoE) implementation. The changes streamline the MoE layer architecture by consolidating shared expert logic and updating internal APIs to match the upstream vLLM structure. Additionally, minor enhancements were made to the rejection sampler and worker compilation reporting to support new functionality and interface requirements. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
Suggested PR Title:\n\n\n[Ops][Feature] Refactor Ascend Fused MoE and update worker compilation times\n\n\nSuggested PR Summary:\n\nmarkdown\n### What this PR does / why we need it?\nThis PR refactors the Ascend Fused MoE implementation by merging AscendSharedFusedMoE into AscendFusedMoE and updating the runner to inherit from MoERunner. It introduces shared expert consistency validation and multi-stream overlap configurations. Additionally, it updates the rejection sampler with synthetic mode parameters and modifies the worker to return structured CompilationTimes. Feedback includes fixing a function call bug where enable_sp was used as a boolean, removing placeholder comments, and cleaning up unreachable code in the worker.\n\n### Does this PR introduce _any_ user-facing change?\nNo significant user-facing API changes, though internal MoE logic and worker return types for compilation are updated.\n\n### How was this patch tested?\nThe PR includes a consistency check for shared expert computation.\n\nFixes #\n
vLLM-Ascend CI Failure Analysis ReportOverview
The CI matrix exercises both vllm versions: the new commit Failed Jobs Summary
Other jobs (singlecard-full (0), multicard-2-full(0) on v0.19.1) passed. Issue AnalysisIssue 1:
|
| Item | Detail |
|---|---|
| Category | Code Bug |
| Error Type | AttributeError |
| Affected Tests | most singlecard-full(1) and many other failures cascade from this |
| Root Cause Commit | c08f3b2a62fd — "Measure encoder compile time seperate from llm backbone (#39240)" |
| Changed File | vllm/config/compilation.py (added encoder_compilation_time: float = field(default=0.0, init=False)) |
| Impact in vllm-ascend | vllm_ascend/worker/worker.py:567 |
Error Traceback:
File "vllm_ascend/worker/worker.py", line 567, in compile_or_warm_up_model
encoder=self.vllm_config.compilation_config.encoder_compilation_time,
AttributeError: 'CompilationConfig' object has no attribute 'encoder_compilation_time'
Explanation: Newer vLLM split encoder compile time tracking from the language-model backbone, adding encoder_compilation_time to CompilationConfig. The v0.19.1 release predates this and lacks the attribute. vllm-ascend already conditionally imports CompilationTimes based on vllm_version_is("0.19.1"), but the assignment site reads the attribute directly without the same guard.
Fix Suggestion: Read it via getattr(..., "encoder_compilation_time", 0.0) so the access is safe regardless of vllm version.
Issue 2: AscendRMSNormGated.__init__() got an unexpected keyword argument activation (13 occurrences)
| Item | Detail |
|---|---|
| Category | Code Bug |
| Error Type | TypeError |
| Affected Tests | Qwen3-Next, Mamba, GDN-style models |
| Root Cause Commit | 893611813 — "[Qwen][Bugfix] Fixes sigmoid activation in torch impl of RMSNormGated. (#40245)" |
| Changed File | vllm/model_executor/layers/layernorm.py:417 (added activation: str = "swish" kwarg) |
| Impact in vllm-ascend | vllm_ascend/ops/layernorm.py:160 AscendRMSNormGated.__init__ |
Error Traceback:
File "vllm/model_executor/layers/mamba/gdn_linear_attn.py", line 365, in __init__
self.norm = RMSNormGated(...)
TypeError: AscendRMSNormGated.__init__() got an unexpected keyword argument 'activation'
Explanation: Upstream RMSNormGated.__init__ accepts a new activation kwarg ("swish"/"silu"/"sigmoid"), and Qwen3-Next/GDN now pass it. The Ascend subclass overrides __init__ and doesn't accept the new kwarg, so it fails at construction.
Fix Suggestion: Accept activation in AscendRMSNormGated.__init__ and forward it to super().__init__().
Issue 3: EagleCudaGraphManager import broken — split into Prefill... / Decode... (9 occurrences)
| Item | Detail |
|---|---|
| Category | Code Bug |
| Error Type | ImportError |
| Affected Tests | EAGLE / spec-decode tests |
| Root Cause Commit | 4c7c69b4e — "[Model Runner V2] Skip attention metadata rebuild before draft prefill (#40410)" |
| Changed File | vllm/v1/worker/gpu/spec_decode/eagle/cudagraph.py (renamed EagleCudaGraphManager → EagleCudaGraphManagerBase + new PrefillEagleCudaGraphManager/DecodeEagleCudaGraphManager) |
| Impact in vllm-ascend | vllm_ascend/worker/v2/spec_decode/eagle/speculator.py:34, aclgraph.py:30 |
Error Traceback:
File "vllm_ascend/worker/v2/spec_decode/eagle/speculator.py", line 34, in <module>
from vllm.v1.worker.gpu.spec_decode.eagle.cudagraph import EagleCudaGraphManager
ImportError: cannot import name 'EagleCudaGraphManager' from 'vllm.v1.worker.gpu.spec_decode.eagle.cudagraph'
Explanation: The single EagleCudaGraphManager class was split into a base + two subclasses (prefill/decode). The Ascend wrapper still imports the old name, and patches vllm_speculator.EagleCudaGraphManager, an attribute that no longer exists.
Fix Suggestion: Add a version guard. For new vllm, import DecodeEagleCudaGraphManager (the closest analogue of the previous single class) and patch vllm_speculator.DecodeEagleCudaGraphManager (and PrefillEagleCudaGraphManager if needed).
Issue 4: AscendFusedMoEMethod.apply() got an unexpected keyword argument topk_weights (8 occurrences)
| Item | Detail |
|---|---|
| Category | Code Bug — substantial refactor |
| Error Type | TypeError |
| Affected Tests | many MoE-based tests (DeepSeek, Qwen3-MoE, MTP, etc.) |
| Root Cause Commits | many; key ones include 5e584ce9e "Remove SharedFusedMoE class", 809d83c2d "Combine MoERunnerBase + DefaultMoERunner", 4d51588e2 "DeepSeek V4 Rebased" |
| Changed File | vllm/model_executor/layers/fused_moe/fused_moe_method_base.py:157 (apply now takes (layer, x, topk_weights, topk_ids, shared_experts_input)) |
| Impact in vllm-ascend | vllm_ascend/ops/fused_moe/fused_moe.py:107 (AscendFusedMoEMethod.apply) |
Error Traceback:
File "vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py", line 503, in _apply_quant_method
result = self.quant_method.apply(...)
TypeError: AscendFusedMoEMethod.apply() got an unexpected keyword argument 'topk_weights'
Explanation: Upstream MoE has been refactored end-to-end: FusedMoEMethodBase.apply() no longer takes routing-related arguments; routing happens in a separate runner stage and the result is passed in as topk_weights/topk_ids. The Ascend implementation still calls select_experts itself based on router_logits, so the new caller signature is incompatible.
Fix Suggestion: This is a dedicated adaptation effort that needs to: (a) accept the new signature; (b) move select_experts into the runner / router stage; (c) audit forward_impl callers in the quantization methods (w8a8_dynamic.py, w4a8.py, etc.). Recommended to address in a follow-up PR scoped to MoE.
Issue 5: _all_lora_classes is now a tuple, not a set (4 occurrences)
| Item | Detail |
|---|---|
| Category | Code Bug |
| Error Type | AttributeError: 'tuple' object has no attribute 'add' |
| Affected Tests | LoRA tests (test_llama32_lora_tp2, test_qwen3_multi_loras, test_ilama_lora, test_qwen3_reranker_lora) |
| Root Cause Commit | a250f1bd5 — "[Bugfix] LoRA for DeepSeek V3.2 (#35077)" |
| Changed File | vllm/lora/utils.py:78 (_all_lora_classes: set[...] = {...} → tuple[...] = (...)) |
| Impact in vllm-ascend | vllm_ascend/lora/utils.py:188–199 |
Error Traceback:
File "vllm_ascend/lora/utils.py", line 188, in refresh_all_lora_classes
vllm.lora.utils._all_lora_classes.add(AscendColumnParallelLinearWithLoRA)
AttributeError: 'tuple' object has no attribute 'add'
Explanation: Upstream changed the registry from a set to an ordered tuple because the from_layer() matcher now requires a stable, more-specific-first ordering. The Ascend hot-patch still calls .add(...) on it.
Fix Suggestion: Rebuild the tuple by appending the Ascend classes (_all_lora_classes = (*_all_lora_classes, Ascend...)) when on the new vLLM. Keep set.add behavior under vllm_version_is("0.19.1").
Issue 6: ProfilingChunkScheduler.__init__() got an unexpected keyword argument hash_block_size (1 occurrence)
| Item | Detail |
|---|---|
| Category | Code Bug |
| Error Type | TypeError |
| Affected Tests | test_profiling_chunk_performance.py |
| Root Cause Commit | 7b1bc0a3e — "[Bugfix] Cap SWA/chunked-local runtime admission to startup pool-sizing bound (#40946)" (introduced hash_block_size in Scheduler.__init__) |
| Changed File | vllm/v1/core/sched/scheduler.py:74 (new `hash_block_size: int |
| Impact in vllm-ascend | vllm_ascend/core/scheduler_profiling_chunk.py:57 |
Error Traceback:
File "vllm_ascend/core/scheduler_profiling_chunk.py", line 67, in __init__
super().__init__(...)
TypeError: ProfilingChunkScheduler.__init__() got an unexpected keyword argument 'hash_block_size'
Explanation: Upstream Scheduler now accepts a new optional hash_block_size kwarg passed by the engine. The Ascend subclass overrides __init__ and doesn't list / forward this kwarg.
Fix Suggestion: Add hash_block_size: int | None = None between block_size and mm_registry, forward to super().__init__().
Issue 7: '_OpNamespace' '_moe_C' object has no attribute 'topk_softmax' (1 occurrence)
| Item | Detail |
|---|---|
| Category | Code Bug — needs new dispatch override |
| Error Type | AttributeError |
| Affected Tests | qwen3-MoE, MoE-router using softmax routing |
| Root Cause Commit | (router refactor — 447c372ac "Move remaining PrepareAndFinalize…", and friends) |
| Changed File | vllm/model_executor/layers/fused_moe/router/fused_topk_router.py:24 calls ops.topk_softmax which dispatches to torch.ops._moe_C.topk_softmax |
| Impact in vllm-ascend | needs a Platform/router override (vllm_ascend/ops/) |
Explanation: The upstream MoE router calls ops.topk_softmax which expects a CUDA _moe_C op the Ascend build does not provide. There is currently no Ascend override for dispatch_topk_softmax_func.
Fix Suggestion: Provide a NPU dispatch override (e.g., monkey-patch or a Platform-level hook) that returns a torch_npu-based topk-softmax implementation. Tracked for the larger MoE port (Issue 4 follow-up).
Issue 8: Number of global experts mismatch (excluding redundancy) (1 occurrence)
| Item | Detail |
|---|---|
| Category | Code Bug — likely follow-on of Issue 4 |
| Error Type | AssertionError |
| Affected Tests | test_deepseek3_2_w8a8_pruning_mtp_tp2_ep |
| Impact in vllm-ascend | vllm_ascend/quantization/methods/w8a8_dynamic.py:198 |
The assertion compares router_logits.shape[1] to valid_global_expert_num. After the upstream MoE refactor the shapes flowing into apply differ. Most likely this is fixed by completing Issue 4.
Cascade / Environment Issues (not fixable in this PR)
| # | Symptom | Why we skip |
|---|---|---|
| E1 | Free memory on device … (~85 occurrences) |
Memory pressure on shared CI runners — tune utilization. |
| E2 | RuntimeError: HCCL hcclCommInitRootInfoConfig … error code 7 |
HCCL bring-up flake; not a code regression. |
| E3 | vllm.v1.engine.exceptions.EngineDeadError: … |
Wrapper raised by parent process for the real failure. |
| E4 | RuntimeError: Failed to start data parallel worker 0 |
Wrapper re-throwing the underlying engine init failure. |
| E5 | KeyError: 'zero' (sympy assumptions) |
Internal torch dynamo race during teardown. |
| E6 | UnboundLocalError: tracer_output |
Inside torch._dynamo.convert_frame._compile; uninvolved. |
| E7 | AssertionError: Test0: / Output text drift (Johnson is 34/27) |
Likely accuracy follow-on once Issues 1–4 are fixed. |
Summary Table
| # | Error | Category | Upstream Commit | Affected vllm-ascend Path | Fix |
|---|---|---|---|---|---|
| 1 | encoder_compilation_time AttributeError |
Code Bug | c08f3b2a6 (#39240) |
worker/worker.py:567 |
getattr fallback |
| 2 | AscendRMSNormGated activation TypeError |
Code Bug | 893611813 (#40245) |
ops/layernorm.py:160, _310p/ops/layernorm.py:43 |
Accept activation kwarg |
| 3 | EagleCudaGraphManager ImportError |
Code Bug | 4c7c69b4e (#40410) |
worker/v2/spec_decode/eagle/{speculator,aclgraph}.py |
Version-guard import + patch new class names |
| 4 | AscendFusedMoEMethod.apply topk_weights TypeError |
Code Bug | many (e.g., 5e584ce9e, 809d83c2d, 4d51588e2) |
ops/fused_moe/fused_moe.py:107 |
Major refactor — follow-up PR |
| 5 | _all_lora_classes is tuple |
Code Bug | a250f1bd5 (#35077) |
lora/utils.py:188 |
Rebuild tuple instead of .add() |
| 6 | ProfilingChunkScheduler hash_block_size TypeError |
Code Bug | 7b1bc0a3e (#40946) |
core/scheduler_profiling_chunk.py:57 |
Forward new kwarg |
| 7 | _moe_C.topk_softmax AttributeError |
Code Bug | MoE router refactor | router dispatch override needed | Provide torch_npu topk-softmax (with Issue 4) |
| 8 | global experts shape mismatch | Code Bug | follow-on of Issue 4 | quantization/methods/w8a8_dynamic.py:198 |
Resolve once Issue 4 is fixed |
Recommended Actions (in this branch)
- Apply mechanical fixes for Issues 1, 2, 3, 5, 6 in this PR (low-risk, version-guarded).
- File a tracking issue for Issues 4 + 7 + 8 — coordinated MoE port — and address in a dedicated PR.
- Re-run e2e once mechanical fixes land; the cascading errors (E3–E5) should drop substantially.
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
|
New breaking:
|
…0410 vLLM PR #40410 split the single EagleCudaGraphManager into separate PrefillEagleCudaGraphManager and DecodeEagleCudaGraphManager, each with a different capture() signature: - Prefill: capture(forward_fn, full_cg_attn_states, ...) - Decode: capture(forward_fn, model_state, input_buffers, block_tables, ...) The upstream speculator now calls self.prefill_cudagraph_manager.capture() with only (forward_fn, attn_states), but EagleAclGraphManager still had the old decode-style signature requiring 4 extra positional args, causing: TypeError: EagleAclGraphManager.capture() missing 4 required positional arguments: 'input_buffers', 'block_tables', 'attn_groups', 'kv_cache_config' Fix by importing PrefillEagleCudaGraphManager and dispatching capture() to the correct parent class based on self.is_draft_model_prefill. Signed-off-by: gcanlin <canlinguosdu@gmail.com>
…or vLLM PR #40654 vLLM PR #40654 added seq_lens_cpu_upper_bound as a new required field to InputBatch (a CPU upper-bound on seq_lens to avoid GPU->CPU sync). AscendInputBatch inherits from InputBatch and must supply this field. Compute it the same way as upstream: num_computed_tokens_np + num_scheduled_tokens, zero-padded to num_reqs_padded, then pass it when constructing AscendInputBatch in prepare_inputs(). Signed-off-by: gcanlin <canlinguosdu@gmail.com>
…n Ascend (vLLM PR #40860) vLLM PR #40860 ([Feat] DeepSeek V4 Rebased) introduced resolve_kv_cache_block_sizes() into engine/core.py and added a restriction that hybrid KV cache groups with multiple block sizes do not support context parallelism (dcp_world_size/pcp_world_size > 1), raising: ValueError: Hybrid KV cache groups with multiple block sizes do not support context parallelism (dcp_world_size/pcp_world_size > 1). This restriction is correct for CUDA (the CUDA MLA implementation cannot combine hybrid KV with CP), but Ascend has dedicated CP backends for MLA (mla_cp.py) and SFA (sfa_cp.py) that handle this combination. Fix by patching resolve_kv_cache_block_sizes() to skip the ValueError for multiple-groups + CP on Ascend, and instead compute scheduler_block_size as lcm(group_block_sizes) * dcp * pcp for proper alignment. Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
| ) -> tuple[int, int]: | ||
| """Ascend-compatible resolve_kv_cache_block_sizes. | ||
|
|
||
| vLLM PR #40860 added a restriction that hybrid KV cache groups with |
gcanlin
changed the title
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
gcanlin
changed the title
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
3 participants
What this PR does / why we need it?
Cherry pick #8841 and then continue to upgrading vLLM.
encoder_compilation_timeAttributeErrorc08f3b2a6(#39240)worker/worker.py:567getattrfallbackAscendRMSNormGated activationTypeError893611813(#40245)ops/layernorm.py:160,_310p/ops/layernorm.py:43activationkwargEagleCudaGraphManagerImportError4c7c69b4e(#40410)worker/v2/spec_decode/eagle/{speculator,aclgraph}.pyAscendFusedMoEMethod.apply topk_weightsTypeError5e584ce9e(#35782),809d83c2d(#40560),4d51588e2(#40860))ops/fused_moe/fused_moe.py:107_all_lora_classesis tuplea250f1bd5(#35077)lora/utils.py:188.add()ProfilingChunkScheduler hash_block_sizeTypeError7b1bc0a3e(#40946)core/scheduler_profiling_chunk.py:57_moe_C.topk_softmaxAttributeErrortorch_nputopk-softmax (with Issue 4)quantization/methods/w8a8_dynamic.py:198Does this PR introduce any user-facing change?
How was this patch tested?