Remove unnecessary load_weights methods - #44589
Merged
Merged
Conversation
hmellor
requested review from
22quinn,
mgoin,
patrickvonplaten,
pavanimajety,
robertgshaw2-redhat,
sighingnow,
tjtanaa,
tlrmchlsmth,
tomeras91,
vadiklyutiy,
yewentao256 and
zyongye
as code owners
June 4, 2026 23:47
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
hmellor
force-pushed
the
remove-simple-load-weights
branch
from
June 5, 2026 12:30
9eeec97 to
a9788ab
Compare
MengqingCao
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 6, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
pratapyash
added a commit
to pratapyash/vllm
that referenced
this pull request
Jul 7, 2026
…ower Replace the hand-rolled stacked-params load_weights loop with the class-level hf_to_vllm_mapper (orig_to_new_stacked) + AutoWeightsLoader delegation idiom introduced by vllm-project#44589, mirroring Qwen3OmniMoeAudioEncoder. The one behavior the old loop carried that AutoWeightsLoader does not — pre-zeroing the packed QKV bias because HF Qwen2.5-Omni audio has bias=False for k_proj — moves to Qwen2_5OmniAudioAttention.__init__, which zeroes qkv.bias at construction so the never-loaded K slot is zero rather than uninitialized memory. The old loop's non-persistent-buffer skip is subsumed: AutoWeightsLoader only targets parameters and persistent buffers.
pratapyash
added a commit
to pratapyash/vllm
that referenced
this pull request
Jul 7, 2026
…e branch Carry the native-attention-relevant fixes from the archived compile/cudagraph feature branch (archive/omni-audio-cudagraph-full) onto the slim native-attention-only lineage: - reject the unsupported FLASHINFER backend for the audio encoder (review W1-1) - drop dead attrs in Qwen2_5OmniAudioAttention (W3-1) and the tuple return in the encoder layer forward (W3-5); use the positional-embedding module forward instead of indexing its private buffer (W3-4) - zero the packed QKV bias at construction (HF k_proj has bias=False) and adopt the AutoWeightsLoader + WeightsMapper orig_to_new_stacked idiom from vllm-project#44589, mirroring Qwen3OmniMoeAudioEncoder - adopt the final audio-tower processing test (FLASH_ATTN/TORCH_SDPA parametrization) and the qwen2_5_omni_audio HF-parity generation entry
8 tasks
pratapyash
added a commit
to pratapyash/vllm
that referenced
this pull request
Jul 7, 2026
…e branch Carry the native-attention-relevant fixes from the archived compile/cudagraph feature branch (archive/omni-audio-cudagraph-full) onto the slim native-attention-only lineage: - reject the unsupported FLASHINFER backend for the audio encoder (review W1-1) - drop dead attrs in Qwen2_5OmniAudioAttention (W3-1) and the tuple return in the encoder layer forward (W3-5); use the positional-embedding module forward instead of indexing its private buffer (W3-4) - zero the packed QKV bias at construction (HF k_proj has bias=False) and adopt the AutoWeightsLoader + WeightsMapper orig_to_new_stacked idiom from vllm-project#44589, mirroring Qwen3OmniMoeAudioEncoder - adopt the final audio-tower processing test (FLASH_ATTN/TORCH_SDPA parametrization) and the qwen2_5_omni_audio HF-parity generation entry
pratapyash
added a commit
to pratapyash/vllm
that referenced
this pull request
Jul 7, 2026
The AutoWeightsLoader + WeightsMapper(orig_to_new_stacked=...) idiom requires the vllm-project#44589 infrastructure, which landed after the v0.24.0 release — on this base the mapper kwarg is a TypeError at class-definition time. Restore the manual stacked-params loader in the W2-3-hardened form proven on the v0.23 lineage (skip non-parameter checkpoint keys via params_dict.get; record only weights actually loaded). The K-bias zero-at-construction from the hygiene port is kept; the loader re-zeroes defensively for reload paths.
lkk12014402
pushed a commit
to lkk12014402/vllm
that referenced
this pull request
Jul 8, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
wangyichao1999
pushed a commit
to wangyichao1999/vllm-ascend
that referenced
this pull request
Jul 9, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
noooop
pushed a commit
to noooop/vllm
that referenced
this pull request
Jul 9, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
noooop
pushed a commit
to noooop/vllm
that referenced
this pull request
Jul 9, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Spicy-Stick
pushed a commit
to Spicy-Stick/vllm-ascend
that referenced
this pull request
Jul 10, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com> Signed-off-by: Spicy-Stick <873805887@qq.com>
xqchen7
pushed a commit
to nv-action/vllm-benchmarks
that referenced
this pull request
Jul 15, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com> Signed-off-by: xqchen7 <chenxueqing7@huawei.com>
realliujiaxu
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 15, 2026
### What this PR does / why we need it? This PR is a follow-up to #11875. PR #11875 introduced the vLLM v0.24.0 support. Those adaptation changes are treated as the baseline and are intentionally not repeated in this description. This PR removes the remaining vLLM v0.23.0 compatibility paths from `main`. Most changes only remove `v0.23.0` branches and keep the existing v0.24/main implementation. Those mechanical removals are not listed individually below. #### Release defaults and documentation Update the default `VLLM_TAG` in all maintained Dockerfiles from `v0.23.0` to `v0.24.0`. The main-branch support matrix, slash-command examples, and balance-scheduler design documents are updated accordingly. This prevents source-built images and contributor documentation from continuing to select the unsupported v0.23 release. #### Fused MoE weight layout boundary Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard and 310P unquantized Fused MoE implementations. Upstream vLLM PR vllm-project/vllm#44589 was merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is present in the verified-main revision. Therefore v0.24.0 and verified main do not share the same post-load weight-layout behavior: - v0.24.0 explicitly materializes the transposed weights as contiguous tensors before the NPU layout conversion; - verified main follows the post-PR #44589 path without forcing the same intermediate contiguous layout. The standard and 310P unit tests cover both version-specific layouts, the current MoE runner contract, shared-expert handling, and the 310P-specific communication method. #### Qwen3.5/Qwen3Next output contract Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches from `v0.23.0` to `v0.24.0`. Upstream vLLM PR vllm-project/vllm#46998 was merged as `300e3379`, after the v0.24.0 cut point and before the current verified-main revision. It changed the attention contract from writing into a caller-provided output buffer to returning the output tensor. #### Balance scheduler alignment `BalanceScheduler.schedule()` is a downstream copy of the upstream scheduler body because the balance admission logic cannot be implemented through a small wrapper. The copied body is therefore updated from the v0.23.0 implementation to the v0.24.0 implementation while preserving only the existing balance-scheduling deltas. Both supported upstream references now expose: ```python schedule(self, throttle_prefills: bool = False) ``` The old signature-introspection compatibility code is removed and the disabled path delegates directly to `super().schedule(throttle_prefills)`. The v0.24 scheduler alignment also preserves the corresponding upstream behavior for: - DP prefill throttling; - speculative-token and maximum-length accounting; - hybrid Mamba KV-cache hit handling; - resumed-request bookkeeping; - dynamic speculative decoding; - deferred KV-block freeing; - MRV1-only previous-step request tracking. The balance scheduler unit tests and English/Chinese design documents are updated to use v0.24.0 as the release reference. The drift test continues to verify that the copied scheduler body differs from the pinned upstream release only by the intended balance deltas. #### Deferred removal of owner-maintained patches The following compatibility patches and their unit tests are intentionally retained in this PR: - GLM47 zero-argument tool-call streaming parser; - MiniMax-M2 incremental tool-call parser; - MiniMax usage accounting; - `tool_choice=none` empty-`tool_calls` response cleanup. The first three remain behind the existing `vllm_version_is("0.23.0")` condition, with a TODO explaining that their owners will remove them in a follow-up. The `patch_tool_choice_none_content` registration is also left unchanged for the same ownership reason. These files are not required by the newly supported v0.24/main lanes, but deleting owner-maintained patches is intentionally outside the scope of this compatibility cleanup. #### Test boundaries The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were skipped on main. Since v0.23.0 is removed and the cases remain broken on both v0.24.0 and verified main, they are now explicitly skipped on both supported lanes instead of being unintentionally re-enabled. The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather than v0.23.0. MRV2 remains supported only by the verified-main lane, as established by PR #11875. The HunyuanVL release helper names are updated from `_v023_*` to `_v024_*` because the bundled-processor compatibility path now targets v0.24.0. This is a naming correction only; the HunyuanVL adaptation itself belongs to PR #11875. ### Does this PR introduce _any_ user-facing change? Yes. The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default. There is no additional API change for the supported v0.24.0 and verified-main lanes. ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@85c09e9 --------- Signed-off-by: zhao-stack <2020265299@qq.com> Signed-off-by: shenzhao <shenzhao9@huawei.com> Signed-off-by: MrZ20 <2609716663@qq.com> Co-authored-by: zhao-stack <2020265299@qq.com> Co-authored-by: shenzhao <shenzhao9@huawei.com>
4 tasks
Dao007forever
pushed a commit
to Dao007forever/vllm
that referenced
this pull request
Jul 18, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Dao007forever
pushed a commit
to Dao007forever/vllm
that referenced
this pull request
Jul 18, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
ningjingbengxiaohai
pushed a commit
to ningjingbengxiaohai/vllm-ascend
that referenced
this pull request
Jul 18, 2026
### What this PR does / why we need it? This PR is a follow-up to vllm-project#11875. PR vllm-project#11875 introduced the vLLM v0.24.0 support. Those adaptation changes are treated as the baseline and are intentionally not repeated in this description. This PR removes the remaining vLLM v0.23.0 compatibility paths from `main`. Most changes only remove `v0.23.0` branches and keep the existing v0.24/main implementation. Those mechanical removals are not listed individually below. #### Release defaults and documentation Update the default `VLLM_TAG` in all maintained Dockerfiles from `v0.23.0` to `v0.24.0`. The main-branch support matrix, slash-command examples, and balance-scheduler design documents are updated accordingly. This prevents source-built images and contributor documentation from continuing to select the unsupported v0.23 release. #### Fused MoE weight layout boundary Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard and 310P unquantized Fused MoE implementations. Upstream vLLM PR vllm-project/vllm#44589 was merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is present in the verified-main revision. Therefore v0.24.0 and verified main do not share the same post-load weight-layout behavior: - v0.24.0 explicitly materializes the transposed weights as contiguous tensors before the NPU layout conversion; - verified main follows the post-PR #44589 path without forcing the same intermediate contiguous layout. The standard and 310P unit tests cover both version-specific layouts, the current MoE runner contract, shared-expert handling, and the 310P-specific communication method. #### Qwen3.5/Qwen3Next output contract Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches from `v0.23.0` to `v0.24.0`. Upstream vLLM PR vllm-project/vllm#46998 was merged as `300e3379`, after the v0.24.0 cut point and before the current verified-main revision. It changed the attention contract from writing into a caller-provided output buffer to returning the output tensor. #### Balance scheduler alignment `BalanceScheduler.schedule()` is a downstream copy of the upstream scheduler body because the balance admission logic cannot be implemented through a small wrapper. The copied body is therefore updated from the v0.23.0 implementation to the v0.24.0 implementation while preserving only the existing balance-scheduling deltas. Both supported upstream references now expose: ```python schedule(self, throttle_prefills: bool = False) ``` The old signature-introspection compatibility code is removed and the disabled path delegates directly to `super().schedule(throttle_prefills)`. The v0.24 scheduler alignment also preserves the corresponding upstream behavior for: - DP prefill throttling; - speculative-token and maximum-length accounting; - hybrid Mamba KV-cache hit handling; - resumed-request bookkeeping; - dynamic speculative decoding; - deferred KV-block freeing; - MRV1-only previous-step request tracking. The balance scheduler unit tests and English/Chinese design documents are updated to use v0.24.0 as the release reference. The drift test continues to verify that the copied scheduler body differs from the pinned upstream release only by the intended balance deltas. #### Deferred removal of owner-maintained patches The following compatibility patches and their unit tests are intentionally retained in this PR: - GLM47 zero-argument tool-call streaming parser; - MiniMax-M2 incremental tool-call parser; - MiniMax usage accounting; - `tool_choice=none` empty-`tool_calls` response cleanup. The first three remain behind the existing `vllm_version_is("0.23.0")` condition, with a TODO explaining that their owners will remove them in a follow-up. The `patch_tool_choice_none_content` registration is also left unchanged for the same ownership reason. These files are not required by the newly supported v0.24/main lanes, but deleting owner-maintained patches is intentionally outside the scope of this compatibility cleanup. #### Test boundaries The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were skipped on main. Since v0.23.0 is removed and the cases remain broken on both v0.24.0 and verified main, they are now explicitly skipped on both supported lanes instead of being unintentionally re-enabled. The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather than v0.23.0. MRV2 remains supported only by the verified-main lane, as established by PR vllm-project#11875. The HunyuanVL release helper names are updated from `_v023_*` to `_v024_*` because the bundled-processor compatibility path now targets v0.24.0. This is a naming correction only; the HunyuanVL adaptation itself belongs to PR vllm-project#11875. ### Does this PR introduce _any_ user-facing change? Yes. The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default. There is no additional API change for the supported v0.24.0 and verified-main lanes. ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@85c09e9 --------- Signed-off-by: zhao-stack <2020265299@qq.com> Signed-off-by: shenzhao <shenzhao9@huawei.com> Signed-off-by: MrZ20 <2609716663@qq.com> Co-authored-by: zhao-stack <2020265299@qq.com> Co-authored-by: shenzhao <shenzhao9@huawei.com>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
plasticchris
pushed a commit
to plasticchris/vllm
that referenced
this pull request
Jul 20, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
plasticchris
pushed a commit
to plasticchris/vllm
that referenced
this pull request
Jul 20, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
weijinqian0
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 23, 2026
…2463) ### What this PR does / why we need it? After vllm-project/vllm#43167 vllm-project/vllm#44589, we can remove our two patches about kvcache quant weight. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.25.1 - vLLM main: vllm-project/vllm@54503ec Signed-off-by: menogrey <1299267905@qq.com>
4 tasks
Alex-stack-hub
pushed a commit
to 0moyi0-2024/vllm-ascend_tp
that referenced
this pull request
Jul 27, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Alex-stack-hub
pushed a commit
to 0moyi0-2024/vllm-ascend_tp
that referenced
this pull request
Jul 27, 2026
### What this PR does / why we need it? This PR is a follow-up to vllm-project#11875. PR vllm-project#11875 introduced the vLLM v0.24.0 support. Those adaptation changes are treated as the baseline and are intentionally not repeated in this description. This PR removes the remaining vLLM v0.23.0 compatibility paths from `main`. Most changes only remove `v0.23.0` branches and keep the existing v0.24/main implementation. Those mechanical removals are not listed individually below. #### Release defaults and documentation Update the default `VLLM_TAG` in all maintained Dockerfiles from `v0.23.0` to `v0.24.0`. The main-branch support matrix, slash-command examples, and balance-scheduler design documents are updated accordingly. This prevents source-built images and contributor documentation from continuing to select the unsupported v0.23 release. #### Fused MoE weight layout boundary Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard and 310P unquantized Fused MoE implementations. Upstream vLLM PR vllm-project/vllm#44589 was merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is present in the verified-main revision. Therefore v0.24.0 and verified main do not share the same post-load weight-layout behavior: - v0.24.0 explicitly materializes the transposed weights as contiguous tensors before the NPU layout conversion; - verified main follows the post-PR #44589 path without forcing the same intermediate contiguous layout. The standard and 310P unit tests cover both version-specific layouts, the current MoE runner contract, shared-expert handling, and the 310P-specific communication method. #### Qwen3.5/Qwen3Next output contract Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches from `v0.23.0` to `v0.24.0`. Upstream vLLM PR vllm-project/vllm#46998 was merged as `300e3379`, after the v0.24.0 cut point and before the current verified-main revision. It changed the attention contract from writing into a caller-provided output buffer to returning the output tensor. #### Balance scheduler alignment `BalanceScheduler.schedule()` is a downstream copy of the upstream scheduler body because the balance admission logic cannot be implemented through a small wrapper. The copied body is therefore updated from the v0.23.0 implementation to the v0.24.0 implementation while preserving only the existing balance-scheduling deltas. Both supported upstream references now expose: ```python schedule(self, throttle_prefills: bool = False) ``` The old signature-introspection compatibility code is removed and the disabled path delegates directly to `super().schedule(throttle_prefills)`. The v0.24 scheduler alignment also preserves the corresponding upstream behavior for: - DP prefill throttling; - speculative-token and maximum-length accounting; - hybrid Mamba KV-cache hit handling; - resumed-request bookkeeping; - dynamic speculative decoding; - deferred KV-block freeing; - MRV1-only previous-step request tracking. The balance scheduler unit tests and English/Chinese design documents are updated to use v0.24.0 as the release reference. The drift test continues to verify that the copied scheduler body differs from the pinned upstream release only by the intended balance deltas. #### Deferred removal of owner-maintained patches The following compatibility patches and their unit tests are intentionally retained in this PR: - GLM47 zero-argument tool-call streaming parser; - MiniMax-M2 incremental tool-call parser; - MiniMax usage accounting; - `tool_choice=none` empty-`tool_calls` response cleanup. The first three remain behind the existing `vllm_version_is("0.23.0")` condition, with a TODO explaining that their owners will remove them in a follow-up. The `patch_tool_choice_none_content` registration is also left unchanged for the same ownership reason. These files are not required by the newly supported v0.24/main lanes, but deleting owner-maintained patches is intentionally outside the scope of this compatibility cleanup. #### Test boundaries The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were skipped on main. Since v0.23.0 is removed and the cases remain broken on both v0.24.0 and verified main, they are now explicitly skipped on both supported lanes instead of being unintentionally re-enabled. The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather than v0.23.0. MRV2 remains supported only by the verified-main lane, as established by PR vllm-project#11875. The HunyuanVL release helper names are updated from `_v023_*` to `_v024_*` because the bundled-processor compatibility path now targets v0.24.0. This is a naming correction only; the HunyuanVL adaptation itself belongs to PR vllm-project#11875. ### Does this PR introduce _any_ user-facing change? Yes. The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default. There is no additional API change for the supported v0.24.0 and verified-main lanes. ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@85c09e9 --------- Signed-off-by: zhao-stack <2020265299@qq.com> Signed-off-by: shenzhao <shenzhao9@huawei.com> Signed-off-by: MrZ20 <2609716663@qq.com> Co-authored-by: zhao-stack <2020265299@qq.com> Co-authored-by: shenzhao <shenzhao9@huawei.com>
aditi-amd
pushed a commit
to aditi-amd/vllm
that referenced
this pull request
Aug 4, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds missing functionality to
AutoWeightsLoaderwhich allows us to delete theload_weightsmethod boilerplate from 41 architectures in vLLM. Every one of these architectures can automatically load:qkv_proj/gate_up_projweightsThe specific changes are:
MergedColumnParallelLinearandQKVParallelLinearto load themselves from fused or unfused checkpoints without any special logic provided that the checkpoint weights are mapped correctlyqkv_projlook like this, which maps checkpoint name to a shard inQKVParallelLinear:gate_up_projlook like this, which maps checkpoint name to a shard inMergedColumnParallelLinear:get_unfused_mappermethod as this is what these features are expectingAutoWeightsLoaderThis change actually found a latent bug in the layerwise online-quantization accounting:
Fp8OnlineLinearMethod.create_weightscallsinitialize_online_processing(layer), which snapshotsload_numel_total = get_layer_size(layer)and wraps the weight loaders of tensors that exist at that moment. ButColumnParallelLinear.__init__registersself.biasaftercreate_weightsreturns — so the bias was excluded from the expected total and its loader never wrapped._layerwise_processfinalizes the layer before the q bias arrives: it materializes the meta weight, replays the buffered shards, quantizes, and runs the Marlin prep — which permutes the bias for the kernel epilogue and replaces the param with a bareParameter.output_dim, shape [2304] vs [768]) → the assert inQKVParallelLinear.weight_loader.On main, OPT's old dict-based loader held a stale
params_dictsnapshot, so the late bias write went into the dead pre-Marlin tensor — silent corruption that the test never caught (it only checks dtypes, explicitly not accuracy). The branch's delegation path fetches the live param at load time, which turned the silent bug into a crash.