Skip to content

Remove unnecessary load_weights methods - #44589

Merged
vllm-bot merged 57 commits into
vllm-project:mainfrom
hmellor:remove-simple-load-weights
Jun 29, 2026
Merged

Remove unnecessary load_weights methods#44589
vllm-bot merged 57 commits into
vllm-project:mainfrom
hmellor:remove-simple-load-weights

Conversation

@hmellor

@hmellor hmellor commented Jun 4, 2026

Copy link
Copy Markdown
Member

This PR adds missing functionality to AutoWeightsLoader which allows us to delete the load_weights method boilerplate from 41 architectures in vLLM. Every one of these architectures can automatically load:

  • GPTQ checkpoints with correct bias skipping
  • FP8 checkpoints with various scale formats
  • Checkpoints with fused or sharded qkv_proj/gate_up_proj weights

The specific changes are:

  • Enables MergedColumnParallelLinear and QKVParallelLinear to load themselves from fused or unfused checkpoints without any special logic provided that the checkpoint weights are mapped correctly
    • The mappings for qkv_proj look like this, which maps checkpoint name to a shard in QKVParallelLinear:
      hf_to_vllm_mapper = WeightsMapper(
        orig_to_new_substr={
            # weight_name: (param_name, shard_id)
            ".q_proj": (".qkv_proj", "q"),
            ".k_proj": (".qkv_proj", "k"),
            ".v_proj": (".qkv_proj", "v"),
        }
      )
    • The mappings for gate_up_proj look like this, which maps checkpoint name to a shard in MergedColumnParallelLinear:
      hf_to_vllm_mapper = WeightsMapper(
          orig_to_new_substr={
              # weight_name: (param_name, shard_id)
              ".gate_proj": (".gate_up_proj", "0"),
              ".up_proj": (".gate_up_proj", "1"),
          }
      )
  • When LoRA manager and quant config use the mapper, we provide a get_unfused_mapper method as this is what these features are expecting
  • Add unexpected GPTQ bias skipping to AutoWeightsLoader

This change actually found a latent bug in the layerwise online-quantization accounting:

  1. Fp8OnlineLinearMethod.create_weights calls initialize_online_processing(layer), which snapshots load_numel_total = get_layer_size(layer) and wraps the weight loaders of tensors that exist at that moment. But ColumnParallelLinear.__init__ registers self.bias after create_weights returns — so the bias was excluded from the expected total and its loader never wrapped.
  2. For OPT (qkv biases), the counter therefore hits the weight-only total at the last weight shard, and _layerwise_process finalizes the layer before the q bias arrives: it materializes the meta weight, replays the buffered shards, quantizes, and runs the Marlin prep — which permutes the bias for the kernel epilogue and replaces the param with a bare Parameter.
  3. The trailing q-bias load then hits that post-processed param (no output_dim, shape [2304] vs [768]) → the assert in QKVParallelLinear.weight_loader.

On main, OPT's old dict-based loader held a stale params_dict snapshot, so the late bias write went into the dead pre-Marlin tensor — silent corruption that the test never caught (it only checks dtypes, explicitly not accuracy). The branch's delegation path fetches the live param at load time, which turned the silent bug into a crash.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@hmellor hmellor added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 4, 2026
@mergify mergify Bot added deepseek Related to DeepSeek models llama Related to Llama models qwen Related to Qwen models gpt-oss Related to GPT-OSS models speculative-decoding labels Jun 4, 2026
@hmellor
hmellor requested a review from jeejeelee as a code owner June 5, 2026 10:39
@mergify

mergify Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 5, 2026
hmellor added 4 commits June 5, 2026 12:29
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
hmellor force-pushed the remove-simple-load-weights branch from 9eeec97 to a9788ab Compare June 5, 2026 12:30
@mergify mergify Bot removed the needs-rebase label Jun 5, 2026
MengqingCao pushed a commit to vllm-project/vllm-ascend that referenced this pull request Jul 6, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
pratapyash added a commit to pratapyash/vllm that referenced this pull request Jul 7, 2026
…ower

Replace the hand-rolled stacked-params load_weights loop with the class-level
hf_to_vllm_mapper (orig_to_new_stacked) + AutoWeightsLoader delegation idiom
introduced by vllm-project#44589, mirroring Qwen3OmniMoeAudioEncoder. The one behavior the
old loop carried that AutoWeightsLoader does not — pre-zeroing the packed QKV
bias because HF Qwen2.5-Omni audio has bias=False for k_proj — moves to
Qwen2_5OmniAudioAttention.__init__, which zeroes qkv.bias at construction so
the never-loaded K slot is zero rather than uninitialized memory. The old
loop's non-persistent-buffer skip is subsumed: AutoWeightsLoader only targets
parameters and persistent buffers.
pratapyash added a commit to pratapyash/vllm that referenced this pull request Jul 7, 2026
…e branch

Carry the native-attention-relevant fixes from the archived compile/cudagraph
feature branch (archive/omni-audio-cudagraph-full) onto the slim
native-attention-only lineage:
- reject the unsupported FLASHINFER backend for the audio encoder (review W1-1)
- drop dead attrs in Qwen2_5OmniAudioAttention (W3-1) and the tuple return in
  the encoder layer forward (W3-5); use the positional-embedding module forward
  instead of indexing its private buffer (W3-4)
- zero the packed QKV bias at construction (HF k_proj has bias=False) and adopt
  the AutoWeightsLoader + WeightsMapper orig_to_new_stacked idiom from vllm-project#44589,
  mirroring Qwen3OmniMoeAudioEncoder
- adopt the final audio-tower processing test (FLASH_ATTN/TORCH_SDPA
  parametrization) and the qwen2_5_omni_audio HF-parity generation entry
pratapyash added a commit to pratapyash/vllm that referenced this pull request Jul 7, 2026
…e branch

Carry the native-attention-relevant fixes from the archived compile/cudagraph
feature branch (archive/omni-audio-cudagraph-full) onto the slim
native-attention-only lineage:
- reject the unsupported FLASHINFER backend for the audio encoder (review W1-1)
- drop dead attrs in Qwen2_5OmniAudioAttention (W3-1) and the tuple return in
  the encoder layer forward (W3-5); use the positional-embedding module forward
  instead of indexing its private buffer (W3-4)
- zero the packed QKV bias at construction (HF k_proj has bias=False) and adopt
  the AutoWeightsLoader + WeightsMapper orig_to_new_stacked idiom from vllm-project#44589,
  mirroring Qwen3OmniMoeAudioEncoder
- adopt the final audio-tower processing test (FLASH_ATTN/TORCH_SDPA
  parametrization) and the qwen2_5_omni_audio HF-parity generation entry
pratapyash added a commit to pratapyash/vllm that referenced this pull request Jul 7, 2026
The AutoWeightsLoader + WeightsMapper(orig_to_new_stacked=...) idiom requires
the vllm-project#44589 infrastructure, which landed after the v0.24.0 release — on this
base the mapper kwarg is a TypeError at class-definition time. Restore the
manual stacked-params loader in the W2-3-hardened form proven on the v0.23
lineage (skip non-parameter checkpoint keys via params_dict.get; record only
weights actually loaded). The K-bias zero-at-construction from the hygiene
port is kept; the loader re-zeroes defensively for reload paths.
lkk12014402 pushed a commit to lkk12014402/vllm that referenced this pull request Jul 8, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
wangyichao1999 pushed a commit to wangyichao1999/vllm-ascend that referenced this pull request Jul 9, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
noooop pushed a commit to noooop/vllm that referenced this pull request Jul 9, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
noooop pushed a commit to noooop/vllm that referenced this pull request Jul 9, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Spicy-Stick pushed a commit to Spicy-Stick/vllm-ascend that referenced this pull request Jul 10, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

---

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

---

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

---

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
---
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Signed-off-by: Spicy-Stick <873805887@qq.com>
xqchen7 pushed a commit to nv-action/vllm-benchmarks that referenced this pull request Jul 15, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

---

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

---

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

---

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
---
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Signed-off-by: xqchen7 <chenxueqing7@huawei.com>
realliujiaxu pushed a commit to vllm-project/vllm-ascend that referenced this pull request Jul 15, 2026
### What this PR does / why we need it?

This PR is a follow-up to
#11875.

PR #11875 introduced the vLLM v0.24.0 support. Those adaptation changes
are treated as the baseline and are intentionally not repeated in this
description.

This PR removes the remaining vLLM v0.23.0 compatibility paths from
`main`.

Most changes only remove `v0.23.0` branches and keep the existing
v0.24/main implementation. Those mechanical removals are not listed
individually below.

#### Release defaults and documentation

Update the default `VLLM_TAG` in all maintained Dockerfiles from
`v0.23.0` to `v0.24.0`.

The main-branch support matrix, slash-command examples, and
balance-scheduler design documents are updated accordingly. This
prevents source-built images and contributor documentation from
continuing to select the unsupported v0.23 release.

#### Fused MoE weight layout boundary

Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard
and 310P unquantized Fused MoE implementations.

Upstream vLLM PR vllm-project/vllm#44589 was
merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is
present in the verified-main revision. Therefore v0.24.0 and verified
main do not share the same post-load weight-layout behavior:

- v0.24.0 explicitly materializes the transposed weights as contiguous
tensors before the NPU layout conversion;
- verified main follows the post-PR #44589 path without forcing the same
intermediate contiguous layout.

The standard and 310P unit tests cover both version-specific layouts,
the current MoE runner contract, shared-expert handling, and the
310P-specific communication method.

#### Qwen3.5/Qwen3Next output contract

Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches
from `v0.23.0` to `v0.24.0`.

Upstream vLLM PR vllm-project/vllm#46998 was
merged as `300e3379`, after the v0.24.0 cut point and before the current
verified-main revision. It changed the attention contract from writing
into a caller-provided output buffer to returning the output tensor.

#### Balance scheduler alignment

`BalanceScheduler.schedule()` is a downstream copy of the upstream
scheduler body because the balance admission logic cannot be implemented
through a small wrapper.

The copied body is therefore updated from the v0.23.0 implementation to
the v0.24.0 implementation while preserving only the existing
balance-scheduling deltas.

Both supported upstream references now expose:

```python
schedule(self, throttle_prefills: bool = False)
```

The old signature-introspection compatibility code is removed and the
disabled path delegates directly to
`super().schedule(throttle_prefills)`.

The v0.24 scheduler alignment also preserves the corresponding upstream
behavior for:

- DP prefill throttling;
- speculative-token and maximum-length accounting;
- hybrid Mamba KV-cache hit handling;
- resumed-request bookkeeping;
- dynamic speculative decoding;
- deferred KV-block freeing;
- MRV1-only previous-step request tracking.

The balance scheduler unit tests and English/Chinese design documents
are updated to use v0.24.0 as the release reference. The drift test
continues to verify that the copied scheduler body differs from the
pinned upstream release only by the intended balance deltas.

#### Deferred removal of owner-maintained patches

The following compatibility patches and their unit tests are
intentionally retained in this PR:

- GLM47 zero-argument tool-call streaming parser;
- MiniMax-M2 incremental tool-call parser;
- MiniMax usage accounting;
- `tool_choice=none` empty-`tool_calls` response cleanup.

The first three remain behind the existing `vllm_version_is("0.23.0")`
condition, with a TODO explaining that their owners will remove them in
a follow-up. The `patch_tool_choice_none_content` registration is also
left unchanged for the same ownership reason.

These files are not required by the newly supported v0.24/main lanes,
but deleting owner-maintained patches is intentionally outside the scope
of this compatibility cleanup.

#### Test boundaries

The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were
skipped on main. Since v0.23.0 is removed and the cases remain broken on
both v0.24.0 and verified main, they are now explicitly skipped on both
supported lanes instead of being unintentionally re-enabled.

The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather
than v0.23.0. MRV2 remains supported only by the verified-main lane, as
established by PR #11875.

The HunyuanVL release helper names are updated from `_v023_*` to
`_v024_*` because the bundled-processor compatibility path now targets
v0.24.0. This is a naming correction only; the HunyuanVL adaptation
itself belongs to PR #11875.

### Does this PR introduce _any_ user-facing change?

Yes.

The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker
builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default.

There is no additional API change for the supported v0.24.0 and
verified-main lanes.

### How was this patch tested?


- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@85c09e9

---------

Signed-off-by: zhao-stack <2020265299@qq.com>
Signed-off-by: shenzhao <shenzhao9@huawei.com>
Signed-off-by: MrZ20 <2609716663@qq.com>
Co-authored-by: zhao-stack <2020265299@qq.com>
Co-authored-by: shenzhao <shenzhao9@huawei.com>
Dao007forever pushed a commit to Dao007forever/vllm that referenced this pull request Jul 18, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Dao007forever pushed a commit to Dao007forever/vllm that referenced this pull request Jul 18, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
ningjingbengxiaohai pushed a commit to ningjingbengxiaohai/vllm-ascend that referenced this pull request Jul 18, 2026
### What this PR does / why we need it?

This PR is a follow-up to
vllm-project#11875.

PR vllm-project#11875 introduced the vLLM v0.24.0 support. Those adaptation changes
are treated as the baseline and are intentionally not repeated in this
description.

This PR removes the remaining vLLM v0.23.0 compatibility paths from
`main`.

Most changes only remove `v0.23.0` branches and keep the existing
v0.24/main implementation. Those mechanical removals are not listed
individually below.

#### Release defaults and documentation

Update the default `VLLM_TAG` in all maintained Dockerfiles from
`v0.23.0` to `v0.24.0`.

The main-branch support matrix, slash-command examples, and
balance-scheduler design documents are updated accordingly. This
prevents source-built images and contributor documentation from
continuing to select the unsupported v0.23 release.

#### Fused MoE weight layout boundary

Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard
and 310P unquantized Fused MoE implementations.

Upstream vLLM PR vllm-project/vllm#44589 was
merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is
present in the verified-main revision. Therefore v0.24.0 and verified
main do not share the same post-load weight-layout behavior:

- v0.24.0 explicitly materializes the transposed weights as contiguous
tensors before the NPU layout conversion;
- verified main follows the post-PR #44589 path without forcing the same
intermediate contiguous layout.

The standard and 310P unit tests cover both version-specific layouts,
the current MoE runner contract, shared-expert handling, and the
310P-specific communication method.

#### Qwen3.5/Qwen3Next output contract

Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches
from `v0.23.0` to `v0.24.0`.

Upstream vLLM PR vllm-project/vllm#46998 was
merged as `300e3379`, after the v0.24.0 cut point and before the current
verified-main revision. It changed the attention contract from writing
into a caller-provided output buffer to returning the output tensor.

#### Balance scheduler alignment

`BalanceScheduler.schedule()` is a downstream copy of the upstream
scheduler body because the balance admission logic cannot be implemented
through a small wrapper.

The copied body is therefore updated from the v0.23.0 implementation to
the v0.24.0 implementation while preserving only the existing
balance-scheduling deltas.

Both supported upstream references now expose:

```python
schedule(self, throttle_prefills: bool = False)
```

The old signature-introspection compatibility code is removed and the
disabled path delegates directly to
`super().schedule(throttle_prefills)`.

The v0.24 scheduler alignment also preserves the corresponding upstream
behavior for:

- DP prefill throttling;
- speculative-token and maximum-length accounting;
- hybrid Mamba KV-cache hit handling;
- resumed-request bookkeeping;
- dynamic speculative decoding;
- deferred KV-block freeing;
- MRV1-only previous-step request tracking.

The balance scheduler unit tests and English/Chinese design documents
are updated to use v0.24.0 as the release reference. The drift test
continues to verify that the copied scheduler body differs from the
pinned upstream release only by the intended balance deltas.

#### Deferred removal of owner-maintained patches

The following compatibility patches and their unit tests are
intentionally retained in this PR:

- GLM47 zero-argument tool-call streaming parser;
- MiniMax-M2 incremental tool-call parser;
- MiniMax usage accounting;
- `tool_choice=none` empty-`tool_calls` response cleanup.

The first three remain behind the existing `vllm_version_is("0.23.0")`
condition, with a TODO explaining that their owners will remove them in
a follow-up. The `patch_tool_choice_none_content` registration is also
left unchanged for the same ownership reason.

These files are not required by the newly supported v0.24/main lanes,
but deleting owner-maintained patches is intentionally outside the scope
of this compatibility cleanup.

#### Test boundaries

The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were
skipped on main. Since v0.23.0 is removed and the cases remain broken on
both v0.24.0 and verified main, they are now explicitly skipped on both
supported lanes instead of being unintentionally re-enabled.

The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather
than v0.23.0. MRV2 remains supported only by the verified-main lane, as
established by PR vllm-project#11875.

The HunyuanVL release helper names are updated from `_v023_*` to
`_v024_*` because the bundled-processor compatibility path now targets
v0.24.0. This is a naming correction only; the HunyuanVL adaptation
itself belongs to PR vllm-project#11875.

### Does this PR introduce _any_ user-facing change?

Yes.

The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker
builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default.

There is no additional API change for the supported v0.24.0 and
verified-main lanes.

### How was this patch tested?


- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@85c09e9

---------

Signed-off-by: zhao-stack <2020265299@qq.com>
Signed-off-by: shenzhao <shenzhao9@huawei.com>
Signed-off-by: MrZ20 <2609716663@qq.com>
Co-authored-by: zhao-stack <2020265299@qq.com>
Co-authored-by: shenzhao <shenzhao9@huawei.com>
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
weijinqian0 pushed a commit to vllm-project/vllm-ascend that referenced this pull request Jul 23, 2026
…2463)

### What this PR does / why we need it?
After vllm-project/vllm#43167
vllm-project/vllm#44589, we can remove our two
patches about kvcache quant weight.

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

- vLLM version: v0.25.1
- vLLM main:
vllm-project/vllm@54503ec

Signed-off-by: menogrey <1299267905@qq.com>
Alex-stack-hub pushed a commit to 0moyi0-2024/vllm-ascend_tp that referenced this pull request Jul 27, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Alex-stack-hub pushed a commit to 0moyi0-2024/vllm-ascend_tp that referenced this pull request Jul 27, 2026
### What this PR does / why we need it?

This PR is a follow-up to
vllm-project#11875.

PR vllm-project#11875 introduced the vLLM v0.24.0 support. Those adaptation changes
are treated as the baseline and are intentionally not repeated in this
description.

This PR removes the remaining vLLM v0.23.0 compatibility paths from
`main`.

Most changes only remove `v0.23.0` branches and keep the existing
v0.24/main implementation. Those mechanical removals are not listed
individually below.

#### Release defaults and documentation

Update the default `VLLM_TAG` in all maintained Dockerfiles from
`v0.23.0` to `v0.24.0`.

The main-branch support matrix, slash-command examples, and
balance-scheduler design documents are updated accordingly. This
prevents source-built images and contributor documentation from
continuing to select the unsupported v0.23 release.

#### Fused MoE weight layout boundary

Keep an explicit `vllm_version_is("0.24.0")` branch in both the standard
and 310P unquantized Fused MoE implementations.

Upstream vLLM PR vllm-project/vllm#44589 was
merged as `5051698e`, 26 commits after the v0.24.0 cut point, and is
present in the verified-main revision. Therefore v0.24.0 and verified
main do not share the same post-load weight-layout behavior:

- v0.24.0 explicitly materializes the transposed weights as contiguous
tensors before the NPU layout conversion;
- verified main follows the post-PR #44589 path without forcing the same
intermediate contiguous layout.

The standard and 310P unit tests cover both version-specific layouts,
the current MoE runner contract, shared-expert handling, and the
310P-specific communication method.

#### Qwen3.5/Qwen3Next output contract

Change the version boundary in the GDN and Qwen3.5/Qwen3Next patches
from `v0.23.0` to `v0.24.0`.

Upstream vLLM PR vllm-project/vllm#46998 was
merged as `300e3379`, after the v0.24.0 cut point and before the current
verified-main revision. It changed the attention contract from writing
into a caller-provided output buffer to returning the output tensor.

#### Balance scheduler alignment

`BalanceScheduler.schedule()` is a downstream copy of the upstream
scheduler body because the balance admission logic cannot be implemented
through a small wrapper.

The copied body is therefore updated from the v0.23.0 implementation to
the v0.24.0 implementation while preserving only the existing
balance-scheduling deltas.

Both supported upstream references now expose:

```python
schedule(self, throttle_prefills: bool = False)
```

The old signature-introspection compatibility code is removed and the
disabled path delegates directly to
`super().schedule(throttle_prefills)`.

The v0.24 scheduler alignment also preserves the corresponding upstream
behavior for:

- DP prefill throttling;
- speculative-token and maximum-length accounting;
- hybrid Mamba KV-cache hit handling;
- resumed-request bookkeeping;
- dynamic speculative decoding;
- deferred KV-block freeing;
- MRV1-only previous-step request tracking.

The balance scheduler unit tests and English/Chinese design documents
are updated to use v0.24.0 as the release reference. The drift test
continues to verify that the copied scheduler body differs from the
pinned upstream release only by the intended balance deltas.

#### Deferred removal of owner-maintained patches

The following compatibility patches and their unit tests are
intentionally retained in this PR:

- GLM47 zero-argument tool-call streaming parser;
- MiniMax-M2 incremental tool-call parser;
- MiniMax usage accounting;
- `tool_choice=none` empty-`tool_calls` response cleanup.

The first three remain behind the existing `vllm_version_is("0.23.0")`
condition, with a TODO explaining that their owners will remove them in
a follow-up. The `patch_tool_choice_none_content` registration is also
left unchanged for the same ownership reason.

These files are not required by the newly supported v0.24/main lanes,
but deleting owner-maintained patches is intentionally outside the scope
of this compatibility cleanup.

#### Test boundaries

The two Qwen3 MoE/EPLB E2E files previously ran only on v0.23.0 and were
skipped on main. Since v0.23.0 is removed and the cases remain broken on
both v0.24.0 and verified main, they are now explicitly skipped on both
supported lanes instead of being unintentionally re-enabled.

The newly rebased MRV2 data-parallel test is skipped on v0.24.0 rather
than v0.23.0. MRV2 remains supported only by the verified-main lane, as
established by PR vllm-project#11875.

The HunyuanVL release helper names are updated from `_v023_*` to
`_v024_*` because the bundled-processor compatibility path now targets
v0.24.0. This is a naming correction only; the HunyuanVL adaptation
itself belongs to PR vllm-project#11875.

### Does this PR introduce _any_ user-facing change?

Yes.

The vLLM Ascend main branch no longer supports vLLM v0.23.0. Docker
builds that do not override `VLLM_TAG` now use vLLM v0.24.0 by default.

There is no additional API change for the supported v0.24.0 and
verified-main lanes.

### How was this patch tested?


- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@85c09e9

---------

Signed-off-by: zhao-stack <2020265299@qq.com>
Signed-off-by: shenzhao <shenzhao9@huawei.com>
Signed-off-by: MrZ20 <2609716663@qq.com>
Co-authored-by: zhao-stack <2020265299@qq.com>
Co-authored-by: shenzhao <shenzhao9@huawei.com>
aditi-amd pushed a commit to aditi-amd/vllm that referenced this pull request Aug 4, 2026
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models gpt-oss Related to GPT-OSS models llama Related to Llama models mistral Related to Mistral models qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed speculative-decoding

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants