[LoRA] feat: Support LoRA for DeepSeek V4 by HollowMan6 · Pull Request #52986 · vllm-project/vllm

HollowMan6 · 2026-08-19T17:44:08Z

Purpose

Add LoRA support for DeepSeek V4 (DSV4). DSV4 is a MoE model with quantized (fp8/mxfp4) experts and an MTP draft head. Three issues blocked merge=False (live-adapter) LoRA, causing train/inference inconsistency:

Non-LoRA params on a LoRA-wrapped module (e.g. gate.tid2eid, gate.e_score_correction_bias) live at <head>.base_layer.<leaf> in the live namespace, but the checkpoint name is plain. The DSV4 load_weights strict params_dict[name] lookup KeyError'd / orphaned these on the initial profile load.
Expert mapping param_name vs weight_name prefix confusion. RoutedExperts.make_expert_params_mapping applied a single lora_base_layer_prefix to both sides. But get_expert_mapping resolves param_name via getattr against this layer's bare w13_weight/w2_weight (no prefix), while make_expert_params_mapping indexes the model-wide params_dict (prefix included). One prefix for both sides silently dropped per-expert weights.
weight_scale_inv / scale not forwarded through the LoRA wrapper. Custom kernels read weight_scale_inv through the wrapper; without __getattr__ forwarding the attribute lookup failed.

This PR adds SupportsLoRA to DeepseekV4ForCausalLM, declares packed_modules_mapping and lora_skip_prefixes (the MTP draft head is not LoRA-adapted), adds .base_layer.-namespace variants to the weights mapper scale regexes, reconciles non-LoRA params in the inner load_weights, splits the expert mapping prefix into checkpoint-side (weight_name) and live-side (param_name), and forwards public attribute misses in BaseLayerWithLoRA to base_layer.

Test Plan

Test with verl E2E under verl-project/verl#7483

Test Result

DSV4 merge=False LoRA GRPO, 2-node (Megatron backend):

step1 pearson (rollout/actor log-prob correlation) = 0.99705
step2 pearson = 0.9950
vs merge=True (folded, no LoRA runtime) baseline run15 = 0.99525 / 0.99440

Before this PR, merge=False diverged (pearson ~0.43) due to dropped per-expert weights and orphaned non-LoRA params; after, it aligns with the merge=True baseline to within noise.

Details

`BaseLayerWithLoRA.getattr` (`vllm/lora/layers/base.py`)

Forwards public attribute misses to base_layer (e.g. weight_scale_inv read by custom kernels through the wrapper). Private (_-prefixed) names are framework bookkeeping and stay local.

`RoutedExperts` dual-prefix mapping (`vllm/model_executor/layers/fused_moe/routed_experts.py`)

build_expert_params_mapping now passes the base_layer. prefix to both lora_base_layer_prefix (checkpoint weight_name side) and lora_base_layer_prefix_on_param_name (live param_name side).
make_expert_params_mapping gains lora_base_layer_prefix_on_param_name and applies it to the w13/w2 param names, while weight_name keeps the existing lora_base_layer_prefix. This matches the fact that get_expert_mapping resolves param_name via getattr on bare w13_weight/w2_weight, whereas make_expert_params_mapping indexes model-wide params_dict (prefix included).

`DeepseekV4Model.load_weights` reconcile (`vllm/models/deepseek_v4/nvidia/model.py`)

Inner strict-loader else-branch: when a plain incoming name is missing from params_dict, try <head>.base_layer.<leaf> and use it if live. Covers non-LoRA params on a wrapped module during the initial checkpoint load (profile_run), guarded so weight-sync (where the suffix is already present) is a no-op.
_make_deepseek_v4_weights_mapper: adds .base_layer.-namespace variants of the w[123].scale → weight_scale/weight_scale_inv regexes for LoRA-wrapped experts.
DeepseekV4ForCausalLM: adds SupportsLoRA, packed_modules_mapping (gate_up_proj, fused_wqa_wkv, compressor.fused_wkv_wgate), and lora_skip_prefixes = ["mtp."].

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Add LoRA support for DeepSeek V4 (DSV4). DSV4 is a MoE model with quantized (fp8/mxfp4) experts, MLA, and an MTP draft head. Three issues blocked merge=False (live-adapter) LoRA, causing train/inference inconsistency: 1. **Non-LoRA params on a LoRA-wrapped module** (e.g. `gate.tid2eid`, `gate.e_score_correction_bias`) live at `<head>.base_layer.<leaf>` in the live namespace, but the checkpoint name is plain. The DSV4 `load_weights` strict `params_dict[name]` lookup KeyError'd / orphaned these on the initial profile load. 2. **Expert mapping `param_name` vs `weight_name` prefix confusion.** `RoutedExperts.make_expert_params_mapping` applied a single `lora_base_layer_prefix` to both sides. But `get_expert_mapping` resolves `param_name` via `getattr` against this layer's bare `w13_weight`/`w2_weight` (no prefix), while `make_expert_params_mapping` indexes the model-wide `params_dict` (prefix included). One prefix for both sides silently dropped per-expert weights. 3. **`weight_scale_inv` / scale not forwarded through the LoRA wrapper.** Custom kernels read `weight_scale_inv` through the wrapper; without `__getattr__` forwarding the attribute lookup failed. This PR adds `SupportsLoRA` to `DeepseekV4ForCausalLM`, declares `packed_modules_mapping` and `lora_skip_prefixes` (the MTP draft head is not LoRA-adapted), adds `.base_layer.`-namespace variants to the weights mapper scale regexes, reconciles non-LoRA params in the inner `load_weights`, splits the expert mapping prefix into checkpoint-side (`weight_name`) and live-side (`param_name`), and forwards public attribute misses in `BaseLayerWithLoRA` to `base_layer`. Signed-off-by: Hollow Man <hollowman@opensuse.org>

Signed-off-by: Hollow Man <hollowman@opensuse.org>

jeejeelee · 2026-08-21T08:41:38Z



 class BaseLayerWithLoRA(nn.Module):
+    def __getattr__(self, name):


QQ： could you please explain this in detail?

This transparently forwards attribute misses to the wrapped base_layer. When LoRA is applied, vLLM wraps the original layer in a LoRA shim (FusedMoEWithLoRA/BaseLayerWithLoRA) and demotes the original into a .base_layer. submodule. The weight loader resolves params via recursive getattr (AutoWeightsLoader). Without getattr, getattr(shim, "weight") raises AttributeError, So getattr makes shim.weight resolve to shim.base_layer.weight.

Mechanically: it checks own _parameters/_buffers/modules first, then for public names forwards to base_layer; private (-prefixed) names stay local.

jeejeelee · 2026-08-21T08:46:19Z

        num_redundant_experts: int = 0,
        routed_experts_prefix: str = "routed_experts",
        lora_base_layer_prefix: str = "",
+        lora_base_layer_prefix_on_param_name: str = "",


do we need this argument?

They must be independent because the two resolution paths use different namespaces: get_expert_mapping resolves param_name via getattr against bare w13_weight/w2_weight (no prefix), while make_expert_params_mapping indexes the model-wide params_dict (prefixed). One shared arg would either drop the ckpt prefix (Fp8 experts not found) or add a unneeded prefix to the getattr path (attribute not found).

HollowMan6 · 2026-08-22T07:42:45Z

Sorry, get this closed accidentally, just reopened at #53361

HollowMan6 requested review from jeejeelee, mgoin, pavanimajety and zyongye as code owners August 19, 2026 17:44

Copilot AI lite review requested due to automatic review settings August 19, 2026 17:44

claude Bot reviewed Aug 19, 2026

View reviewed changes

Copilot AI reviewed Aug 19, 2026

mergify Bot added the deepseek Related to DeepSeek models label Aug 19, 2026

HollowMan6 mentioned this pull request Aug 19, 2026

[vllm] feat: Support LoRA for DeepSeek V4 & GLM5.3 Flash verl-project/verl#7483

Open

8 tasks

HollowMan6 force-pushed the dsv4_lora branch from 0e90469 to 30a8e4c Compare August 19, 2026 20:33

mergify Bot added the DSv4 label Aug 19, 2026

additional base_layer check

56c9afb

Signed-off-by: Hollow Man <hollowman@opensuse.org>

jeejeelee reviewed Aug 21, 2026

View reviewed changes

HollowMan6 closed this by deleting the head repository Aug 22, 2026

Navigation Menu

Uh oh!

Uh oh!

[LoRA] feat: Support LoRA for DeepSeek V4 - #52986

[LoRA] feat: Support LoRA for DeepSeek V4#52986
HollowMan6 wants to merge 2 commits into
vllm-project:mainfrom
HollowMan6:dsv4_lora

HollowMan6 commented Aug 19, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

jeejeelee Aug 21, 2026

Uh oh!

HollowMan6 Aug 21, 2026

Uh oh!

jeejeelee Aug 21, 2026

Uh oh!

HollowMan6 Aug 21, 2026

Uh oh!

HollowMan6 commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		class BaseLayerWithLoRA(nn.Module):
		def __getattr__(self, name):

Uh oh!

Uh oh!

Conversation

HollowMan6 commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Details

BaseLayerWithLoRA.__getattr__ (vllm/lora/layers/base.py)

RoutedExperts dual-prefix mapping (vllm/model_executor/layers/fused_moe/routed_experts.py)

DeepseekV4Model.load_weights reconcile (vllm/models/deepseek_v4/nvidia/model.py)

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

jeejeelee Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

HollowMan6 Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

jeejeelee Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

HollowMan6 Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

HollowMan6 commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HollowMan6 commented Aug 19, 2026 •

edited

Loading

`BaseLayerWithLoRA.getattr` (`vllm/lora/layers/base.py`)

`RoutedExperts` dual-prefix mapping (`vllm/model_executor/layers/fused_moe/routed_experts.py`)

`DeepseekV4Model.load_weights` reconcile (`vllm/models/deepseek_v4/nvidia/model.py`)