[LoRA] feat: Support LoRA for DeepSeek V4 - #53361
Merged
Merged
Conversation
Add LoRA support for DeepSeek V4 (DSV4). DSV4 is a MoE model with quantized (fp8/mxfp4) experts, MLA, and an MTP draft head. Three issues blocked merge=False (live-adapter) LoRA, causing train/inference inconsistency: 1. **Non-LoRA params on a LoRA-wrapped module** (e.g. `gate.tid2eid`, `gate.e_score_correction_bias`) live at `<head>.base_layer.<leaf>` in the live namespace, but the checkpoint name is plain. The DSV4 `load_weights` strict `params_dict[name]` lookup KeyError'd / orphaned these on the initial profile load. 2. **Expert mapping `param_name` vs `weight_name` prefix confusion.** `RoutedExperts.make_expert_params_mapping` applied a single `lora_base_layer_prefix` to both sides. But `get_expert_mapping` resolves `param_name` via `getattr` against this layer's bare `w13_weight`/`w2_weight` (no prefix), while `make_expert_params_mapping` indexes the model-wide `params_dict` (prefix included). One prefix for both sides silently dropped per-expert weights. 3. **`weight_scale_inv` / scale not forwarded through the LoRA wrapper.** Custom kernels read `weight_scale_inv` through the wrapper; without `__getattr__` forwarding the attribute lookup failed. This PR adds `SupportsLoRA` to `DeepseekV4ForCausalLM`, declares `packed_modules_mapping` and `lora_skip_prefixes` (the MTP draft head is not LoRA-adapted), adds `.base_layer.`-namespace variants to the weights mapper scale regexes, reconciles non-LoRA params in the inner `load_weights`, splits the expert mapping prefix into checkpoint-side (`weight_name`) and live-side (`param_name`), and forwards public attribute misses in `BaseLayerWithLoRA` to `base_layer`. Signed-off-by: Hollow Man <hollowman@opensuse.org>
Signed-off-by: Hollow Man <hollowman@opensuse.org>
HollowMan6
requested review from
jeejeelee,
mgoin,
pavanimajety and
zyongye
as code owners
August 22, 2026 07:42
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
4 tasks
8 tasks
jeejeelee
approved these changes
Aug 24, 2026
Member
|
/ci run |
jeejeelee
enabled auto-merge (squash)
August 24, 2026 02:46
|
✅ Triggered Buildkite CI #85274 for commit |
github-actions
Bot
added
the
ready
khushali9
pushed a commit
to khushali9/vllm
that referenced
this pull request
Aug 29, 2026
Signed-off-by: Hollow Man <hollowman@opensuse.org> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: khushali9 <khushali.desai9@gmail.com>
am-cohere
pushed a commit
to am-cohere/vllm
that referenced
this pull request
Sep 1, 2026
Signed-off-by: Hollow Man <hollowman@opensuse.org> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
mikeshawcode
pushed a commit
to mikeshawcode/vllm
that referenced
this pull request
Sep 1, 2026
Signed-off-by: Hollow Man <hollowman@opensuse.org> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: mikeshawcode <michaelwshaw2@gmail.com>
mikeshawcode
pushed a commit
to mikeshawcode/vllm
that referenced
this pull request
Sep 1, 2026
Signed-off-by: Hollow Man <hollowman@opensuse.org> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: mikeshawcode <michaelwshaw2@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add LoRA support for DeepSeek V4 (DSV4). DSV4 is a MoE model with quantized (fp8/mxfp4) experts and an MTP draft head. Three issues blocked merge=False (live-adapter) LoRA, causing train/inference inconsistency:
Non-LoRA params on a LoRA-wrapped module (e.g.
gate.tid2eid,gate.e_score_correction_bias) live at<head>.base_layer.<leaf>in the live namespace, but the checkpoint name is plain. The DSV4load_weightsstrictparams_dict[name]lookup KeyError'd / orphaned these on the initial profile load.Expert mapping
param_namevsweight_nameprefix confusion.RoutedExperts.make_expert_params_mappingapplied a singlelora_base_layer_prefixto both sides. Butget_expert_mappingresolvesparam_nameviagetattragainst this layer's barew13_weight/w2_weight(no prefix), whilemake_expert_params_mappingindexes the model-wideparams_dict(prefix included). One prefix for both sides silently dropped per-expert weights.weight_scale_inv/ scale not forwarded through the LoRA wrapper. Custom kernels readweight_scale_invthrough the wrapper; without__getattr__forwarding the attribute lookup failed.This PR adds
SupportsLoRAtoDeepseekV4ForCausalLM, declarespacked_modules_mappingandlora_skip_prefixes(the MTP draft head is not LoRA-adapted), adds.base_layer.-namespace variants to the weights mapper scale regexes, reconciles non-LoRA params in the innerload_weights, splits the expert mapping prefix into checkpoint-side (weight_name) and live-side (param_name), and forwards public attribute misses inBaseLayerWithLoRAtobase_layer.Test Plan
Test with verl E2E under verl-project/verl#7483
Test Result
DSV4 merge=False LoRA GRPO, 2-node (Megatron backend):
Before this PR, merge=False diverged (pearson ~0.43) due to dropped per-expert weights and orphaned non-LoRA params; after, it aligns with the merge=True baseline to within noise.
Details
BaseLayerWithLoRA.__getattr__(vllm/lora/layers/base.py)Forwards public attribute misses to
base_layer(e.g.weight_scale_invread by custom kernels through the wrapper). Private (_-prefixed) names are framework bookkeeping and stay local.RoutedExpertsdual-prefix mapping (vllm/model_executor/layers/fused_moe/routed_experts.py)build_expert_params_mappingnow passes thebase_layer.prefix to bothlora_base_layer_prefix(checkpointweight_nameside) andlora_base_layer_prefix_on_param_name(liveparam_nameside).make_expert_params_mappinggainslora_base_layer_prefix_on_param_nameand applies it to thew13/w2param names, whileweight_namekeeps the existinglora_base_layer_prefix. This matches the fact thatget_expert_mappingresolvesparam_nameviagetattron barew13_weight/w2_weight, whereasmake_expert_params_mappingindexes model-wideparams_dict(prefix included).DeepseekV4Model.load_weightsreconcile (vllm/models/deepseek_v4/nvidia/model.py)nameis missing fromparams_dict, try<head>.base_layer.<leaf>and use it if live. Covers non-LoRA params on a wrapped module during the initial checkpoint load (profile_run), guarded so weight-sync (where the suffix is already present) is a no-op._make_deepseek_v4_weights_mapper: adds.base_layer.-namespace variants of thew[123].scale→weight_scale/weight_scale_invregexes for LoRA-wrapped experts.DeepseekV4ForCausalLM: addsSupportsLoRA,packed_modules_mapping(gate_up_proj,fused_wqa_wkv,compressor.fused_wkv_wgate), andlora_skip_prefixes = ["mtp."].Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.