Fix/qwen3 vl text lora noop

[VBS2004]

​

Purpose

​
Fixes #48019.
​
Text PEFT LoRAs silently no-op on Qwen3-VL wrapper models. Qwen3VLForConditionalGeneration mounts its language model at self.language_model, so real text module paths are language_model.model.layers.<N>.... Its hf_to_vllm_mapper had no bare model. prefix rule, so a text LoRA key like base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight is stripped to model.layers.0.self_attn.q_proj, matches no prefix, and resolves to a module that does not exist. The adapter loads but binds to nothing and behaves identically to the base model — no error, no warning.
​
Fix: append one prefix rule, last, to the mapper:
​

"model.": "language_model.model.",

​
WeightsMapper._map_name applies prefix rules sequentially (no break on first match), so the existing model.visual. and model.language_model. rules still fire first and base-weight loading is unaffected; only bare model.<x> text-LoRA keys fall through to the new rule.
​

Test Plan

​
Static check over the model's real hf_to_vllm_mapper, running representative text PEFT LoRA keys through vLLM's parse_fine_tuned_lora_name prefix-mapping path and checking whether the resolved module name lands under the wrapper's real language_model. subtree. This isolates the mapping logic (no GPU/weights required).
​
Also verified there is no regression on base-weight loading: all base checkpoint keys (model.language_model.*, model.visual.*, lm_head.*) map identically before and after the change.
​
Text-LoRA keys checked (q/k/v/o_proj, gate/up/down_proj):
​

base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight
base_model.model.model.layers.0.mlp.gate_proj.lora_A.weight
... etc.

​

Test Result

​
Before — 0/8 text-LoRA modules resolve under language_model. (silent no-op):
​

model.layers.0.self_attn.q_proj                 -> not found
model.layers.0.mlp.gate_proj                    -> not found

​
After — 8/8 resolve to the real module path:
​

language_model.model.layers.0.self_attn.q_proj  -> binds
language_model.model.layers.0.mlp.gate_proj     -> binds

​
Base-weight mapping regression check — identical before/after:
​

model.language_model.layers.0.self_attn.q_proj.weight -> language_model.model.layers.0.self_attn.q_proj.weight
model.visual.blocks.0.attn.qkv.weight                 -> visual.blocks.0.attn.qkv.weight
lm_head.weight                                        -> language_model.lm_head.weight

​
Follow-up (separate PR): add a warning when a LoRA is added but zero module names match any real module, so future silent no-ops surface loudly.
​

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[VBS2004]

Friendly ping for review — this is a minimal one-line fix for #48019 (text LoRA silently no-ops on Qwen3-VL wrapper models). It adds a single model. → language_model.model. prefix rule to hf_to_vllm_mapper.

I verified statically that text-LoRA modules go from 0/8 → 8/8 binding, with all base-weight mappings unchanged (no regression). Since pre-run-check requires a maintainer label for first-time contributors, could a code owner add the ready label to trigger CI? Happy to add an e2e logprob comparison on GPU if useful. cc @LiquidGunay (issue author)

[ErenAta16]

Verified this against the same pattern already used in qwen2_vl.py (checked the current source directly): it also carries a bare "model.": "language_model.model." rule as the last entry in orig_to_new_prefix, after the more specific model.language_model. and model.visual. rules. Since WeightsMapper._map_name walks the dict in order and does not break after a match, putting the catch-all last is exactly what keeps the vision/projector/lm_head keys routed by their specific rules while only unmatched bare model.<x> keys (i.e. the text-LoRA case) fall through to it. This PR applies that identical, already-proven shape to qwen3_vl.py, and the base-weight before/after comparison in the description confirms no existing mapping changes.

The fix is minimal and low-risk (one added line, ordering preserved, no changes to any other rule), and the reasoning in the test plan matches the actual mapper semantics rather than just describing the symptom. I don't see a downside to landing this on its own ahead of the broader multi-model follow-up discussed in #48019.

Approving.

[VBS2004]

Since this is a LoRA-mapping fix, @jeejeelee (LoRA code owner) — would appreciate your review when you have a moment. The change mirrors the existing bare model. catch-all already in qwen2_vl.py.

[ErenAta16]

Approval still stands. Since this has been waiting on the label gate for two weeks, I went looking for how widespread the pattern is, in case scope helps it get prioritized.

I took each model's real orig_to_new_prefix off current main and ran a representative text-LoRA key (model.layers.0.self_attn.q_proj, i.e. post-PEFT-prefix-stripping) through the same sequential prefix substitution WeightsMapper._map_name performs, checking whether it lands under language_model.:

model	SupportsLoRA	text-LoRA key resolves to	ok
qwen2_vl	yes	language_model.model.layers.0.self_attn.q_proj	yes
gemma4_mm	yes	language_model.model.layers.0.self_attn.q_proj	yes
qwen3_vl	yes	model.layers.0.self_attn.q_proj	no (this PR)
gemma3_mm	yes	model.layers.0.self_attn.q_proj	no
llava	yes	model.layers.0.self_attn.q_proj	no
llava_next_video	yes	model.layers.0.self_attn.q_proj	no
paligemma	yes	model.layers.0.self_attn.q_proj	no
pixtral	yes	model.layers.0.self_attn.q_proj	no
interns1	yes	model.layers.0.self_attn.q_proj	no
lfm2_vl	yes	model.layers.0.self_attn.q_proj	no
glm4_1v	yes	model.layers.0.self_attn.q_proj	no
granite4_vision	yes	model.layers.0.self_attn.q_proj	no
minicpmv4_6	yes	model.layers.0.self_attn.q_proj	no
cheers	yes	model.layers.0.self_attn.q_proj	no
gemma4	yes	model.layers.0.self_attn.q_proj	no
qwen3_asr	yes	model.layers.0.self_attn.q_proj	no

All of these declare SupportsLoRA, all mount the language model at self.language_model (checked), and all carry the specific model.language_model. rule while lacking any bare model. catch-all. qwen2_vl and gemma4_mm are the only two that have one, and gemma4_mm uses "model" without the trailing dot rather than "model.".

Confirmed the LoRA path really does run through this mapper: vllm/lora/worker_manager.py:135-151 pulls hf_to_vllm_mapper off the model and passes it down to vllm/lora/utils.py:177, which calls _map_name(name).

Two caveats I want to be explicit about, since this is static analysis and I have no GPU to confirm end to end. First, it proves the prefix mapping doesn't resolve, not that every one of these silently no-ops, a model could compensate elsewhere in its LoRA module resolution. Second, some of these may simply not have text LoRAs published for them, making it theoretical.

That said, the shape is identical to the bug this PR fixes, and the failure mode is the bad kind: the adapter loads, no error, no warning, output matches base. If even a few of these are genuinely affected, users are running LoRAs that do nothing and have no way to tell.

@jeejeelee this might be worth the label on its own merits, but the broader question is whether a bare catch-all belongs in a shared place rather than being added per model as each one is discovered. Something in WeightsMapper or a shared helper for wrapper models would stop this recurring, and would also settle the "model." versus "model" inconsistency between the two models that do have it.

Happy to file this as a separate issue with the table if that's more useful than a comment buried in a PR thread. Not planning to open a PR against the other models myself, that needs someone who can actually run a text LoRA on each.

[linitra24]

@VBS2004 Hi. I’m still unclear about how the following LoRA key would be produced for Qwen3-VL:

base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight

When I train a standard PEFT LoRA using Qwen3VLForConditionalGeneration, the generated keys look like:

base_model.model.model.language_model.layers.0.self_attn.q_proj.lora_A.weight

These keys can already be mapped by the existing model.language_model. rule, so I cannot reproduce the issue described here.

Could you share the exact training model class or script that produces the base_model.model.model.layers.* prefix for Qwen3-VL?

[ErenAta16]

@linitra24 Good question, and the answer is that both key shapes are real, they just come from training against different classes. Your observation is correct and so is the bug report, they're describing different adapters.

Training against Qwen3VLForConditionalGeneration (what you did): the module lives at model.language_model.layers.0... inside the HF wrapper, PEFT prefixes base_model.model., so you get base_model.model.model.language_model.layers.0.... Strips to model.language_model.layers.0..., matches the existing rule. Works today, nothing to fix.

Training against the text CausalLM path (what the bug is about): the LM is loaded on its own, so the module is just model.layers.0... with no language_model segment. PEFT gives base_model.model.model.layers.0..., which strips to model.layers.0... and matches nothing.

@LiquidGunay said as much in #48019, it's easy to miss in the environment block:

Adapter type: PEFT LoRA trained against the text CausalLM path

along with --language-model-only. So the scenario is a text-only LoRA, trained on the underlying LLM, then served on the VL wrapper. That's a normal thing to want: one text adapter reused across the plain LM and the VL model, without retraining through the multimodal wrapper.

The evidence in #48019 is also empirical rather than just a mapping argument, which I'd trust over any static analysis including my own:

Before the fix:
  mean |vLLM adapter - vLLM base logprob delta| ~= 0.00030    <- adapter does nothing
  mean |HF adapter - HF base delta|            ~= 0.29197    <- same adapter works in HF
  HF-vLLM delta Pearson                        ~= 0.0308

After the fix:
  mean |vLLM adapter - vLLM base logprob delta| ~= 0.29139
  HF-vLLM adapter logprob MAE                  ~= 0.00973
  HF-vLLM delta Pearson                        ~= 0.99923

A delta of 0.0003 is the adapter having no measurable effect, and Pearson going 0.03 → 0.999 against HF is about as clean a confirmation as you get.

To reproduce, train against the text path rather than the VL class:

from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("<the underlying text model>")
# attach PEFT LoRA here, save

then serve the VL model with language_model_only=True and apply that adapter. Inspecting the saved adapter_model.safetensors keys is enough to see it without running generation: text-path training gives base_model.model.model.layers.*, wrapper training gives base_model.model.model.language_model.layers.*.

Worth adding to the PR description, since you almost certainly aren't the last person to try reproducing it with a wrapper-trained adapter and conclude it isn't real.

Separately: the silent-failure aspect is arguably the more serious half. LiquidGunay's first suggested fix in #48019, warning when a LoRA loads but matches zero modules, would have turned this into a five-minute diagnosis instead of a logprob comparison against HF. Neither this PR nor #49464 does that, and it'd be valuable independently of the mapping fix.

[linitra24]

@ErenAta16 Thanks for the explanation, but I still cannot reproduce this path.

For the official Qwen/Qwen3-VL-2B-Instruct checkpoint, the following fails because Qwen3VLConfig is not supported by AutoModelForCausalLM:

AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-VL-2B-Instruct")

Using Qwen3VLForConditionalGeneration with PEFT produces keys like:

base_model.model.model.language_model.layers.*

which are already handled by the existing model.language_model. mapping.

Could you provide a minimal reproducible example, including the exact Transformers/PEFT versions, model class, loading code, and training script that produce:

base_model.model.model.layers.*

for Qwen3-VL?

[VBS2004]

Superseded by #49464, which applies the same fix in the shared LoRA-mapper path (with a destination guard) and covers Qwen3-VL plus the other affected wrappers. Keeping this open only until #49464 lands; happy to close now if maintainers prefer. Original diagnosis / repro stays here for reference.