[LoRA][Nemotron] Add LoRA support for Nemotron VL models (for the language model only) by danisereb · Pull Request #56231 · vllm-project/vllm

danisereb · 2026-09-10T08:03:49Z

Purpose

At this time Nemotron VL models fail if used with LoRA adapters:

ValueError: NemotronH_Nano_VL_V2 does not support LoRA yet.

This PR enables LoRA on the language model of Nemotron VL models.

Both text and media embeddings flow through the language model after the multimodal merge, adapters apply to media tokens too.
What's not covered is LoRA on the encoders and connectors themselves (RADIO, mlp1, Parakeet), which vLLM gates behind the supports_tower_connector_lora opt-in that this model does not declare.

From docs/models/supported_models.md:

!!! note
    vLLM currently supports adding LoRA adapters to the language backbone for most multimodal models. Additionally, vLLM now experimentally supports adding LoRA to the tower and connector modules for some multimodal models. See [this page](../features/lora.md).

Note: AI assistance was used in this PR.

Test Plan

Verify correctness using evals (gsm8k) with Nemotron VL model (with/without LoRA adapter).

Test Result

GSM8K results for Nemotron-3-Nano-Omni-30B-A3B:

vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
--served-model-name my_model --host 0.0.0.0 --port 8000 \
--trust-remote-code --tensor-parallel-size 4 --enable-expert-parallel \
--enable-lora --max-lora-rank 8 --max-loras 1 \
--lora-modules '{"name": "lora_model", "path": "/my_home/hf_models/oridos/nemotron-tinystories-lora/", "base_model_name": "my_model"}'

# vLLM main (nightly-2a02f6efe319c885e3ccbcecde402e0028f9ec1e), base model

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8203|±  |0.0106|
|     |       |strict-match    |     5|exact_match|↑  |0.8415|±  |0.0101|

# This PR, base model

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8241|±  |0.0105|
|     |       |strict-match    |     5|exact_match|↑  |0.8423|±  |0.0100|

# This PR,
# with LoRA adapter:
# https://huggingface.co/oridos/nemotron-tinystories-lora

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8256|±  |0.0105|
|     |       |strict-match    |     5|exact_match|↑  |0.8218|±  |0.0105|

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

qthequartermasterman

This is timely. I was just looking this afternoon what would be needed for LoRA support for Nemotron 3 Omni. This PR looks good to me.

@jeejeelee @DarkLight1337

linitra24

Overall, looks good to me. I also noticed that this model is not listed in docs/models/supported_models.md. Should we add it there as well?

linitra24 · 2026-09-11T02:17:43Z

+        # the LoRA delta is undefined for tokens with no mapping entry.
+        embed_tokens = self.get_language_model().model.embed_tokens
+        if isinstance(embed_tokens, BaseLayerWithLoRA):
+            embed_tokens = embed_tokens.base_layer


QQ: If embed_tokens is adapted during training, wouldn’t the internally generated text/special tokens that are not replaced by video embeddings also receive the LoRA delta?

I had to dig deeper to give a proper answer (with some help from Claude for the vision/video code), and you're correct.

Video marker tokens - the per-frame separator text (e.g. Frame 1 sampled at 0.50 seconds:) and the <img> / </img> tags wrapping each frame do not get the embedding LoRA delta.

They're generated inside the encoder by _create_final_video_embeddings, so they never appear in the scheduled token batch that vLLM builds the per-token LoRA adapter index mapping from - and there can be more of them than max_num_batched_tokens, which is what that mapping is sized for.

Without this PR, that path raises an error, so video + LoRA doesn't work at all today.

This issue is unlikely to be a problem in practice:

peft with target_modules="all-linear" does not include nn.Embedding. you'd have to name embed_tokens explicitly. The public adapter I tested (https://huggingface.co/oridos/nemotron-tinystories-lora) has no embedding weights. I am not sure if LoRA weights for the embedding is common / popular.

With no embedding LoRA, this change is a no-op (the delta would be zero).

For basic enablement of LoRA for Nemotron VL models, I think this change is OK.

@linitra24 tell me if something here doesn't align with your understanding of the flow.

@jeejeelee is there any code change you think would help?
Raising an error if embedding LoRA is detected doesn't sound like a good idea - no model in vLLM rejects adapters with embedding weights today, so Nemotron VL would become uniquely strict.

Note:
Similar flow (and likely limitation) seems to exist for Qwen3-VL (see function _create_final_video_embeddings in file vllm/model_executor/models/qwen3_vl.py).

jeejeelee · 2026-09-12T04:54:32Z

please respond to @linitra24's comment

danisereb · 2026-09-14T12:47:44Z

@linitra24 I added the Nemotron-VL models to supported_models

mergify · 2026-09-14T13:05:56Z

Documentation preview: https://vllm--56231.org.readthedocs.build/en/56231/

jeejeelee · 2026-09-15T14:42:37Z

/ci run

linitra24

LGTM, thanks!

github-actions · 2026-09-15T14:43:30Z

❌ This PR is 282 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

…orted_models Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

danisereb · 2026-09-15T14:52:21Z

Rebased on main to allow CI run.

@jeejeelee can you re-enable auto-merge ?

benchislett · 2026-09-15T15:04:56Z

/ci run

github-actions · 2026-09-15T15:06:54Z

❌ This PR is 3 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

danisereb · 2026-09-15T15:15:40Z

/ci run

github-actions · 2026-09-15T15:16:00Z

✅ Triggered Buildkite CI #89118 for commit 7dfdd854a813.

qthequartermasterman · 2026-09-21T14:05:40Z

@danisereb Do you plan on adding LoRA support to the vision/audio encoders as well at some point?

linitra24 · 2026-09-21T14:14:23Z

@danisereb Do you plan on adding LoRA support to the vision/audio encoders as well at some point?

If needed, I'd be happy to take this on.

danisereb · 2026-09-28T07:09:17Z

@qthequartermasterman
I do not plan to add LoRA support for vision/audio at this time.

@linitra24
It would be great if you take this!

qthequartermasterman · 2026-09-28T13:49:54Z

@danisereb @linitra24 My colleague @Nan2018 actually has just opened a PR for vision/audio support: #58658.

danisereb marked this pull request as ready for review September 10, 2026 09:16

danisereb requested a review from tomeras91 as a code owner September 10, 2026 09:16

claude Bot reviewed Sep 10, 2026

View reviewed changes

danisereb changed the title ~~Add LoRA support for Nemotron VL models (language model only)~~ Add LoRA support for Nemotron VL models (for the language model only) Sep 10, 2026

qthequartermasterman approved these changes Sep 10, 2026

View reviewed changes

linitra24 reviewed Sep 11, 2026

View reviewed changes

mergify Bot added the documentation Improvements or additions to documentation label Sep 14, 2026

jeejeelee approved these changes Sep 15, 2026

View reviewed changes

jeejeelee enabled auto-merge (squash) September 15, 2026 14:42

linitra24 approved these changes Sep 15, 2026

View reviewed changes

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 15, 2026

danisereb added 4 commits September 15, 2026 17:51

Add LoRA support for NemotronH_Nano_VL_V2 (language model only)

00e2ed4

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Add handling for LoRA with vision-language models

a59a35f

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Change the fix to be nemotron specific

d0b4040

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Add NemotronH_Nano_Omni_Reasoning_V3 and NemotronH_Nano_VL_V2 to supp…

12be5bc

…orted_models Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

auto-merge was automatically disabled September 15, 2026 14:52
Head branch was pushed to by a user without write access

danisereb force-pushed the nemo_vl_lora branch from 6382e5b to 12be5bc Compare September 15, 2026 14:52

danisereb changed the title ~~Add LoRA support for Nemotron VL models (for the language model only)~~ [Nemotron][LoRA] Add LoRA support for Nemotron VL models (for the language model only) Sep 15, 2026

danisereb changed the title ~~[Nemotron][LoRA] Add LoRA support for Nemotron VL models (for the language model only)~~ [LoRA][Nemotron] Add LoRA support for Nemotron VL models (for the language model only) Sep 15, 2026

benchislett enabled auto-merge (squash) September 15, 2026 15:04

Merge branch 'main' into nemo_vl_lora

7dfdd85

vllm-bot merged commit a31ec3a into vllm-project:main Sep 16, 2026
122 of 125 checks passed

danisereb deleted the nemo_vl_lora branch September 28, 2026 07:07

qthequartermasterman mentioned this pull request Sep 28, 2026

[Model][LoRA] Add tower/connector LoRA support for Nemotron Nano/Omni VL (vision + audio) #58658

Open

biswapanda mentioned this pull request Sep 28, 2026

fix(vllm): backport Nemotron changes for vLLM0.29 compatibility PrimeIntellect-ai/prime-rl#3707

Closed

Conversation

danisereb commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

qthequartermasterman left a comment

Choose a reason for hiding this comment

Uh oh!

linitra24 left a comment

Choose a reason for hiding this comment

Uh oh!

linitra24 Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

danisereb Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeejeelee commented Sep 12, 2026

Uh oh!

danisereb commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Sep 14, 2026

Uh oh!

jeejeelee commented Sep 15, 2026

Uh oh!

linitra24 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

danisereb commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benchislett commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

danisereb commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

Uh oh!

qthequartermasterman commented Sep 21, 2026

Uh oh!

linitra24 commented Sep 21, 2026

Uh oh!

danisereb commented Sep 28, 2026

Uh oh!

qthequartermasterman commented Sep 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

danisereb commented Sep 10, 2026 •

edited

Loading

danisereb Sep 14, 2026 •

edited

Loading

danisereb commented Sep 14, 2026 •

edited

Loading

danisereb commented Sep 15, 2026 •

edited

Loading