Fix MoE backend selection for LoRA (unquantized MoE) by danisereb · Pull Request #40273 · vllm-project/vllm

danisereb · 2026-04-19T06:51:47Z

Purpose

When using LoRA adapters with Nemotron Nano BF16:
https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

The following error was raised:

Using FlashInfer CUTLASS Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].
...
File "/my_home/workspace/my_vllm/vllm/lora/layers/fused_moe.py", line 164, in _inject_lora_into_fused_moe
assert isinstance(m_fused_moe_fn.impl.fused_experts, TritonExperts)

In previous vLLM versions the default backend for unquantized MoE was TritonExperts.
The new default backend "FlashInfer CUTLASS" does not support LoRA (see class FlashInferExperts, function moe_sum).

This PR selects TritonExperts when LoRA is enabled.

This change aligns with select_fp8_moe_backend, select_mxfp8_moe_backend, select_gpt_oss_mxfp4_moe_backend.

Test Plan

Add new test(s) in pytest tests/kernels/moe/test_unquantized_backend_selection.py

Check that LoRA works with Nemotron Nano BF16.

Test Result

All tests in test_unquantized_backend_selection.py passed.

LoRA adapters now work with Nemotron Nano BF16 (TP1/2/4):

Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON'].

When running without LoRA adapters, the expected default backend is selected:

Using FlashInfer CUTLASS Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

gemini-code-assist

Code Review

This pull request introduces logic to force the Triton backend for unquantized MoE when LoRA is enabled and adds a corresponding test case. Feedback indicates that the current early return implementation is problematic because it bypasses support for BATCHED_TRITON (required for models like DeepSeek-V3), skips backend selection logging, and overrides user-specified backend preferences. A suggestion was provided to filter the available backends instead of returning early.

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Should keep ROCm behavior unchanged. Also update tests. Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

tomeras91

Thanks @danisereb!
Added a few suggestions

- Use Triton for both CUDA and ROCm (aligned with select_fp8_moe_backend). - Update tests accordingly. Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

danisereb · 2026-04-19T14:34:38Z

    if current_platform.is_out_of_tree():
        return UnquantizedMoeBackend.OOT, None

+    if moe_config.is_lora_enabled:


This logic is now aligned with select_fp8_moe_backend (early exit if LoRA is enabled).

tomeras91

Much better now.
Left a small nit

tomeras91 · 2026-04-19T15:40:09Z

+
+
+@skipif_not_cuda_rocm
+def test_select_explicit_triton_ignores_flashinfer_env(monkeypatch):


nit: This test can run on all platforms.. nothing CUDA/ROCm specific about it

Wasn't sure about XPUs.

…oject#40273)" This reverts commit d1135a5.

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com> Signed-off-by: Avinash Singh <avinashsingh.rcoem@gmail.com>

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com> Signed-off-by: Adrian <info@zzit.ch>

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

gemini-code-assist Bot reviewed Apr 19, 2026

View reviewed changes

Comment thread vllm/model_executor/layers/fused_moe/oracle/unquantized.py

danisereb force-pushed the fix_lora_moe_backend branch from 9056200 to 0168ea9 Compare April 19, 2026 07:48

danisereb changed the title ~~Fix backend selection for LoRA (unquantized MoE)~~ Fix MoE backend selection for LoRA (unquantized MoE) Apr 19, 2026

Fix backend selection for LoRA (unquantized MoE)

d57b541

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

danisereb force-pushed the fix_lora_moe_backend branch from 4f5f7fe to d57b541 Compare April 19, 2026 08:19

netanel-haber reviewed Apr 19, 2026

View reviewed changes

Comment thread tests/kernels/moe/test_unquantized_backend_selection.py Outdated

Cleanup tests

f8001c9

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

amitz-nv reviewed Apr 19, 2026

View reviewed changes

Comment thread vllm/model_executor/layers/fused_moe/oracle/unquantized.py Outdated

amitz-nv reviewed Apr 19, 2026

View reviewed changes

Comment thread vllm/model_executor/layers/fused_moe/oracle/unquantized.py Outdated

danisereb marked this pull request as ready for review April 19, 2026 09:36

danisereb requested review from WoosukKwon, mgoin, pavanimajety, tlrmchlsmth and yewentao256 as code owners April 19, 2026 09:36

claude Bot reviewed Apr 19, 2026

View reviewed changes

Remove BATCHED_TRITON as option for LoRA

4c0c318

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

danisereb force-pushed the fix_lora_moe_backend branch from 51f70ad to 4c0c318 Compare April 19, 2026 10:13

Apply the changes only for CUDA.

cdf3444

Should keep ROCm behavior unchanged. Also update tests. Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

danisereb force-pushed the fix_lora_moe_backend branch from 2c410cb to cdf3444 Compare April 19, 2026 11:15

danisereb added 2 commits April 19, 2026 14:26

Add more tests

af6c03e

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Refactor tests

077cede

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

tomeras91 requested changes Apr 19, 2026

View reviewed changes

Comment thread tests/kernels/moe/test_unquantized_backend_selection.py Outdated

Comment thread tests/kernels/moe/test_unquantized_backend_selection.py Outdated

Comment thread vllm/model_executor/layers/fused_moe/oracle/unquantized.py Outdated

Code review changes.

6b852cd

- Use Triton for both CUDA and ROCm (aligned with select_fp8_moe_backend). - Update tests accordingly. Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

danisereb force-pushed the fix_lora_moe_backend branch from 82f0ef3 to 6b852cd Compare April 19, 2026 14:25

danisereb added 2 commits April 19, 2026 17:27

Remove cuda/rocm check to align with select_fp8_moe_backend

ff7e521

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Fix skipif var name

8012f92

Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

danisereb commented Apr 19, 2026

View reviewed changes

tomeras91 approved these changes Apr 19, 2026

View reviewed changes

tomeras91 added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 19, 2026

robertgshaw2-redhat approved these changes Apr 19, 2026

View reviewed changes

robertgshaw2-redhat enabled auto-merge (squash) April 19, 2026 15:45

robertgshaw2-redhat merged commit d1135a5 into vllm-project:main Apr 19, 2026
68 checks passed

vllm-agent pushed a commit to vllm-agent/vllm that referenced this pull request Apr 20, 2026

Revert "Fix MoE backend selection for LoRA (unquantized MoE) (vllm-pr…

c685fd5

…oject#40273)" This reverts commit d1135a5.

vllm-agent mentioned this pull request Apr 20, 2026

Revert "Fix MoE backend selection for LoRA (unquantized MoE)" (#40273) #40313

Draft

bnellnm pushed a commit to neuralmagic/vllm that referenced this pull request Apr 20, 2026

Fix MoE backend selection for LoRA (unquantized MoE) (vllm-project#40273

22f50df

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

baonudesifeizhai pushed a commit to baonudesifeizhai/vllm that referenced this pull request Apr 23, 2026

Fix MoE backend selection for LoRA (unquantized MoE) (vllm-project#40273

57c997b

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Lafunamor pushed a commit to Lafunamor/vllm that referenced this pull request May 1, 2026

Fix MoE backend selection for LoRA (unquantized MoE) (vllm-project#40273

96435b9

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com> Signed-off-by: Adrian <info@zzit.ch>

mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026

Fix MoE backend selection for LoRA (unquantized MoE) (vllm-project#40273

1693864

) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix MoE backend selection for LoRA (unquantized MoE)#40273

Fix MoE backend selection for LoRA (unquantized MoE)#40273
robertgshaw2-redhat merged 9 commits intovllm-project:mainfrom
de-inf:fix_lora_moe_backend

danisereb commented Apr 19, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

claude Bot left a comment

Uh oh!

tomeras91 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danisereb Apr 19, 2026 •

edited

Loading

Uh oh!

tomeras91 left a comment

Uh oh!

tomeras91 Apr 19, 2026

Uh oh!

danisereb Apr 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants



		@skipif_not_cuda_rocm
		def test_select_explicit_triton_ignores_flashinfer_env(monkeypatch):

Uh oh!

Conversation

danisereb commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

tomeras91 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danisereb Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomeras91 left a comment

Choose a reason for hiding this comment

Uh oh!

tomeras91 Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

danisereb Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

danisereb commented Apr 19, 2026 •

edited

Loading

danisereb Apr 19, 2026 •

edited

Loading