Support NemotronH_Omni_Reasoning_V3 in SGLang by rystewart-nvidia · Pull Request #35599 · sgl-project/sglang

rystewart-nvidia · 2026-08-20T00:58:00Z

Motivation

The new Nemotron checkpoint uses the NemotronH_Omni_Reasoning_V3 architecture, which SGLang does not currently recognize. It falls back to the generic Transformers implementation, and the embedded MTP and quantized vision weights do not load correctly.

Modifications

Register NemotronH_Omni_Reasoning_V3 as a multimodal model using the existing Nano Nemotron VL implementation.
Reuse the existing Nemotron-H runtime and Mamba configuration.
Unwrap the language model config for embedded MTP and handle the checkpoint's language_model. weight prefix.
Load and apply the vision projector's final LayerNorm.
Support the alternate RADIO weight layout used by the quantized checkpoints, including split Q/K/V weights.
Add tests

Most of this reuses the existing Nemotron-H and Nano Nemotron VL paths.

Accuracy Tests

Speed Tests and Profiling

This PR is mainly adding model support. Runtime tests included CUDA graphs and speculative decoding to make sure those paths work.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations. (No documentation change needed; this does not add a new user-facing option.)
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ⏳ Run #34506937330
Latest PR Test (Extra): ❌ Run #34506936523
Latest PR Test (AMD ROCm 10): ❌ Run #34506937053

rystewart-nvidia · 2026-08-20T23:40:14Z

/tag-and-rerun-ci

nvpohanh · 2026-08-21T13:12:29Z

/tag-and-rerun-ci

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

init_moe_gemm_config only looked for hf_config.text_config, so VLMs whose language sub-config is named llm_config (NemotronH_Omni_Reasoning_V3) fell back to the top-level config, found no MoE top-k attribute, and skipped initialize_moe_config entirely. get_moe_runner_backend() then returned auto instead of the configured backend, leaving NVFP4 MoE weights prepared for CUTLASS while the runner resolved to FlashInfer TRTLLM: NotImplementedError: Unsupported moe_runner_backend for NVFP4 MoE: MoeRunnerBackend.FLASHINFER_TRTLLM. Fall back to hf_text_config, which resolves llm_config / language_config / thinker_config and returns the top-level config when there is no sub-config. Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 64ff8e9)

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

rystewart-nvidia · 2026-08-28T14:35:05Z

/tag-and-rerun-ci extra

nvpohanh · 2026-08-31T05:39:16Z

@rystewart-nvidia could you fix the conflicts? thanks

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

nvpohanh · 2026-09-02T11:56:07Z

/rerun-failed-ci

Fridge003 · 2026-09-03T01:45:03Z

@codex review

chatgpt-codex-connector · 2026-09-03T01:45:17Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-03T01:51:35.945369Z	`0545f5c`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0545f5c4b0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: rystewart-nvidia <rystewart@nvidia.com>

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

nvpohanh · 2026-09-04T05:53:00Z

/rerun-failed-ci

nvpohanh · 2026-09-04T11:53:24Z

/rerun-failed-ci

elvischenv

@rystewart-nvidia could you fix the conflicts? thanks

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

nvpohanh · 2026-09-09T12:37:13Z

/rerun-failed-ci

Signed-off-by: Ryan Stewart <rystewart@nvidia.com> Signed-off-by: rystewart-nvidia <rystewart@nvidia.com> Co-authored-by: elvischenv <219235043+elvischenv@users.noreply.github.com> Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>

rystewart-nvidia marked this pull request as ready for review August 20, 2026 22:01

rystewart-nvidia requested review from HaiShaw, JustinTong0323, mickqian, yctseng0211, yhyang201 and yuan-luo as code owners August 20, 2026 22:01

github-actions Bot added the run-ci label Aug 21, 2026

rystewart-nvidia requested review from Ying1123, hnyls2002, merrymercy and xiezhq-hermann as code owners August 25, 2026 17:23

rystewart-nvidia and others added 7 commits August 26, 2026 17:31

add native Nemotron 3.5 Super VL support

efe3848

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

load quantized RADIO exports for Nemotron 3.5 Super VL

b1af8de

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

use canonical text config for MoE initialization

61a761d

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

validate Super VL checkpoint weights

5d762fa

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

organize Super VL tests by source component

3982dab

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

stream Nemotron-H VL weights during loading

d794cad

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

rystewart-nvidia force-pushed the nemotron-3.5-super-vl branch from 2b023ca to d794cad Compare August 26, 2026 19:30

rystewart-nvidia and others added 4 commits August 27, 2026 01:57

delegate DFlash layer capture for Nemotron-H VL

53ebe8a

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

reuse multimodal embeddings in Nemotron-H MTP

2e8e568

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

Merge branch 'main' into nemotron-3.5-super-vl

9c4e8ae

Merge branch 'main' into nemotron-3.5-super-vl

de33e6e

rystewart-nvidia force-pushed the nemotron-3.5-super-vl branch from 9bd55ae to de33e6e Compare August 28, 2026 04:20

Merge branch 'main' into nemotron-3.5-super-vl

ca58394

rystewart-nvidia and others added 6 commits August 31, 2026 06:37

resolve merge conflict

633a878

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

Merge branch 'main' into nemotron-3.5-super-vl

2fecbaf

Merge branch 'main' into nemotron-3.5-super-vl

aa9e434

Merge branch 'main' into nemotron-3.5-super-vl

ad5c3b5

Merge branch 'main' into nemotron-3.5-super-vl

6bd4cbc

Merge branch 'main' into nemotron-3.5-super-vl

0545f5c

chatgpt-codex-connector Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread python/sglang/srt/models/nemotron_h_mtp.py Outdated

map quantized MTP metadata for Nemotron-H

9c744d4

Signed-off-by: rystewart-nvidia <rystewart@nvidia.com>

Fridge003 reviewed Sep 3, 2026

View reviewed changes

Comment thread python/sglang/srt/managers/scheduler.py Outdated

Comment thread python/sglang/srt/arg_groups/overrides.py Outdated

Comment thread test/registered/unit/managers/test_scheduler_moe_config.py Outdated

rystewart-nvidia added 3 commits September 4, 2026 02:15

cleanup tests

07f7f82

Signed-off-by: rystewart-nvidia <rystewart@nvidia.com>

preserve existing config selection behavior

faec577

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

Merge branch 'main' into nemotron-3.5-super-vl

1ec6ec8

nvpohanh mentioned this pull request Sep 4, 2026

[Rust] Gate health on startup warmup completion #37994

Merged

elvischenv reviewed Sep 8, 2026

View reviewed changes

rystewart-nvidia and others added 4 commits September 8, 2026 15:56

resolve merge conflicts

6ce9ed4

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

Merge branch 'main' into nemotron-3.5-super-vl

94ce165

resolve merge conflicts

1d21a08

Signed-off-by: Ryan Stewart <rystewart@nvidia.com>

Merge branch 'main' into nemotron-3.5-super-vl

18ca2e8

Merge branch 'main' into nemotron-3.5-super-vl

9b71076

Fridge003 approved these changes Sep 9, 2026

View reviewed changes

nvpohanh and others added 2 commits September 10, 2026 11:07

Merge branch 'main' into nemotron-3.5-super-vl

e78fcac

preserve standalone LM heads for Nemotron-H MTP

ee6482b

Fridge003 merged commit fae8cd8 into sgl-project:main Sep 10, 2026
141 of 165 checks passed

Uh oh!

Conversation

rystewart-nvidia commented Aug 20, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

CI States

Uh oh!

rystewart-nvidia commented Aug 20, 2026

Uh oh!

nvpohanh commented Aug 21, 2026

Uh oh!

rystewart-nvidia commented Aug 28, 2026

Uh oh!

nvpohanh commented Aug 31, 2026

Uh oh!

nvpohanh commented Sep 2, 2026

Uh oh!

Fridge003 commented Sep 3, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvpohanh commented Sep 4, 2026

Uh oh!

nvpohanh commented Sep 4, 2026

Uh oh!

elvischenv left a comment

Choose a reason for hiding this comment

Uh oh!

nvpohanh commented Sep 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rystewart-nvidia commented Aug 20, 2026 •

edited by github-actions Bot

Loading

chatgpt-codex-connector Bot commented Sep 3, 2026 •

edited

Loading