Skip to content

[Bugfix][Model] Nemotron 3.5 Super VL: support the native transformers config #64506

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tomeras91 merged 4 commits into from
Oct 29, 2026
Merged

[Bugfix][Model] Nemotron 3.5 Super VL: support the native transformers config #64506

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9efeeb9
[Model] NemotronH Omni: read vision_hidden_size, fall back to vit_hid…
roikoren755 Oct 26, 2026
9d839d9
[CI] Test NemotronH Omni native config with Nemotron 3.5 Super VL
roikoren755 Oct 29, 2026
e56b29a
[Doc] Add Nemotron 3.5 Super VL to supported models
roikoren755 Oct 29, 2026
7ce77a5
Merge branch 'main' into bugfix/support-nemotron-h-omni-without-trust…
vllm-agent Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions docs/models/supported_models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
| `MuseSparkForConditionalGeneration` | Muse Spark Open | T + I<sup>+</sup> + V<sup>+</sup> + A<sup>+</sup> | `meta-models/Muse-Spark-Open-816B-A42B-NVFP4`, etc. | ✅︎ | ✅︎ |
| `Nemotron3_5TranscribeForConditionalGeneration` | Nemotron-3.5-Transcribe | T + A | `nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B`, `nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-FP8`, `nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4` | | ✅︎ |
| `NemotronH_Nano_Omni_Reasoning_V3`, `NemotronH_Nano_VL_V2` | Nemotron Nano 3 Omni, Nemotron Nano V2 VL | T + I<sup>E+</sup> + V<sup>+</sup> + A<sup>*</sup> | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16` | ✅︎ | |
| `NemotronH_Omni_Reasoning_V3` | Nemotron 3.5 Super VL | T + I<sup>E+</sup> + V<sup>+</sup> | `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16`, `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`, etc. | ✅︎ | |
| `RadioMBartPrefixForCausalLM` | Nemotron Parse 3.5 | T + I | `nvidia/Nemotron-3.5-Parse` | | |
| `NVLM_D_Model` | NVLM-D 1.0 | T + I<sup>+</sup> | `nvidia/NVLM-D-72B`, etc. | | ✅︎ |
| `OpenCUAForConditionalGeneration` | OpenCUA-7B | T + I<sup>E+</sup> | `xlangai/OpenCUA-7B` | ✅︎ | ✅︎ |
Expand Down Expand Up @@ -722,10 +723,13 @@ Some models are supported only via the [Transformers modeling backend](#transfor
its [DFlash](../features/speculative_decoding/README.md) path.

!!! note
Both Nemotron Nano architecture names map to the same vLLM implementation.
Both Nemotron Nano architecture names and `NemotronH_Omni_Reasoning_V3` map to
the same vLLM implementation.
Audio inputs are only supported by the Omni variants; `NemotronH_Nano_VL_V2`
checkpoints such as `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16` accept text,
image and video only.
image and video only, as do the Nemotron 3.5 Super VL checkpoints.
`NemotronH_Omni_Reasoning_V3` checkpoints load without `--trust-remote-code`
on `transformers>=5.18.0`; older versions still require the flag.

!!! note
The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.
Expand Down
9 changes: 7 additions & 2 deletions tests/models/registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1253,9 +1253,14 @@ def check_available_online(
"NemotronH_Super_Omni_Reasoning_V3": _HfExamplesInfo(
"nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16", is_available_online=False
),
# TODO: Change repo id once pertinent archs are public.
"NemotronH_Omni_Reasoning_V3": _HfExamplesInfo(
"nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16", is_available_online=False
"nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16",
min_transformers_version="5.18.0",
max_model_len=4096,
use_original_num_layers=True,
hf_overrides={
"text_config": {"layers_block_type": ["linear_attention", "full_attention"]}
},
),
"Nemotron3_5TranscribeForConditionalGeneration": _HfExamplesInfo(
"nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B", is_available_online=False
Expand Down
4 changes: 3 additions & 1 deletion vllm/model_executor/models/nano_nemotron_vl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -997,7 +997,9 @@ def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
)

# Construct the vision projection.
vit_hidden_size = config.vit_hidden_size
vit_hidden_size = getattr(config, "vision_hidden_size", None)
if vit_hidden_size is None:
vit_hidden_size = config.vit_hidden_size
vision_projection_hidden_size = config.projector_hidden_size
llm_hidden_size = config.text_config.hidden_size

Expand Down