[MM] Move Nemotron VL image normalization to device by jzakrzew · Pull Request #57924 · vllm-project/vllm

jzakrzew · 2026-09-21T10:21:52Z

Purpose

Move image and video rescaling/normalization for the Nano Nemotron VL models from CPU preprocessing to the model device. This follows the shared GPU-side normalization mechanism introduced in #50411.

Before this change, the processor converted raw uint8 pixels to float32 before bicubic interpolation, normalized them on the CPU, and returned pixels in the model dtype. This made resizing more expensive and doubled the amount of pixel data transferred for bfloat16 models.

This PR:

opts NemotronH_Nano_VL_V2 into mm_device_do_normalize
keeps resized image and video pixels as uint8 through preprocessing and transfer (restoring pre-Optimize nemotron VL image/video preprocessing #40283 numerical behavior)
applies the existing FusedInputNorm immediately before the C-RADIO encoder
handles both fixed-resolution NCHW inputs and dynamic-resolution packed patches
preserves CPU normalization when device normalization is disabled

Bicubic interpolation remains under torch.compile. Both normalization modes, with and without resizing, produce one Dynamo graph each with zero graph breaks under the Inductor backend.

Performance

The renderer and executor are separate processes, so the timings below are
reported per stage and are not added together. Executor work may overlap with
subsequent renderer work.

Setup:

CPU: Intel Core i9-10920X, one PyTorch intra-op thread
GPU: NVIDIA GeForce RTX 3070 Ti
Model dtype: bfloat16
Reported values: median latency

One 1536 x 1536 image

Stage	Before	After	Improvement
Renderer: resize and normalize/prepare pixels	20.619 ms	7.161 ms	2.88x
Executor: H2D transfer and device normalization	1.758 ms	1.255 ms	1.40x
Pixel payload	13.50 MiB	6.75 MiB	2x smaller

32 video frames at 512 x 512

Stage	Before	After	Improvement
Renderer: resize and normalize/prepare pixels	50.683 ms	20.509 ms	2.47x
Executor: H2D transfer and device normalization	6.226 ms	4.378 ms	1.42x
Pixel payload	48.00 MiB	24.00 MiB	2x smaller

Test Plan

The tests cover:

retaining uint8 through compiled bicubic interpolation when device normalization is enabled
exact bfloat16 agreement between CPU normalization and FusedInputNorm
both fixed-resolution NCHW and dynamic-resolution packed-patch layouts
CPU-normalized and device-normalized paths for both image and video processors
rejection of partially enabled rescaling/normalization.

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

mergify · 2026-09-21T11:26:03Z

Documentation preview: https://vllm--57924.org.readthedocs.build/en/57924/

netanel-haber · 2026-09-22T13:00:22Z

Hey @jzakrzew - can you please run a baseline correctness image only and a video eval for nemotron 3 nano omni [the newer nano vl variant] with the relevant snippet e.g. https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4#general-invocation-1%C3%97gpu-eg-1%C3%97b200, and then run with this layered on top, and prove the accuracy is on par? Then, append the script that runs these evals to the description for future reference.
It doesn't really matter which evals you choose, as the point is ruling out drift, not attaining a specific score, but this table [from the same model card] can give you sanity that your before and after make sense:

netanel-haber · 2026-09-22T13:03:57Z

+|--------------------|--------------------------------------|-------------------------------------------------------------|
+| `qwen2-vl`         | `Qwen2VLForConditionalGeneration`    | `Qwen/Qwen2-VL-2B-Instruct`, etc.                           |
+| `qwen2.5-vl`       | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc.                         |
+| `nano-nemotron-vl` | `NemotronH_Nano_VL_V2`               | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16`, etc.          |


Mention https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4

Added to the table.

netanel-haber · 2026-09-22T13:08:07Z

    size: tuple[int, int] | None = None,
    norm_mean: torch.Tensor | None = None,
    norm_std: torch.Tensor | None = None,
+    do_cpu_normalize: bool = True,


I'd support removing the optionality completely and just forcing gpu norm as part of the pr. This would simplify this change a lot. Any specific reason you would want this branching?

although only if/when safe to do so, https://github.com/vllm/project/vllm/commit/2fa490470ddf0fe578f7b70cd3aa8f6c1e06b2c6 - I see it might not be universally supported yet? Or was that just August 6 and now there are no caveats?

Bottom line, please check if this flag is still necessary or we can go all in. The answer may be yes of course

this is just a function-level default value to preserve the API. The actual value used in normal execution path comes from do_normalize argument to NanoNemotronVLProcessor.__init__

Keeping the CPU path. main forces mm_device_do_normalize off under the Rust frontend, so this model still needs it. The global --no-mm-device-do-normalize from #50411 is the switch, and it's what the before/after evals in the PR conversation use: "off" is bit-identical to main. The processor-level default only preserves the API.

netanel-haber · 2026-09-22T13:09:20Z

Also: Please get perf numbers for same table for https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 as that is the new and mainly used checkpoint of this arch

netanel-haber · 2026-09-22T13:29:48Z

+                    rescale_factor=1.0 / 255.0,
+                )
+                if multimodal_config.mm_device_do_normalize
+                else FusedInputNorm.identity()


I think rather than have this identity thing, if mm_devide_do_normalize can actually be False, have input norm be FusedInputNorm | None, and then only call it if input_norm is not None, so as not to incur a perf hit for non device normalized runs?

Done: input_norm is FusedMMInputNorm | None and is skipped when off. build_mm_input_norm() doesn't fit this checkpoint: it stores norm_mean/norm_std, so the helper would return the identity.

netanel-haber · 2026-09-22T13:30:37Z

+        else:
+            assert pixel_values.ndim == 3
+            flattened = pixel_values.reshape(-1, shape[-1])
+        normalized = self.input_norm(flattened, self.llm_dtype)


Can you prove this produces the same dtype we had before post-norm?

Or I guess the question is - this seems to change when the norm happens at least, no? So the accuracy evals you will run will prove whether or not that changes anything

Both paths emit llm_dtype (bf16). The CPU path is bit-identical to main (110/110 tensors), and without a resize the device norm is bit-identical to the CPU norm; with a resize, the only difference is the uint8 rounding. c8cfbb1 adds a test that runs the processors' uint8 output through the device norm and checks the dtype against the CPU path. On accuracy, the evals in the PR conversation show no significant change on any set.

Co-authored-by: Codex <codex@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

…malization Assisted-by: Codex Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

mergify · 2026-10-14T21:13:39Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jzakrzew.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…malization Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

…malization Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

FusedMMInputNorm dispatches to a Triton kernel on GPU platforms, so feeding it CPU tensors fails there. Move the module and pixels to the platform device, as tests/model_executor/layers/test_mm_input_norm.py does. Also give the vision layernorm helper model an input_norm, which extract_feature now reads. Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Run the processors' uint8 output through the model's device normalization and check it against the CPU-normalized pixels: same dtype, values within the resize rounding, for the image, video and dynamic-resolution layouts. Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

brandonpelfrey · 2026-10-22T21:13:36Z

@netanel-haber @jzakrzew I'm helping Jakub complete the remaining testing for this PR and get it merged. Here are the before/after evals and the NVFP4 perf numbers you asked for; the script to reproduce them is attached.

Accuracy. Nemotron-3-Nano-Omni NVFP4 with the model card invocation, on one build of this PR: two servers that differ only by --no-mm-device-do-normalize. The "off" server reproduces main exactly. Greedy decoding, paired per question, 95% CI by image or video.

Set	N	Δ on − off (pp)	95% CI
CV-Bench-2D	1,438	+0.07	[−0.28, +0.42]
ChartQA_TEST	2,500	+0.20	[−0.20, +0.64]
DocVQA_VAL	5,349	+0.06	[−0.17, +0.30]
InfoVQA_VAL	2,801	−0.07	[−0.58, +0.44]
Video-MME short / medium / long	900 each	+0.33 / +0.56 / −0.78	long: [−2.00, +0.44]
All	14,788	+0.06	[−0.13, +0.24]

No set shows a significant drop. BF16 pooled over 12,988 questions: −0.11 [−0.25, +0.03]. The Video-MME long interval is wide (300 videos); a second build gave −0.11.

Perf (NVFP4, H100).

Renderer: 32.2 → 13.4 ms for a 1536² image and 60.5 → 31.7 ms for 32 × 512² frames. Payload halved.
TTFT: −26 to −34% for 2048²–4096² images, −58% for a 148-frame video.
Video throughput is about 4× at 8 requests in flight.

Next: I pushed two test commits (085ebd0, c8cfbb1).

The device-norm tests now run on the platform device; on a GPU runner they failed with CPU tensors.
A new check confirms the device path gives the encoder the same dtype as the CPU path.

I've also replied on the inline threads. Ready for re-review.

Reproduce: attached as eval_57924.py.txt (rename to .py):
python eval_57924.py reproduce nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 --on-gpus 0,1,2,3 --off-gpus 4,5,6,7 --preset quick takes about 25 min on 4 + 4 H100; --preset full takes about 3 h. On one build it reproduces the answers above byte for byte.

brandonpelfrey · 2026-10-22T21:21:18Z

eval_57924.py
Attaching the accuracy comparison script.

netanel-haber · 2026-10-29T12:00:40Z

 | `llama-nemotron-vl-embed`  | `LlamaNemotronVLForEmbedding`              | `nvidia/llama-nemotron-embed-vl-1b-v2`                |
 | `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2`               |
 | `mistral3`                 | `Mistral3ForConditionalGeneration`         | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. |
+| `nano-nemotron-vl`         | `NemotronH_Nano_VL_V2`                     | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4` |


Nit: registry.py maps four architectures to NemotronH_Nano_VL_V2 (L542-545), including NemotronH_Super_Omni_Reasoning_V3, so the Super checkpoints also get device normalization from this PR. Could additional rows be added for super models as well?

seems like NemotronH_Super_Omni_Reasoning_V3 has no models in HF? I added the other two.

List Nano Omni and Super VL checkpoint architectures separately without widening the table. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

mergify · 2026-10-29T13:28:43Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jzakrzew.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…malization Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com> Co-authored-by: Codex <noreply@openai.com> # Conflicts: # tests/models/multimodal/test_nano_nemotron_vl.py

netanel-haber · 2026-10-29T13:57:52Z

LGTM - all of my review items were completed.

tomeras91 · 2026-10-29T14:15:03Z

/ci run

github-actions · 2026-10-29T14:16:10Z

✅ Triggered Buildkite CI #98333 for commit 1c3311fea720.

ℹ️ This PR is 11 commits behind upstream main. Merged upstream main into this branch as 1c3311fea720 before starting CI. Pull the branch before pushing more changes.

jzakrzew requested a review from tomeras91 as a code owner September 21, 2026 10:21

claude Bot reviewed Sep 21, 2026

View reviewed changes

mergify Bot added the multi-modality Related to multi-modality (#4194) label Sep 21, 2026

jzakrzew changed the title ~~[MM] Move NemotronH VL image normalization to device~~ [MM] Move Nemotron VL image normalization to device Sep 21, 2026

jzakrzew and others added 3 commits September 21, 2026 13:25

Move Nemotron VL image normalization to device

525229b

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

Test Nemotron VL video device normalization

59ea70b

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

Document Nemotron VL device normalization

ee18b49

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

jzakrzew force-pushed the nemotron-device-normalization branch from fdf8c4d to ee18b49 Compare September 21, 2026 11:25

mergify Bot added the documentation Improvements or additions to documentation label Sep 21, 2026

netanel-haber reviewed Sep 22, 2026

View reviewed changes

shengliangxu mentioned this pull request Sep 23, 2026

[Model] Update RADIO vision encoder to match the current Transformers version #58468

Merged

jzakrzew added 3 commits September 29, 2026 09:15

Merge upstream/main into nemotron-device-normalization

ce511aa

Co-authored-by: Codex <codex@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

addressing review feedback

201e9c5

Co-authored-by: Codex <codex@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

Merge remote-tracking branch 'upstream/main' into nemotron-device-nor…

1d64db9

…malization Assisted-by: Codex Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

lesj0610 mentioned this pull request Oct 6, 2026

[Bugfix] Warn only on per-request do_normalize/do_rescale overrides #60240

Merged

6 tasks done

mergify Bot added the needs-rebase label Oct 14, 2026

Merge remote-tracking branch 'upstream/main' into nemotron-device-nor…

3ac420b

…malization Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

mergify Bot removed the needs-rebase label Oct 15, 2026

brandonpelfrey and others added 3 commits October 22, 2026 19:52

Merge remote-tracking branch 'upstream/main' into nemotron-device-nor…

851dd43

…malization Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

dlesage-nvidia mentioned this pull request Oct 29, 2026

[MM] Add portable exact CV-CUDA Nemotron vision optimizations dlesage-nvidia/vllm#11

Draft

4 of 6 tasks

netanel-haber reviewed Oct 29, 2026

View reviewed changes

[Docs] Document Nemotron Super device normalization

8880629

List Nano Omni and Super VL checkpoint architectures separately without widening the table. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>

jzakrzew force-pushed the nemotron-device-normalization branch from 9976267 to 8880629 Compare October 29, 2026 12:17

mergify Bot added the needs-rebase label Oct 29, 2026

Merge remote-tracking branch 'upstream/main' into nemotron-device-nor…

9b17478

…malization Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com> Co-authored-by: Codex <noreply@openai.com> # Conflicts: # tests/models/multimodal/test_nano_nemotron_vl.py

mergify Bot removed the needs-rebase label Oct 29, 2026

netanel-haber approved these changes Oct 29, 2026

View reviewed changes

tomeras91 enabled auto-merge (squash) October 29, 2026 14:09

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 29, 2026

tomeras91 approved these changes Oct 29, 2026

View reviewed changes

Merge branch 'main' into nemotron-device-normalization

1c3311f

tomeras91 merged commit 5a4f307 into vllm-project:main Oct 29, 2026
113 checks passed

brandonpelfrey mentioned this pull request Oct 30, 2026

[Multimodal] Cache PyNvVideoCodec decoder caps #64618

Merged

6 tasks done

dlesage-nvidia mentioned this pull request Oct 30, 2026

[MM] Add opt-in CV-CUDA image preprocessing for Nemotron VL dlesage-nvidia/vllm#12

Draft

6 tasks done

Conversation

jzakrzew commented Sep 21, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Performance

One 1536 x 1536 image

32 video frames at 512 x 512

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mergify Bot commented Sep 21, 2026

Uh oh!

netanel-haber commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

netanel-haber commented Sep 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Oct 14, 2026

Uh oh!

brandonpelfrey commented Oct 22, 2026

Uh oh!

brandonpelfrey commented Oct 22, 2026

Uh oh!

netanel-haber Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Oct 29, 2026

Uh oh!

netanel-haber commented Oct 29, 2026

Uh oh!

tomeras91 commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jzakrzew commented Sep 21, 2026 •

edited by github-actions Bot

Loading

netanel-haber commented Sep 22, 2026 •

edited

Loading

netanel-haber Oct 29, 2026 •

edited

Loading