[feat] Support modelopt_mixed for Turing and Ampere via Marlin

[ir1ka]

Leverages Marlin kernels to enable modelopt_mixed quantization support, extending compatibility to NVIDIA Turing and Ampere architectures.

Due to limitations in Marlin, tensor dimensions must be aligned; however, the output dimensions of certain layers in the nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 model were misaligned. Consequently, zero-padding was applied.
As a precautionary measure, zero-padding was restricted exclusively to layers utilizing FP8 per-tensor quantization.

@jinzhen-lin Could you please take a look at this PR when you have a moment?

#38776

PR #45295 has already implemented padding for marlin n/k, so only modelopt_mixed support for turing and ampere needs to be enabled.

Purpose

Support model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 for Turing or Ampere.

Test Plan

Test model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4.
Validated implementation using nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4, ensuring functional correctness and performance stability on 4x RTX3090.

Command: vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 --served-model-name Nemotron-3-Super --tensor-parallel-size 4 --enable-expert-parallel --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser nemotron_v3 --kv-cache-memory-bytes 3G --max-model-len auto --async-scheduling --enable-prefix-caching --enable-chunked-prefill --max-num-seqs 4

Test Result

The vllm serve command launched successfully, and performance tests appear normal.

vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:15:32 [loggers.py:259] Engine 000: Avg prompt throughput: 2.9 tokens/s, Avg generation throughput: 46.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO:     127.0.0.1:35860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:15:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:15:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO:     127.0.0.1:46700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:18:42 [loggers.py:259] Engine 000: Avg prompt throughput: 2.2 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
vllm-nemotron-3-super-1  | (APIServer pid=1) INFO 04-04 13:18:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

Previously, the model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 failed to launch because the modelopt_mixed constraint restricted deployment to SM89 or newer architectures only.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[ir1ka]

Since I do not have a Turing card with sufficient VRAM, I have not tested it on Turing; however, it should work correctly on Turing.

[gemini-code-assist]

Code Review

This pull request implements padding for Marlin FP8 kernels to support unaligned dimensions, updates CUDA tile size logic, and lowers the minimum compute capability for ModelOpt quantization. Key feedback points out potential runtime errors from uninitialized variables in the padding logic. Additionally, the current implementation fails to pad the K dimension and lacks support for misaligned channel-wise quantized layers, both of which will lead to kernel failures.

[ir1ka]

I ran some tests using lm-eval, and it appears there are some issues with accuracy. However, as I lack sufficient hardware to conduct comprehensive testing, I have only compared it against the AWQ-quantized model cyankiwi/NVIDIA-Nemotron-3-Super-120B-A12B-AWQ-4bit.

Based on vLLM v0.19.0.

## vLLM v0.19.0 + this pr
local-completions ({'pretrained': 'Nemotron-3-Super', 'base_url': 'http://localhost:8000/v1/completions', 'tokenizer': 'nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4'}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: auto
|    Tasks     |Version|Filter|n-shot|  Metric  |   | Value |   |Stderr|
|--------------|------:|------|-----:|----------|---|------:|---|-----:|
|lambada_openai|      1|none  |     0|acc       |↑  | 0.4784|±  |0.0070|
|              |       |none  |     0|perplexity|↓  |16.5389|±  |0.6143|

## vLLM v0.19.0
local-completions ({'pretrained': 'Nemotron-3-Super', 'base_url': 'http://localhost:8000/v1/completions', 'tokenizer': 'cyankiwi/NVIDIA-Nemotron-3-Super-120B-A12B-AWQ-4bit'}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: auto
|    Tasks     |Version|Filter|n-shot|  Metric  |   |Value |   |Stderr|
|--------------|------:|------|-----:|----------|---|-----:|---|-----:|
|lambada_openai|      1|none  |     0|acc       |↑  |0.7644|±  |0.0059|
|              |       |none  |     0|perplexity|↓  |2.9850|±  |0.0600|

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[ir1ka]

Based on the mainline version (0.19.2rc1.dev66+gb47840019), the acc/PPL degradation issue has been resolved.

The test methods and results are as follows:

vLLM start command:

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 --served-model-name Nemotron-3-Super --tensor-parallel-size 4 --enable-expert-parallel --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser nemotron_v3 --kv-cache-memory-bytes 2g --max-model-len auto --async-scheduling --enable-prefix-caching --enable-chunked-prefill --max-num-seqs 4 --mamba_ssm_cache_dtype float32 --seed 42

Test result of lm-eval:

# command
lm_eval --model local-completions  --model_args base_url=http://127.0.0.1:8000/v1/completions,model=Nemotron-3-Super,tokenizer=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 --trust_remote_code --tasks lambada_openai --batch_size auto --seed 42

# results
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'Nemotron-3-Super', 'tokenizer': 'nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4'}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: auto
|    Tasks     |Version|Filter|n-shot|  Metric  |   |Value |   |Stderr|
|--------------|------:|------|-----:|----------|---|-----:|---|-----:|
|lambada_openai|      1|none  |     0|acc       |↑  |0.7623|±  |0.0059|
|              |       |none  |     0|perplexity|↓  |3.0024|±  |0.0603|

[ir1ka]

@robertgshaw2-redhat Hello, this PR is ready for review.
Thanks.

[jinzhen-lin]

Do gptq / awq / nvfp4 / mxfp4 have similar issues as well?

[ir1ka]

It appears that AWQ is not affected by this issue; the model I tested was cyankiwi/NVIDIA-Nemotron-3-Super-120B-A12B-AWQ-4bit.
The problem may stem from differences in weight sharding logic or the specific quantization method used (modelopt_mixed).

I am currently unable to test the GPTQ, NVFP4, and MXFP4 methods, as I do not have access to corresponding models for them.
NVFP4 should theoretically be affected as well; however, I did not encounter this issue during my testing, so I will not be making any modifications regarding it for the time being. The model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 is a mixed-precision model utilizing NVFP4, FP8, and BF16; while this theoretically included a test of NVFP4, it did not trigger the issue.

This problem originates from gptq_marlin_repack, which requires that the N and K dimensions be aligned. This operator is invoked by the following files: vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py, vllm/model_executor/layers/quantization/utils/marlin_utils_fp8.py, and vllm/model_executor/kernels/linear/mixed_precision/marlin.py. However, marlin_gemm does not appear to perform any checks for this alignment; I am currently unsure whether the marlin_gemm operator itself imposes this same requirement.

[ir1ka]

The scope of this PR might be too broad; should I split it into two separate PRs?

1. One specifically for adding k-dim padding support for Marlin FP8;
2. The other to enable modelopt_mixed support for Ampere, building upon the first PR.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ir1ka.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ir1ka.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ir1ka.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork