[jlqibm]

Motivation

On pre-Blackwell GPUs (SM < 100), every compressed-tensors NVFP4 checkpoint fails
at quant-scheme construction on the first attention linear -- it never reaches
weight loading:

variant	group_size	strategy	failure today
nvfp4 (w4a4)	16	tensor_group	NotImplementedError: Current platform does not support w4a4 nvfp4 quantization.
nvfp4a16	16	tensor_group	ImportError about W4A16Sparse24

Two independent root causes:

1. _is_wNa16_group_channel never checks type == INT, so a weight-only
   4-bit float config satisfies it. It runs first, outside the
   is_activation_quantization_format gate, and short-circuits with raise ImportError for any non-pack_quantized format -- which is why nvfp4a16
   reports a misleading sparse-24 import error. vLLM has the identical missing
   check but escapes it by predicate ordering.
2. No weight-only Marlin path exists for CT fp4 linears. The only fp4 linear
   scheme, CompressedTensorsW4A4Fp4, reports get_min_capability() == 100 and
   calls fp4_gemm unconditionally.

The dense FP4 Marlin kernel already implements exactly what NVFP4 needs --
group_blocks == 1 (gs=16) and E4M3 group scales -- and NVFP4 always carries a
real global scale. So NVFP4 can be served weight-only (fp4 weights, bf16/fp16
activations) on SM80+ with no kernel changes.

Modifications

• compressed_tensors.py
  • _is_wNa16_group_channel now requires type == INT. This is narrower than
    reordering the predicates: int4 pack-quantized is INT by definition so it
    cannot regress, and it closes the trap for the next float4 variant instead of
    stepping around it.
  • New _is_fp4a16_nvfp4 predicate dispatching nvfp4a16 to the weight-only
    scheme.
  • The SM100 NotImplementedError becomes a graceful downgrade to the
    weight-only scheme with a warning_once, following the existing precedent two
    branches below where _is_fp8_w8a8 falls back to CompressedTensorsW8A16Fp8.
  • A present-but-not-NVFP4 input quant still raises rather than silently
    downgrading.
• New CompressedTensorsW4A16Fp4 sibling scheme with
  get_min_capability() -> 80, rather than vLLM's single-class use_a16 flag.
  get_min_capability is a @classmethod resolved on the instance's class, so one
  shared class returning <= 90 would silently lower the Blackwell gate for the
  real w4a4 path too. has_input_global_scale distinguishes a downgraded w4a4
  checkpoint (which carries input_global_scale on disk, dropped post-load) from a
  native a16 one.
• compressed_tensors_w4a4_nvfp4.py gains the Marlin branch in
  process_weights_after_loading / apply_weights. Three attribute fixes are
  prerequisites, since the prep helper reads what CT does not set: params_dtype
  (a missing value trips a raise), quant_config (the group_size guard is
  if getattr(...) is not None, so a missing attr means no validation at all),
  and rebinding weight_packed -> weight.

Note the global-scale inversion: compressed-tensors stores NVFP4 global scales as
divisors (1/scale) while Marlin wants the scale. Missing the inversion overflows
the bf16 dequant bias multiply to inf, zeroing all logits and emitting token 0
forever -- which is why the end-to-end checks below assert on coherent text, not
just a clean load.

No change to Blackwell behavior. MXFP4 is deliberately out of scope: the dense
kernel has no group_blocks == 2 instantiation, its template cannot decode E8M0
scales, and it requires a non-empty global_scale that MXFP4 lacks. MXFP4 now
fails with an honest "no compatible scheme" instead of the sparse-24 ImportError.

Accuracy Tests

End-to-end on H100 (SM90), offline Engine, greedy, prompt "The capital of France is". downgrade is whether the weight-only substitution
warning fired -- it must fire for a w4a4 checkpoint and stay silent for a16:

model	TP	rc	dropped experts	downgrade warning	generation
granite-5.0-20b nvfp4a16	1	0	0	silent	" Paris.\" We need to output the same sentence..."
granite-5.0-20b nvfp4	1	0	0	fires	" Paris.\" We need to output the same sentence. So answer:..."
granite-5.0-120b nvfp4a16	4	0	0	silent	" Paris. The capital of France is Paris..."
granite-5.0-120b nvfp4	4	0	0	fires	" Paris. The capital of France is Paris..."
granite-5.0-230b nvfp4a16	8	0	0	silent	". Question1: What is the capital of France? Answer1"
granite-5.0-230b nvfp4	8	0	0	fires	". The capital of France is Paris. The capital of..."

The warning fires for every w4a4 checkpoint and stays silent for every a16
checkpoint at every TP size. No regression on the bf16 base, both fp8 variants,
or w4a16-c4 (the int4 path most exposed to the type == INT change).

Also verified on real SM90 with no capability mock:
nvfp4(w4a4) -> CompressedTensorsW4A16Fp4 has_input_global_scale=True,
nvfp4a16 -> CompressedTensorsW4A16Fp4 has_input_global_scale=False.

Speed Tests and Profiling

Not a performance change -- this makes previously-unloadable checkpoints load.
Weight-only FP4 Marlin is the same kernel path ModelOptNvfp4A16LinearMethod
already uses on these GPUs.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations. (No doc change: no new user-facing flag or option -- previously-failing checkpoints now load.)
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Tests added:

• test/registered/unit/layers/quantization/test_compressed_tensors_fp4_linear_scheme.py -- 11 CPU scheme-selection tests (register_cpu_ci, override_platform(is_sm90=True)): SM90 downgrade for both variants, SM100 still selecting native CompressedTensorsW4A4Fp4 (guarding the Blackwell gate), int4 pack-quantized still selecting CompressedTensorsWNA16 (guarding the type == INT fix), non-NVFP4 input quant raising, and the downgrade warning firing for w4a4 but not a16.
• test/registered/kernel/quantization/test_compressed_tensors_fp4_linear_numeric.py -- GPU numeric test for the weight-only NVFP4 dense Marlin path (rtol=atol=0.04), modeled on the existing test_nvfp4_marlin.py.
• The existing GPU numeric test test/registered/kernels/ops/quantization/test_nvfp4_marlin.py passes unchanged (4 passed).

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #38024523677
Latest PR Test (Extra): ❌ Run #38024523541
Latest PR Test (AMD ROCm 10): ❌ Run #38024523503

[BBuf]

This always allocates two projections, but Marlin expects one when is_gated=False, so repacking fails its shape assertion. Could we size this from layer.moe_runner_config.is_gated, or reject non-gated experts explicitly?

[jlqibm]

Fixed. w13 is now sized from layer.moe_runner_config.is_gated, for both the weights and the
per-shard global/input scales. Non-gated experts get one shard, which matches what
prepare_moe_nvfp4_layer_for_marlin repacks. A CPU test checks the shapes, and the GPU test below
runs relu2 and silu non-gated experts end to end.

[BBuf]

Could we consolidate these into one CPU scheme test file and one GPU numerical test file? The Linear/MoE selection tests duplicate the config and capability setup and could use subTests. We can also drop the standalone overflow test since the numerical comparison already catches a missing reciprocal, and shorten the repeated docstrings.

[jlqibm]

Done. The tests are consolidated into one CPU file and one GPU file:

• test/registered/unit/layers/quantization/test_compressed_tensors_fp4_schemes.py
  • Scheme selection is now a single table of (linear/MoE, variant, capability) cases run as
    subTests, sharing one config and capability setup. Each case asserts the exact scheme class,
    so the table also covers one format's predicate catching another's checkpoints.
  • The weight-creation and --fp4-gemm-backend selection tests share builders and subTests.
• test/registered/kernels/ops/quantization/test_compressed_tensors_fp4_numeric.py holds the
  linear dequant-reference test, the FlashInfer mm_bf16_fp4 test and the MoE expert test.

I dropped the standalone overflow test. The numeric comparison already fails on a missing
reciprocal, and it now also asserts that the output is finite. The docstrings are shortened.
Overall that's 935 lines down to 644.

The stacked #41288 and #41289 now add their MXFP4 cases to these same two files instead of
adding separate ones. Everything passes on GB200 on each branch's own head.