[AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs

[ColinZ22]

Summary

Adds support in SGLang for loading and inference for ModelOpt (and Quark) NVFP4 checkpoints by dequantizing the supplied NVFP4 weights and requantizing them to MXFP4 at load time to allow efficient inference using supported AMD hardware (e.g. AMD MI355x).

Enabled via --quantization quark_mxfp4. Weights dequantized then requantized to MXFP4 during loading, so the resulting model occupies MXFP4 memory without ever holding a full-precision copy. Also includes the config-routing and shared-expert-fusion fixes needed to load ModelOpt exports across various architectures using the existing SGLang AMD Quark integration.

Modifications

Online Requantization Path

• layers/quantization/quark/quark.py: _detect_nvfp4_source detects ModelOpt and AMD Quark NVFP4 layouts; _parse_nvfp4_excludes extracts producer exclude lists; from_config routing gated on requantization_method. _maybe_disable_shared_experts_fusion delegates to the existing can_fuse_shared_expert() to correctly handle shared_expert_gate and mtp. exclusions.
• layers/quantization/quark/schemes/quark_w4a4_mxfp4.py: NVFP4 linear scheme: _create_weights_from_nvfp4 allocates NVFP4 staging buffers directly on device, loads all source shards, then dequant-requant to MXFP4 once all bytes are present. Registers (and discards) the NVFP4 per-tensor input_scale slot since MXFP4 uses dynamic activation quantization.
• layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py: NVFP4 fused-MoE scheme: per-expert weight/scale registration, online expert-wise dequant-requant, no-op input_scale slots for w13/w2.
• layers/quantization/dequantization.py, layers/quantization/online_quantization.py, layers/quantization/quark/utils.py: Nvfp4SourceConfig + dequant helpers and CopyNumelCounter to track progressive shard loading

Fixes required to load ModelOpt exports

• Config-loading (model_loader/weight_utils.py): Some ModelOpt NVFP4 checkpoints store quant metadata only in hf_quant_config.json; others duplicate it in config.json. get_quant_config now reads hf_quant_config.json preferentially when present and FP4-typed, routing through from_config.
• Shared-expert-fusion (layers/quantization/quark/quark.py): Some NVFP4 checkpoints keep shared experts in BF16 while routed experts are FP4. Shared-expert fusion remaps mlp.shared_experts into the quantized FusedMoE, causing a shape mismatch. QuarkConfig.__init__ now calls _maybe_disable_shared_experts_fusion which delegates to can_fuse_shared_expert() (which is the same detection path introduced in [AMD] Qwen3.5 MXFP4 breaks after shared expert fusion is enabled #22948 and scoped to Quark in [AMD] move shared expert check function to quark #27057) and sets disable_shared_experts_fusion globally before model construction.

Tests

• test/registered/quant/test_quark_mxfp4.py: Added TestNVFP4ToMXFP4MOETP1 testing NVFP4→MXFP4 MoE requantization accuracy on nvidia/Qwen3-30B-A3B-NVFP4.

Docs

• docs_new/docs/advanced_features/quantization.mdx: documents the NVFP4-MXFP4 requant path (supported checkpoint layouts, and example command.

Accuracy Tests

All numbers below are obtained using lm_eval using SGLang backend with --quantization quark_mxfp4 on AMD MI355X GPUs. (max_gen_toks=32768)

GSM8k (lm_eval, flexible-extract)

Model	Original Release (bf16/fp8)	NVFP4-to-MXFP4	% Recovery
MiniMax-M2.7-NVFP4	0.918	0.920	100.2%
GLM-5.1-NVFP4	0.952	0.928	97.5%
Kimi-K2.6-NVFP4	0.939	0.930	99.0%
Qwen3.5-397B-A17B-NVFP4	0.954	0.945	99.1%
Qwen3.5-397B-A17B-NVFP4-V2	0.954	0.941	98.7%
DeepSeek-R1-NVFP4	0.958	0.950	99.2%

---

CI States

Latest PR Test (Base): ✅ Run #31758298579
Latest PR Test (Extra): ❌ Run #31758298347

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[ColinZ22]

Main PR checks workflow is completed with success; All PR changes verified and passed tests (test_quark_config.py on CPU and test_quark_mxfp4.py on MI35x).

All other test failures shown are unrelated (only 2 NPU related tests fail due to infra timeout issues), and all code paths used in PR are verified locally with lm_eval using GSM8k, AIME25, and GPQA-Diamond-CoT benchmarks; both throughput and accuracy match reference paths within run-to-run noise.

Pending/in-progress AMD stage-c jobs add additional AMD coverage, but are unrelated to and do not block this change, and completed tests in that suite are already passing (e.g. stage-c-test-large-8-gpu-amd part 0).

All PR changes are validated and ready for merge.

[BowenBao]

LGTM, and thanks for checking CI and testing locally.

[BowenBao]

let's try a few smoke tests on h100 / h200, make sure no regressions there. @ColinZ22

[ColinZ22]

let's try a few smoke tests on h100 / h200, make sure no regressions there. @ColinZ22

Ran the following smoke tests using H100 to verify modified shared files:

Nvidia CI test_modelopt_fp8.py and test_modelopt_loader.py passed locally;

sglang.test.few_shot_gsm8k with --num-questions 200 --num-shots 8

• Qwen3-8B BF16 score: 0.940
• Qwen3-30B-A3B-Instruct-2507 score: 0.940
• Qwen3-30B-A3B-Instruct-2507-FP8 score: 0.945
• nvidia/Llama-3.3-70B-Instruct-FP8 score: 0.955
• nvidia/Qwen3-30B-A3B-NVFP4 score: 0.910

CC @BowenBao @HaiShaw

[HaiShaw]

Define a static and hidden REQUANTIZATION_METHODS = ["quark_mxfp4"] seems to be fragile and not generic, consider an explicit knob maybe?

[HaiShaw]

why is bias None?

[ColinZ22]

In the MXFP4 and FP8 quant specs here, None means that the tensor category is not quantized. Currently both Quark linear and MoE does not support output tensor and bias quantization, so the specs have to specify None for both.

[HaiShaw]

why is bias None

[ColinZ22]

In the MXFP4 and FP8 quant specs here, None means that the tensor category is not quantized. Currently both Quark linear and MoE does not support output tensor and bias quantization, so the specs have to specify None for both.

[HaiShaw]

Are we certain that no with_bias in this path?

[ColinZ22]

Yes, this is consistent with the ModelOpt NVFP4 MoE loader, which also registers no bias params.

[BowenBao]

/rerun-failed-ci

[ColinZ22]

/rerun-failed-ci

[ColinZ22]

/rerun-failed-ci

[ColinZ22]

/rerun-failed-ci

[ColinZ22]

All CI failures are all unrelated to PR changes:

• base-c-test-perf-8/16-npu-a3 caused by Ascend TPOT perf-threshold assert on MiniMax M2.5 W8A8 and Kimi K2.6 W4A8, both unrelated different hardware, backend, and quant schemes.

All PR code paths passing checks (test_quark_config.py and test_quark_mxfp4.py on MI35x), all other base tests passing.