[AMD][MXFP4] Reland "Online MXFP4 quantization 2/N - FP8 to MXFP4 requantization on AMD GPUs"

[fxmarty-amd]

Motivation and description: please refer to #18182 (original PR), the unit tests gsm8k thresholds, as well as #18005 (comment) and:

---

This PR relands #18182 with the only change being e187c11, removing online quantization logging logic from model_runner.py.

#18182 was reverted by #28213 due to a wrongful merge conflict in model_runner.py affecting modelopt quantization code path, not accounting for the previous fix #27284 that fixed a bug present in the first place in #18005.

The CI running on #18182 did not capture this issue (test/registered/quant/test_modelopt_fp8.py did not run), and I was not aware of #27284, did not run tests on modelopt code path. Apologies for the trouble @mmangkad @merrymercy

Why test_modelopt_fp8.py did not run?

TL;DR: It seems to me there should be a way/label to run the PR CI without skipping subsequent workflows (e.g. 1-gpu-large), to validate feature PRs.

Unrelated to #18182, base-b-test-1-gpu-small / base-b-test-1-gpu-small (1) failed in its CI with

✗ FAILED:
  /actions-runner/_work/sglang/sglang/test/registered/sampling/test_original_logprobs.py (exit code -9)

in turn resulting in:

Error: Fast-fail: skipping — root cause job(s): wait-for-base-b, base-b-test-1-gpu-small / base-b-test-1-gpu-small (1)

errors in 1-gpu-large tests (e.g. https://github.com/sgl-project/sglang/actions/runs/27459884924/job/81172119782?pr=18182), resulting in test_modelopt_fp8.py to be not run. Other 1-gpu-small tests are skipped as well.

Note that stage-b-test-1-gpu-small-amd-mi35x on AMD did also NOT run in the latest CI of #18182, with the errors:

The workflow is not valid. .github/workflows/pr-test-amd.yml (Line: 198, Col: 3): 
The workflow is not valid. .github/workflows/pr-test-amd-rocm720.yml (Line: 214, Col: 3): 

Testing

Passing on MI355X:

CUDA_VISIBLE_DEVICES="6,7" pytest test/registered/quant/test_quark_mxfp4.py -s -vvvvv -k "TestOnlineQuantizationMemoryLoadDense or TestOnlineQuantizationMemoryLoadMOE or TestFP8ToMXFP4DenseTP1 or TestFP8ToMXFP4DenseTP2 or TestFP8ToMXFP4MOETP1"

Passing on H100:

pytest test/registered/quant/test_modelopt_fp8.py -s -vvvvv
pytest test/registered/unit/model_loader/test_modelopt_loader.py -s -vvvvv

& let's see the CI.

---

CI States

Latest PR Test (Base): ✅ Run #29748202307
Latest PR Test (Extra): ✅ Run #29748202077

[yctseng0211]

@fxmarty-amd mi355 runners are under maintenance this week for GPU hang issue.
@michaelzhang-ai can you update the latest status of MI355 maintenance?

[michaelzhang-ai]

Mi355 runner is not available yet.

[fxmarty-amd]

@michaelzhang-ai @yctseng0211 mi355 test in this run on an other PR: https://github.com/sgl-project/sglang/actions/runs/29615538361/job/88014527301 passed 2 days ago, are MI355 runners available now?

[fxmarty-amd]

Okay now stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) passed:

Test Summary: 5/5 passed
============================================================
✓ PASSED:
  /sglang-checkout/test/registered/amd/test_dsa_skip_logits_clean.py
  /sglang-checkout/test/registered/attention/test_verify_splitkv.py
  /sglang-checkout/test/registered/quant/test_quark_mxfp4.py
  /sglang-checkout/test/registered/unit/layers/test_mamba_state_scatter_triton.py
  /sglang-checkout/test/registered/unit/models/test_deepseek_mla_dispatch.py
============================================================

Other mi325/mi350 tests are waiting for runners...

[michaelzhang-ai]

@michaelzhang-ai @yctseng0211 mi355 test in this run on an other PR: https://github.com/sgl-project/sglang/actions/runs/29615538361/job/88014527301 passed 2 days ago, are MI355 runners available now?

yes, it is available now

[fxmarty-amd]

@amd-bot ci-status

[amd-bot]

@fxmarty-amd

CI Status for PR #28291

Merge verdict: ✅ Safe to merge on CI signal. PR CI is complete (no fast-fail cascade — all skips are normal path-filter skips). There are 2 executed test failures, both unrelated to this PR (an Intel XPU graph-capture timeout and an AMD mock-model PD-disaggregation crash — neither touches quantization code). Most importantly, this PR's core feature is genuinely exercised and passing: the new test_quark_mxfp4.py ran on the AMD MI35x (gfx950) runner and passed (GSM8K acc 0.878–0.932), actually exercising the online FP8→MXFP4 requantization path. A human may still want to rerun the two red jobs since both look like pre-existing infra/flake failures.

Note

Changed MXFP4/quark/fp8 paths ARE covered by test/registered/quant/test_quark_mxfp4.py, which ran and passed on AMD MI35x (stage-b-test-1-gpu-small-amd-mi35x, 5/5 passed, elapsed 822s, log shows "Using online MXFP4 quantization in dense linear layers"). The test is intentionally skipped on CUDA (5 skipped — the feature requires an AMD ROCm/MXFP4-capable device), which is expected, not a coverage gap.

Changed files: fp8.py (+127/-51), quark_w4a4_mxfp4_moe.py (+375/-28), quark_w4a4_mxfp4.py (+214/-33), quark.py (+83/-15), dequantization.py (new), online_quantization.py (new), test_quark_mxfp4.py (+134), +7 more (all AMD/CUDA quantization + docs).

Executed CI failure attribution: AMD: 1 failure (0 related) · Others (XPU): 1 failure (0 related) · 2 aggregator jobs (finish, pr-test-amd-extra-finish) collapsed into these roots.

AMD Executed Failures

Job	Test File	Test Function	Error	Related?	Why
extra-a-test-2-gpu-large-amd	test/registered/mock_model/test_e2e_pd.py	test_pd_transfer_canary_clean	500 != 200; decode server SIGKILLed (rc=-9, exit 137)	🟢	Mock-model PD test uses Qwen3-0.6B with load_format='dummy', quantization=None — never enters the MXFP4/quark/fp8 code this PR changes; decode OOM/kill is a PD-infra issue

Other (XPU) Executed Failures

Job	Test File	Test Function	Error	Related?	Why
stage-b-test-1-gpu-xpu	test/registered/xpu/test_xpu_graph.py	N/A (whole file)	timeout after 1200s → exit 255	🟢	Intel XPU graph-capture test; PR changes only AMD-ROCm/CUDA quantization files — zero XPU code touched

Details / what to do before merge

• No PR-related action required for merge. Both failures are in code paths this PR does not modify, and the PR's own MXFP4 test suite passes on the correct hardware (MI35x).
• test_e2e_pd.py (AMD): decode process exited rc=-9 (SIGKILL, likely OOM/watchdog on the disaggregation fixture), causing the router to return 500. Suggest a rerun of extra-a-test-2-gpu-large-amd; if it persists it's a mock-model PD-infra issue to track separately, not a blocker for this PR.
• test_xpu_graph.py (XPU): timed out (all 7 other XPU tests in the shard passed). Suggest a rerun of stage-b-test-1-gpu-xpu; an isolated XPU-graph timeout unrelated to an AMD-quantization PR is almost certainly flake/infra.

---

Generated by amd-bot using Claude Code CLI

[fxmarty-amd]

CI status looks good.

Additionally, running:

cat > tests_fp8.txt << 'EOF'
test/registered/quant/test_w8a8_quantization.py
test/registered/quant/test_fp8_blockwise_row_padding.py
test/registered/quant/test_fp8_utils.py
test/registered/quant/test_modelopt_fp8.py
test/registered/quant/test_quant_config_parsing.py
test/registered/models_e2e/test_gemma4_fp8_per_expert_loading.py
test/registered/jit/test_mxfp8_moe.py
test/registered/unit/layers/quantization/test_mxfp4_sm90_cutlass.py
test/registered/moe/test_fused_moe.py
test/registered/moe/test_triton_fused_moe.py
test/registered/moe/test_triton_moe_channel_fp8_kernel.py
test/registered/unit/layers/quantization/test_flashinfer_trtllm_fp8_fallback.py
test/registered/quant/test_fp8_kernel.py
EOF

and

pytest @tests_fp8.txt  -s -vvvvv

on 8xH100 gives:

= 38 passed, 10 skipped, 16 warnings, 516 subtests passed in 747.09s (0:12:27) =