[mjkvaak-amd]

Overview

Stacked on #62750: only the last commit belongs to this PR. Draft until ROCm/aiter#6734 is in an AITER release that vLLM pins.

#62750 rotates the input of mxfp4_hadamard32 layers in PyTorch (InputHadamardLinearMethod) before the wrapped linear method quantizes it. On AITER's MXFP4 path this adds a separate rotation kernel and a round trip of the activations through memory, which roughly doubles the per-layer time of small GEMMs (see Test Result).

ROCm/aiter#6734 adds hadamard_rotate=True to per_1x32_f4_quant_hip, which rotates each 32-wide block inside the activation quant. This PR uses it:

• LinearMethodBase.fuse_input_hadamard(layer, size) -> bool, default False. InputHadamardLinearMethod calls it after the wrapped method's process_weights_after_loading; if the method returns True, the wrapper keeps rotating the weight but skips its own input rotation. This is the generic hook promised in [Quantization] Block Hadamard input rotation for linear layers: online mxfp4_hadamard32 and Quark online-rotation checkpoints #62750: any backend whose activation quant can rotate can take over, and nothing else changes.
• Online MXFP4 (Mxfp4OnlineLinearMethod) and Quark OCP MX (QuarkOCP_MX through QuarkLinearMethod / QuarkScheme) forward the call to their MxFp4LinearKernel.
• AiterMxfp4LinearKernel accepts when it uses the ASM GEMM and the size is 32, and passes hadamard_rotate=True to gemm_with_dynamic_quant, which forwards it to both HIP quant calls. The Triton branch quantizes with dynamic_mxfp4_quant, which has no rotation, so that path keeps the PyTorch rotation (asserted in the op).
• hadamard_rotate is only passed to AITER when it is True, so AITER releases without [Bug]: vllm-0.5.3.post1部署Qwen2-72b-instruct-awq模型，刚开始服务正常，但是并发高的时候就报错 #6734 keep working as long as nothing fuses.
• Fixes the fake implementation of gemm_with_dynamic_quant, whose signature had drifted from the op (missing x_scales, wrong argument order), and adds a FakeTensorMode test for it.

The server logs once per method type: Mxfp4OnlineLinearMethod rotates its input by the 32-wide Hadamard matrix itself.

When the ASM path is used

AiterMxfp4LinearKernel uses the ASM GEMM only for (N, K) present in AITER's tuned a4w4 config (is_asm_fp4_gemm_tuned, M = 1024 and 8192). On Qwen3.8-2.4T at TP8, out_proj (N = 8192, K = 2048) is tuned in AITER today, but in_proj_qkvz (N = 4608, K = 8192) is not, so it runs the Triton GEMM and can't fuse. A tuned config for that shape (26 values of M, all won by ASM kernels, error ratio 0) moves in_proj_qkvz to ASM; it is part of ROCm/aiter#6734. The numbers below include it in every arm.

Test Plan

• Unit tests (MI355X, image with [Bug]: vllm-0.5.3.post1部署Qwen2-72b-instruct-awq模型，刚开始服务正常，但是并发高的时候就报错 #6734): tests/quantization/test_input_hadamard.py (fused flag on the ASM shape and not on the Triton or emulation paths; rotated MXFP4 error below plain; the wrapper doesn't rotate twice when a method fuses), tests/quantization/test_online_mxfp4.py (fake op dtype and shape), tests/quantization/test_quantization_config_args.py, tests/v1/determinism/test_mx_linear_batch_invariant.py.
• Per-layer microbenchmark on Qwen3.8-2.4T's GDN shapes at TP8: plain MXFP4, PyTorch rotation, fused rotation; CUDA graph, one GPU.
• Fused vs PyTorch rotation output on the same inputs, M from 1024 to 16384.
• TP8 serving, Qwen3.8-2.4T-A95B Quark MXFP4 with dummy weights (--load-format dummy), vllm bench serve random ISL/OSL 1024/1024 at concurrency 4, 64 and 256, 2 reps per arm, fresh caches per arm: RTN mxfp4, mxfp4_hadamard32 with the PyTorch rotation ([Quantization] Block Hadamard input rotation for linear layers: online mxfp4_hadamard32 and Quark online-rotation checkpoints #62750), and mxfp4_hadamard32 fused (this PR) on the GDN projections. Shared experts are online mxfp4 in every arm. Images: vLLM nightly df3bd09 plus the respective diff, [HIP] Fuse a 32-point Hadamard rotation into the MXFP4 per-1x32 quant and AR + RMSNorm epilogue ROCm/aiter#6734 ported to its AITER 0.1.25, and the tuned config.

Test Result

pytest tests/quantization/test_input_hadamard.py tests/quantization/test_online_mxfp4.py \
  tests/quantization/test_quantization_config_args.py tests/v1/determinism/test_mx_linear_batch_invariant.py
80 passed, 4 skipped

Per layer, µs per call (quant + GEMM, CUDA graph, MI355X, AITER a4w4 tuning for in_proj_qkvz applied):

M	in_proj_qkvz plain	PyTorch rotation	Fused	out_proj plain	PyTorch rotation	Fused
1	7.2	15.3	7.4	5.3	10.9	5.0
4	8.6	15.8	7.8	5.1	11.7	5.2
16	9.5	17.1	9.2	5.4	12.6	5.6
64	12.1	21.6	12.6	7.2	24.2	7.5
256	14.7	28.1	15.0	8.0	17.6	8.5
1024	29.3	65.1	29.6	16.9	32.5	17.0
4096	87.6	197.5	90.7	49.5	83.5	46.6
8192	170.6	459.1	175.7	83.5	155.5	88.0

The PyTorch rotation adds 70–170%; the fused rotation adds 0–5%. Compiling the PyTorch rotation (torch.compile) doesn't help. Without the tuned config, in_proj_qkvz runs the Triton GEMM (10.8 µs at M = 1, 120 µs at M = 1024) and keeps the PyTorch rotation.

Fused vs PyTorch rotation: same relative error against the unquantized GEMM to five digits at every M. Up to 0.07% of output elements differ, because the fused quant rotates with an fp32 butterfly while the PyTorch path uses a bf16 matmul with a different summation order. Both round the rotated activations to bf16, and the rare values that round differently can land on the other side of an FP4 rounding boundary. Both paths are deterministic run to run.

TP8 serving (Qwen3.8-2.4T-A95B Quark MXFP4, dummy weights, ISL/OSL 1024/1024, mean of 2 reps):

GDN projections	Output tok/s c=4	c=64	c=256	Mean TPOT ms c=4	c=64	c=256
RTN mxfp4	373.0	1807.1	4028.3	10.45	34.09	60.91
mxfp4_hadamard32, PyTorch rotation (#62750)	326.9 (-12.4%)	1694.4 (-6.2%)	3787.7 (-6.0%)	11.94	36.23	64.86
mxfp4_hadamard32, fused (this PR)	371.3 (-0.5%)	1796.6 (-0.6%)	4024.2 (-0.1%)	10.50	34.21	61.09

Repeats differ by up to 1.6%. The server log of the fused arm has the "rotates its input" line; the other two don't. All three arms quantize the same 322 layers.

The rotation is the same as in #62750, so the accuracy results there carry over; the fused and unfused outputs differ only as described above.

Submission Checklist

• Rebase on [Quantization] Block Hadamard input rotation for linear layers: online mxfp4_hadamard32 and Quark online-rotation checkpoints #62750 once it merges; bump AITER after [HIP] Fuse a 32-point Hadamard rotation into the MXFP4 per-1x32 quant and AR + RMSNorm epilogue ROCm/aiter#6734 is released.

[mergify]

Documentation preview: https://vllm--63272.org.readthedocs.build/en/63272/