Enable GPT-OSS FlashInfer MXFP4 on SM120

[mmangkad]

Summary

Enable FlashInfer CUTLASS MXFP4 MoE for GPT-OSS on SM120 and make it the default based on the performance results below

Accuracy

sglang serve --model-path openai/gpt-oss-20b --trust-remote-code

python -m gpt_oss.evals --model openai/gpt-oss-20b --eval gpqa --n-threads 256 --reasoning-effort low --base-url http://127.0.0.1:30000/v1

Result: 0.5751262626262627

Performance

Marlin:

sglang serve --model-path openai/gpt-oss-20b --trust-remote-code --moe-runner-backend marlin

FlashInfer MXFP4:

sglang serve --model-path openai/gpt-oss-20b --trust-remote-code

Serving benchmark:

for CONC in 256 128 64 32 16 8 4 2 1; do
  python -m sglang.benchmark.serving --model openai/gpt-oss-20b --dataset-name random --random-input-len 512 --random-output-len 512 --random-range-ratio 1.0 --num-prompts "$((CONC * 4))" --max-concurrency "$CONC" --warmup-requests "$((CONC * 2))" --flush-cache
done

Concurrency	Marlin tok/s	FlashInfer tok/s	Gain
1	209.99	220.80	5.1%
2	350.44	360.08	2.8%
4	578.07	594.61	2.9%
8	921.38	964.54	4.7%
16	1520.22	1595.51	5.0%
32	2527.53	2768.05	9.5%
64	4270.19	4847.56	13.5%
128	6636.90	8012.56	20.7%
256	8945.32	11636.11	30.1%

---

CI States

Latest PR Test (Base): ✅ Run #30379156612
Latest PR Test (Extra): ❌ Run #30379156258

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[mmangkad]

/rerun-test test/registered/models_e2e/test_gpt_oss*

[mmangkad]

/rerun-test test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py

[github-actions]

Results for /rerun-test test/registered/models_e2e/test_gpt_oss*:

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_gpt_oss_4gpu_bf16.py
cd test/ && python3 registered/models_e2e/test_gpt_oss_4gpu_mxfp4.py

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_gpt_oss_4gpu_bf16.py
cd test/ && python3 registered/models_e2e/test_gpt_oss_4gpu_mxfp4.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_gpt_oss_sm120.py

[github-actions]

Results for /rerun-test test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py:

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py

[b8zhong]

/rerun-test test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py

[github-actions]

Results for /rerun-test test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py:

🚀 1-gpu-5090 (1 test): ❌ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py

[b8zhong]

Do you know if FI has support for SM90? Then we can completely delete triton kernels

[mmangkad]

Do you know if FI has support for SM90? Then we can completely delete triton kernels

We already support flashinfer_mxfp4 on sm90 with bf16 x mxfp4, though I have not measured it against triton_kernels yet. Even if it is faster (though I expect slower), as I discussed before, we cannot fully delete triton_kernels because minimax m3 (mxfp8) and rocm still use parts of it.