[tnguyeng]

Motivation

amd/MiniMax-M3.1-Flash-Quark-MXFP4 crashes at startup when --disable-shared-experts-fusion is left off. Fusion is on by default.

The routed experts are stored as packed MXFP4. The shared expert is still stored as block FP8, at full width. With fusion on, that shared expert is loaded into the same MoE weight buffer as the routed experts. The loader then copies the full-width FP8 tensor into the half-width MXFP4 buffer:

RuntimeError: The size of tensor a (384) must match the size of tensor b (768) at non-singleton dimension 1

SGLang can requantize the FP8 shared expert to MXFP4 before the copy. This checkpoint never reaches that step: Quark does not mark the routed experts as MXFP4, and the scale tensor is named weight_scale rather than weight_scale_inv.

This PR lets the server start without specifying --disable-shared-experts-fusion.

Modifications

Five files. No new flag.

File	Role
python/sglang/srt/configs/model_config.py	Mark MiniMax-M3.1 Quark as MXFP4 routed experts when the global weight dtype is FP4
python/sglang/srt/model_loader/loader.py	Copy is_fp4_experts onto QuarkConfig
python/sglang/srt/layers/moe/fused_moe_triton/layer.py	Requantize the FP8 shared expert into the packed MXFP4 fused buffer, including Quark weight_scale
python/sglang/srt/layers/quantization/quark/quark.py	Pass weight_scale_is_ue8m0 for float8_e8m0fnu block-FP8 scales
python/sglang/srt/layers/quantization/fp8.py	Decode those E8M0 bytes with 2^(byte - 127) on the unfused block-FP8 path

The requantize happens on the full tensor, before the TP shard copy: FP8 (6144, 3072) becomes packed MXFP4 (6144, 1536), then TP4 shards that to 384, which matches w2.

Reproduction

Fusion on, the command that crashed before this change. Run on the Flash image or a build of this branch, 4x MI355X:

SGLANG_USE_AITER=1 \
HIP_VISIBLE_DEVICES=0,1,2,3 \
NCCL_IB_DISABLE=1 \
NCCL_SOCKET_IFNAME=lo \
SGLANG_WARMUP_TIMEOUT=20000 \
python3 -m sglang.launch_server \
  --model-path amd/MiniMax-M3.1-Flash-Quark-MXFP4 \
  --tp 4 \
  --trust-remote-code \
  --reasoning-parser minimax-m3 \
  --tool-call-parser minimax-m3 \
  --speculative-algorithm DSPARK \
  --fp8-gemm-backend aiter \
  --chunked-prefill-size 32768 \
  --max-running-requests 128 \
  --cuda-graph-max-bs-decode 128 \
  --mem-fraction-static 0.8 \
  --context-length 131072 \
  --page-size 128

Do not pass --disable-shared-experts-fusion. The startup log should contain Shared experts fusion optimization enabled. and Loading FP8 shared expert weights into FP4 fused MoE weights. Adding the flag is the unfused baseline.

Accuracy Tests

Paired runs on the same MI355X host, TP4, DSpark. The only intended difference is shared-expert fusion. GSM8K is sglang_run_eval. NIAH is lm_eval niah_single_2.

Eval	Fusion on	Fusion off	Delta
GSM8K score	0.9650	0.9665	-0.15 pp
GSM8K score std	0.184	0.180	+0.004
NIAH single-2 at 4096	not scored (-1)	not scored (-1)	same
NIAH single-2 at 65536	0.986	0.986	0
NIAH single-2 at 131072	0.952	0.952	0

The GSM8K gap is inside one standard deviation. NIAH at 64k and 128k matches exactly.

Benchmarking

sglang.bench_serve, dataset generated-shared-prefix. Same two runs as the accuracy table. Every point is one shared prefix:

• gsp-num-groups: 1
• gsp-range-ratio: 1.0
• gsp-system-prompt-len: 72000
• gsp-question-len: 8000
• gsp-output-len: 2000

gsp-prompts-per-group steps with concurrency: 32, 64, 128, 256, 512, 1024. Concurrency 4 to 128. The radix cache is flushed after warmup. This is a crash fix, not a throughput win: output tokens/s stays within about 4%.

Concurrency	Prompts per group	Fusion off (tok/s)	Fusion on (tok/s)	Delta
4	32	801.3	792.1	-1.1%
8	64	1289.9	1326.4	+2.8%
16	128	1839.6	1883.2	+2.4%
32	256	2754.8	2697.6	-2.1%
64	512	3406.9	3268.1	-4.1%
128	1024	4082.0	3949.4	-3.2%

Mean TTFT at concurrency 4 is 623 ms unfused and 792 ms fused. DSpark accept length stays around 2.7 to 2.9 on both.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #40849198204
Latest PR Test (Extra): ❌ Run #40849198059
Latest PR Test (AMD ROCm 10): ❌ Run #40849198142

[HaiShaw]

should avoid to put quark specific stuff here.

[tnguyeng]

Thank @HaiShaw for the review. I moved the MiniMax Quark check out of ModelConfig and into QuarkConfig. QuarkConfig now sets is_fp4_experts for minimax_m3_vl when the global weight dtype is fp4. The fused MoE loader still reads that flag, so the shared-expert requantize path is the same, and other Quark models are unchanged.

I applied the change then reran GSM8K with fusion on. Same TP4 DSpark setup as the fusion-off run.

	Fusion on (this rerun)	Fusion off (previous)
GSM8K score	0.9612	0.9665
score std	0.193	0.180