[ischencheng]

Motivation

On Hopper every attention-TP rank keeps the whole MiniMax index K (one head) and every sparse layer scores all of it on each decode step. Index top-k sharing only runs on gfx95, so on CUDA every layer scans. At 128K context that is 33.5 MB per request per layer on each rank, and it grows with concurrency.

The opt-in indexer CP from #41488 and #42614 splits that scan across the ranks, but it only turns on for gfx950 at TP4. This PR turns it on for SM90 at TP4 and TP8. The three kernels are unchanged. MiniMax-M3 and M3.1 both take this path on Hopper. With DSPARK the gate keeps CP off, so this applies to runs without speculation or with EAGLE top-k 1.

Modifications

• unsupported_reason takes an sm90 flag. On SM90 it allows attention TP4 or TP8 with attention CP/DP size 1. The gfx950 gate stays TP4.
• MiniMaxIndexerCP sizes its gather buffers from the group (4 or 8 ranks). At TP8 two ranks serve each index head, so 8 query rows are gathered. K is still read once per block, and each rank reads 1/8 of the blocks.
• Still opt-in with SGLANG_MINIMAX_M3_INDEXER_CP=1. At batch 1 the two all-gathers per layer cost more than the scan they save, see the tables below.

Accuracy Tests

• New test_minimax_indexer_cp.py (SM90): the merged CP selection for every rank equals the native page-128 selector at W=4 and W=8, BF16 and FP8, plus chain verify rows across a block boundary. Passes on 1x H200. Gate cases added to test_minimax_indexer_cp_gate.py.
• Real TP8 and TP4 groups on 8x H200: on every rank the CP selection equals the native selector, eager and in a CUDA graph replayed with changed seq_lens, BF16 and FP8, batch 1-32, 32K and 128K context. 0 rows differ.
• End to end on the dummy-weight model below, the first 32 greedy output ids of every request at batch 8 match across all 4 servers (CP off, on, on, off).
• Exact ties: since [Fix] Minimax-M3: Deterministic tie-breaking in the CUDA radix decode top-k #45174 the CUDA radix top-k keeps the lowest block ids, which is what CP does. Checked once on 1x H200 with index K filled from 3 repeated block patterns (W=4 and 8, 32K and 128K context, BF16 and FP8): 0 rows differ. That check isn't in the test file.

Speed Tests and Profiling

Measured on 10-26, before merging the latest main, and not re-run since.

Per layer per rank, 8x H200, real TP8, BF16 index K, 128K context, us. CUDA graph with 8 calls per replay on disjoint requests, max over ranks, median of 20 replays. CP includes both all-gathers. "% of floor" is index K bytes over 4.8 TB/s.

batch	native	native % of floor	CP, TP8	change
1	19.8	35%	47.4	+139%
8	73.0	77%	65.3	-11%
16	133.6	84%	80.5	-40%
32	255.5	88%	100.9	-61%

The two all-gathers alone take 42-48 us at TP8 and 30-42 us at TP4, so most of the CP time is communication. TP4: batch 16 133.2 -> 85.7, batch 32 252.5 -> 112.8. FP8 index K at TP8: batch 32 137.0 -> 87.0, batch 8 45.5 -> 64.1.

End to end on a 16-layer MiniMax-M3 dummy-weight model with the M3.1 layer layout (every layer sparse, first 3 FFNs dense), TP8 on 8x H200, BF16 KV and index K, page 128, 131K input, 160 greedy output tokens, decode CUDA graph. CP off and on alternated twice in the same container. Decode step in ms, median of 14 log windows of 10 steps each:

batch	off (2 runs)	on (2 runs)	change
1	2.25 / 2.23	2.38 / 2.31	+4.7%
4	2.62 / 2.62	2.63 / 2.63	+0.3%
8	3.52 / 3.52	3.17 / 3.16	-10.0%
16	4.96 / 4.98	3.87 / 3.85	-22.3%
24	6.52 / 6.52	4.58 / 4.59	-29.7%

That is 69 us per layer at batch 16 and 121 us at batch 24, so the full 60-layer model should save roughly 4 ms per step with 16 concurrent 128K requests. It's worth turning on for long-context serving with more than a few requests in flight.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ✅ Run #41689509933
Latest PR Test (Extra): ❌ Run #41689509883
Latest PR Test (AMD ROCm 10): ❌ Run #41689509869

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[BBuf]

[BBuf Review Agent] LGTM. Please wait for the related CPU and Hopper kernel CI.

[BBuf]

/tag-and-rerun-ci

[BBuf]

/rerun-failed-ci