[fans-nv]

Overview

For MiniMax M3 FP8 indexer prefill, auto now uses MSA's persistent CuTe scorer instead of FMHA OnlyScore. Depends on vllm-project/MSA#36 (score-only API) and a follow-up pin bump.

Not a duplicate of #59979: that PR splits work inside FMHA; this one switches the prefill scorer to CuTe. They are complementary; the tables below measure both.

Claims

• On FP8 indexer prefill, CuTe beats the FMHA kernel it replaces on all 78 prefill shapes, and never drops below 0.95× of it.
  • Kernel: 6.9× on GB300 and 7.6× on Rubin.
  • E2E prefill: 0–3% faster up to 32K tokens and 6–10% faster at 150K.
• No significant GSM8K difference on either platform.
• BF16 indexing, the default (indexer_kv_dtype resolves to BF16), keeps FMHA. Default settings are unchanged.
• fmha and cute force a backend without changing precision.
• Native Top16 and decode routing are unchanged.

Validation

Setup: TP2, with the companion MSA API. Measured at 0421c58, before squashing and rebasing. Since then only comments, docs and tests changed, and a redundant Q/K dtype guard was removed; scorer selection and dispatch are identical.

Speed ratios:

• Speedup rows give CuTe's speedup over the column's arm (baseline/CuTe, so values above 1 mean CuTe is faster).
• E2E figures are medians over 3 groups.
• Kernel figures are score µs per rank/layer with a cold L2, as a geomean over 78 prefill shapes.

Accuracy: GSM8K (out of 1,319) uses EAGLE3 k=3, over two independent boots.

GB300 (SM103)

	FMHA	#59979 split	CuTe
GSM8K r1 / r2	1263 / 1256	—	1260 (p=0.77) / 1272 (p=0.011)
CuTe E2E speedup, prefill 512–32K / 150K	1.01–1.03 / 1.06–1.08	not measured	—
CuTe E2E speedup, decode / mixed TPOT	1.00–1.01 / 0.99–1.03	not measured	—
CuTe kernel speedup (wins/losses)	6.9× (78/0)	1.93–1.94× (69–70/6)	—
Kernel µs, Q2048 / KV1M	4464	895	305
Kernel µs, Q128 / KV32K	79.5	14.0	20.1

Vera Rubin (SM107; FP8 main KV, because FlashInfer rejects NVFP4 KV on SM107)

	FMHA	#59979 split	CuTe
GSM8K r1 / r2	1259 / 1261	—	1261 (p=0.63) / 1253 (p=0.30)
Long-context retrieval, 8K–128K	12/12	—	12/12
CuTe E2E speedup, prefill 512–32K / 150K	1.00–1.03 / 1.08–1.10	not measured	—
CuTe E2E speedup, decode / mixed TPOT	1.00 / 1.00–1.03	not measured	—
CuTe kernel speedup (wins/losses)	7.6× (78/0)	1.91× (65/8)	—
Kernel µs, Q2048 / KV1M	3808	488	174
Kernel µs, Q128 / KV32K	60.0	10.0	15.1

Findings:

• Rubin E2E outlier: one group of 512/256/C8 measured 0.71 TTFT; the other two measured 1.08 and 1.09.
• vs FMHA: CuTe never loses at kernel level.
• vs [Model][MiniMax-M3] Split-KV indexer prefill scoring and sync-free prefill metadata #59979: CuTe loses on short-Q / 32K-prefix shapes (Q·H < 2048 on GB300, < 4096 on Rubin), by 1–6 µs, plus Rubin's fr NCU attributes the short-Q losses to the static persistent schedule's tail.

Tests:

pytest tests/kernels/attention/test_minimax_m3.py -k msa_indexer_impl

The table below is for this commit on GB300 (SM103), built from the nightly wheel.

MSA	Result
Pin 968061a (stock)	2 passed, 1 CuTe case skipped (no run_scores)
968061a + score API	3 passed, CuTe active

Whole test_minimax_m3.py with the score API: 126 passed, 45 skipped (all AMD-only). pre-commit run passes.

Details

Prerequisite: the MSA plan(..., score_out=) / run_scores() API must land, and the pin (currently 968061a) must be b

• Until then, auto keeps FMHA and cute errors.
• The B200 CI job runs test_minimax_m3.py, but its CuTe cases skip until the pin is bumped.

Host sync: CuTe plans on device (two small kernels, no D2H) once per batch, outside CUDA Graph capture, and run_scores()h no allocation or sync. It replaces the FMHA plan's seq_lens.cpu()` and boolean-mask gather, removing two host syncs per prefill step. Kernels compile during the existing MiniMax M3 warmup.

--attention-config '{"indexer_kv_dtype":"fp8","minimax_m3_indexer_prefill_backend":"auto"}'

CuTe is admitted for:

• SM100/103/107
• FP8 E4M3
• D128, page 128
• H ∈ {1, 2, 4} per rank
• sparse_local_block ≥ 1

Otherwise, auto keeps FMHA.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀