[shen-shanshan]

Overview

#60834 brought the paged MXFP4 sparse indexer (dense path) to DeepSeek-V4 on ROCm gfx950. It was opt-in:

--attention-config '{"indexer_kv_dtype": "mxfp4"}'

Its measurements show the MXFP4 indexer is better than the FP8 one on almost every axis:

• lower TTFT (up to -30.7%) and TPOT;
• +9.2% KV cache capacity (68 B vs 132 B per compressed token);
• near-lossless accuracy: GSM8K -0.16pt, and GPQA Diamond is unchanged at 0.7121.

The results are copied below. This PR therefore makes MXFP4 the default indexer K cache dtype for DeepSeek-V4 (model_type == "deepseek_v4", i.e. V4-Pro / V4-Flash) on ROCm CDNA4, and logs that it is enabled.

Claims

Nothing else changes:

• CUDA, XPU and CPU behave as before;
• DeepSeek-V4.1 keeps its current opt-in;
• every other model is untouched.

Validation

Benchmark (from #60834, DeepSeek-V4-Pro)

Fixed sequence length (8k1k, TP8, MTP=2):

Conc	Indexer	Output token throughput (tok/s)	Change	Mean TTFT (ms)	Change	Mean TPOT (ms)	Change
1	FP8 (old default)	145.99	—	597.24	—	6.24	—
1	MXFP4 (new default)	156.34	7.09% ↑	413.91	30.70% ↓	5.96	4.49% ↓
4	FP8 (old default)	454.91	—	504.07	—	8.02	—
4	MXFP4 (new default)	466.92	2.64% ↑	437.29	13.25% ↓	7.95	0.87% ↓
16	FP8 (old default)	1020.90	—	744.88	—	14.51	—
16	MXFP4 (new default)	1030.30	0.92% ↑	681.89	8.46% ↓	14.46	0.34% ↓
64	FP8 (old default)	1629.53	—	1637.36	—	36.85	—
64	MXFP4 (new default)	1656.28	1.64% ↑	1622.58	0.90% ↓	36.29	1.52% ↓

Agentic replay (DP8, MTP=3, prefix caching):

Indexer	Output Token Throughput (tok/s)	Mean TTFT (ms)	Mean TPOT (ms)	KV Cache Capacity (tokens)
FP8 (old default)	697.25	2,442.24	46.01	4,589,684
MXFP4 (new default)	675.83 (3.07% ↓)	2,321.05 (4.96% ↓)	43.42 (5.62% ↓)	5,010,749 (9.17% ↑)

• Latency improves: mean TPOT -5.62% and mean TTFT -4.96%.
  • TPOT is the most consistent signal: both MXFP4 runs (44.60 / 42.24 ms) had lower TPOT than both FP8 runs (46.98 / 45.03 ms).
  • TTFT varied by up to 13% between runs of the same configuration, so treat its gain as indicative.
• Output throughput is within noise (-3.07%). On this workload it is driven mainly by the trace replay and by output lengths.
• KV cache capacity: +9.17%. Peak KV usage stayed below 40% here, so this workload does not yet turn the extra capacity into throughput. Higher-concurrency or longer-context workloads should benefit more.

Accuracy (from #60834, DeepSeek-V4-Pro)

GSM8K:

Task	Filter	n-shot	Metric	FP8 (old default)	MXFP4 (new default)	Change
gsm8k	flexible-extract	5	exact_match ↑	0.9492	0.9477	-0.16% ↓
gsm8k	strict-match	5	exact_match ↑	0.9500	0.9484	-0.17% ↓

GPQA Diamond:

Indexer	GPQA Diamond	Empty Responses (reasoning over budget)
FP8 (old default)	0.7121 ± 0.0323	13 / 198
MXFP4 (new default)	0.7121 ± 0.0323	11 / 198

• Accuracy is unchanged on GPQA: both indexers score 0.7121. Per question, 124 are correct in both runs, and the disagreements are symmetric (17 vs 17, McNemar p = 1.0).
• This exercises the indexer's long-context path: reasoning traces run to tens of thousands of tokens, far past the 4096 tokens where sparse top-k selection starts.
• The GSM8K change is negligible (< 0.2% relative).

Details

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[shen-shanshan]

/ci run

[github-actions]

✅ Triggered Buildkite CI #97226 for commit d2514d7e8143.

[shen-shanshan]

/ci retry

[github-actions]

✅ Queued 1 failed job(s) for retry in Buildkite CI #97226.

[shen-shanshan]

/ci run

[github-actions]

✅ Triggered Buildkite CI #97269 for commit 7e8d716a7f88.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 7e8d716a7f88 before starting CI. Pull the branch before pushing more changes.