Skip to content

[Perf] deepseek-v4.1: Select the captured prefill full top-k per row - #45073

Closed
hnyls2002 wants to merge 1 commit into
mainfrom
lsyin/dsv41-captured-full-topk-per-row
Closed

hnyls2002 wants to merge 1 commit into
mainfrom
lsyin/dsv41-captured-full-topk-per-row

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator
  • The prefill CUDA graph's captured full top-k (FullTopKIndexer._deep_gemm_prefill_captured) now selects each row up to its own length with topk_transform_paged_from_metadata, the selection the decode path already runs on the same paged logits, instead of a torch.topk over the whole captured width.
  • The v1 kernel takes k from the output width and caps it at 1024 (kMaxTopK in deepseek_v4/topk_v1.cuh, now TOPK_V1_MAX_K). Outputs wider than that keep the torch top-k over min(k, width) columns, so no configured k is truncated.
  • TestDSV4FullTopKCapturedFallback checks which path each output width takes, and that the torch path keeps every column.

Behavior change: with --cuda-graph-backend-prefill breakable, DeepSeek-V4.1's ratio-1/2 indexer layers inside the prefill graph select with the per-row kernel (index_topk is 512 for DeepSeek-V4.1-Flash).

Checked on GB300 with DeepSeek-V4.1-Flash, TP4 (DSPARK, chunked prefill 8192, decoder replay on), against main (617c8b6) with the same prefill graph:

  • test_deepseek_v4.py passes on GB300 (27 tests).
  • GSM8K (sgl-eval, 200 questions, the model's preset, 8-shot prefix): 98.0%, against main's 97.5%.
  • First-token top-20 distributions on prompts that replay the graph (590 to 14,795 tokens), as total variation distance from main eager: at most 0.073; main's graph is at 0.133 on the same prompt, and a second main eager run at 0.088.
  • TTFT, one request, median of 3:
New tokens main (ms) this PR (ms)
256 34.6 34.7
1,024 44.9 41.8
8,000 109.8 101.1

CI States

Latest PR Test (Base): ❌ Run #39192645755
Latest PR Test (Extra): ❌ Run #39192645634
Latest PR Test (AMD ROCm 10): ❌ Run #39192645601

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

Closing: #45783 drops the captured full top-k this PR changed. Since #43733 the released V4.1 checkpoints run the two-level path in the prefill graph, so this code was reached only through the torch prefill indexer switch or a V4.1 config without candidate blocks.

@hnyls2002 hnyls2002 closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant