Skip to content

[Perf][Kimi K3] Support internal prefill checkpoints in Triton KDA - #56714

Open
gcanlin wants to merge 1 commit into
vllm-project:mainfrom
gcanlin:canlin/kimi-k3-triton-prefill-checkpoint
Open

gcanlin wants to merge 1 commit into
vllm-project:mainfrom
gcanlin:canlin/kimi-k3-triton-prefill-checkpoint

Conversation

@gcanlin

@gcanlin gcanlin commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #52789 and #53614: add 64-token-aligned internal prefill checkpoints to the Kimi-K3 Triton KDA backend to avoid an extra full-model prefill pass.

Test results

16×B200 (2 nodes), TP8/DP2/EP16/DCP8, Triton KDA prefill, FlashInfer MLA, prefix caching enabled. 64 requests per workload, 1 output token, seed 42; cache reset before every workload. Latest completed paired round shown: round 2 except 128K/c16 (round 1). Off disables Triton's checkpoint capability (num_prefill_checkpoint_blocks=0, prefill_checkpoint_alignment=None) on the same source.

Input Concurrency Off mean TTFT (ms) On mean TTFT (ms) Reduction Off P95 (ms) On P95 (ms)
Random 8K 16 3593.4 3339.9 7.1% 4191.7 4213.7
Random 32K 16 10121.3 8999.8 11.1% 12524.3 12221.3
32K prefix + 2K suffix 1 1662.5 1343.2 19.2% 2179.3 1812.3
32K prefix + 2K suffix 16 8362.0 7141.1 14.6% 12874.5 11253.9
128K prefix + 2K suffix 1 3656.0 2790.1 23.7% 6051.7 5584.3
128K prefix + 2K suffix 16 28484.1 19156.9 32.7% 46044.2 35347.5

1,472/1,472 requests in completed workloads succeeded. Single-token outputs differed in 6/704 A/B pairs; repeat-to-repeat differences were 3/320 (off) and 4/384 (on).

Signed-off-by: Canlin <canlinguosdu@gmail.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@gcanlin

gcanlin commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

@ZJY0516 @yewentao256 Because FlashKDA has the accuracy problem, we're using triton KDA temporarily so migrate the optimization for TTFT from FlashKDA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant