(recipe) Kimi-K3 AgentX: FP8 prefill attention, PrefillDelayer from C16, capture from bs=1

[gbyu-amd]

What

This updates recipes/Agentic-Kimi-K3.md to the configuration of a 7-point AgentX sweep on 8×MI355X (C1/4/14/16/48/56/72, 3600 s each):

• ATOM_USE_FLYDSL_FP8_PREFILL_ATTN=1 on every band (FlyDSL FP8 prefill attention).
• ATOM_PREFILL_DECODE_INTERVAL=4 and ATOM_PREFILL_DELAYER_MAX_QUEUE_MS=5000 on CONC ≥ 16 (the PrefillDelayer: 4 protected decode passes after each prefill, plus prefill coalescing bounded by a 5 s queue age). ATOM_ENABLE_PREFILL_DELAYER already defaults to 1, so these two variables are enough to construct the PrefillDelayer. The sweep checked this on every C ≥ 16 point: each one logged PrefillDelayer initialized once.
• CUDA-graph capture sizes start at 1 instead of 2: seq -s, 1, dense [1, …, graph_max].

The recipe changes are limited to these environment variables and launch arguments, plus the matching rows in the configuration table. Nothing else in the launcher or client changes. Every other value (DCP, max-num-seqs, LMCache size, comm-group reuse, ReplaySSM, spec, AIPerf flags and env) already matched the sweep.

Dependencies

Both dependencies are merged: #2378 (2026-09-24) and ROCm/aiter#5796 (2026-09-24). The sweep ran on ATOM d9f0720 + #2378, aiter 2887d4899 + #5796, and triton 3.7.0.

Results vs the current recipe's best measured runs

Deltas are relative to the best local run of the current recipe per band (reuse A/B-aligned). Bold marks a change outside the run-to-run band measured with 5 same-config c64 runs: tok/s ±4.7%, ITL ±6.2%, TTFT ±11.7%.

CONC	delayer	tot tok/s	out tok/s	TTFT p50	TTFT p90	ITL p50	ITL p90	interactivity p90
1	–	+5.6%¹	+5.3%¹	−3.9%	+0.8%	−8.5%	−12.4%	+14.3%
4	–	−0.1%¹	−1.9%¹	−0.7%	−6.3%	−6.3%	−13.3%	+15.4%
14	–	+4.0%	+4.8%	−10.3%	−19.3%	−2.3%	−1.8%	+1.9%
16	✓	+3.9%	+3.3%	+91.7%	+32.9%	−3.2%	−12.3%	+13.9%
48	✓	+9.7%	+10.4%	+36.8%	+18.9%	−13.7%	−15.0%	+17.6%
56	✓	+11.9%	+15.8%	+61.1%	+24.8%	−16.4%	−18.4%	+22.1%
72	✓	+14.7%	+17.1%	+67.3%	+89.1%	−16.0%	−16.0%	+19.0%

¹ At C1/C4 throughput is set by the trace's pacing: both runs served nearly the same requests. Latency is the signal there.

Absolute values of the new runs:

CONC	tot tok/s	per chip	out tok/s	TTFT p50 / p90 (ms)	ITL p50 / p90 (ms)
1	12,981	1,623	104.5	733 / 1,805	6.28 / 6.57
4	23,074	2,884	168.0	588 / 1,339	6.43 / 8.27
14	59,352	7,419	483.1	696 / 1,638	12.07 / 18.84
16	68,033	8,504	507.5	1,450 / 2,897	13.21 / 19.68
48	102,340	12,792	680.9	1,790 / 4,214	47.12 / 62.38
56	108,821	13,603	772.1	2,050 / 4,395	52.74 / 67.07
72	117,353	14,669	837.5	2,492 / 7,496	74.43 / 100.40

Trade-off to be aware of

The PrefillDelayer protects decode by holding prefills, so TTFT rises wherever it is on. The delayer's own counters show the hold rate climbing with concurrency: C16 31%, C48 38%, C56 51%, C72 51%. At C48–C72 this buys +10–17% output throughput and −14–18% ITL. At C16 the throughput change is inside noise, while TTFT p50 goes from 696 ms (C14, delayer off) to 1,450 ms. So the C ≥ 16 threshold is the sweep's setting, not a tuned one; C16 may be better served without it.

Not measured with the new knobs

C2, C8, C12, C32, C40, C64 and C80 follow the same rules (FP8 prefill everywhere, PrefillDelayer from C16 up) but were not re-run in this sweep. The effects above cannot be split per knob either: the ATOM/aiter changes above, FP8 prefill, the bs=1 capture and the delayer all changed together.

How the recipe was checked

The launcher block from this PR and the sweep's server script were each executed up to the server start for all 14 bands, dumping exported env and argv, then diffed. The only remaining differences are default-equal (ATOM_USE_FLYDSL_GATHER_KV_B_PROJ=1 is the default; 0.90 vs 0.9) or logging-only (AITER_LOG_LEVEL). The AIPerf client, including its AIPERF_* env, is identical to what the sweep ran, apart from URLs and paths.

🤖 Generated with Claude Code

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2382 --add-label <label>