[positive666]

Overview

Tune recurrent KDA launch configuration on SM90 while preserving the merged grid-indexing fix. Local kernel measurements improve most tested shapes, but two H20 TP8 serving experiments do not establish a stable C32 E2E benefit.

Claims

• H20 microbenchmark: 11 of 12 tested shapes improve 1.41–20.86%; N32/T1/H16 regresses 0.92%.
• Complete recurrent correctness suite: 23 passed, no skips.
• No general serving acceleration is claimed. The latest A0/K/A1 C32 medians are 755.981 / 755.571 / 756.372 output tokens/s.

Validation

The experiments, source identities, commands, negative results and limits are retained below. The 2026-10-25 follow-up covers 1,173 total requests with 36/36 natural-stop gold answers correct; MTP, native APC and external KV transfer were disabled. It is not pooled with the earlier stock benchmark.

Details

This remains a local-kernel optimization proposal, not an accepted production performance improvement. Open related work #63544 integrates FlashInfer fused KDA on Blackwell; at head 81a449c29f72262941454038450fe78f0c612fd3, its decode guard requires SM100-family hardware and its new prefill path permits SM100/103/107. It does not replace this H20/SM90 implementation. Narrow duplicate searches were refreshed on 2026-10-25; they are not proof of exhaustive absence of overlap.

Full validation, reproduction commands and limitations

Purpose and source

Tune GLM-5.3-Flash recurrent KDA launches on SM90: BV=min(next_power_of_2(V),16), two pipeline stages for small grids (N * HV <= 32), three for larger grids and one warp. Other architectures retain their existing configuration. The merged grid-indexing fix from #56974 is preserved.

Published head: 6c021edc38f49ede76a4ae5ebcf10d8c69f97f1f, an equivalent replay on fixed main 32551991b9aff4d8af837d640e3adcd9609f7d75. The H20 microbenchmark supports a local kernel benefit, with a counterexample; the latest standalone standard serving run does not establish a stable C32 end-to-end gain.

Retained baseline trace attribution (2026-10-26)

CPU-only reanalysis of the original eight-rank 2026-10-23 diagnostic traces (96 requests, C32, MTP/APC/external cache off) links all 371 CUDA Graph launches per rank to unique same-thread execute scopes and explicit GPU correlations. All 12,614 recurrent KDA calls per rank belong to these generation graphs, 34 calls per graph; the 28 mixed and one context step contain none of this target symbol. Raw hashes, the original CPU schedules and target totals match the prior projections. The actual historical post-load records verify 8 local heads and head dimension 128 on all ranks; the pinned image source verifies grid=(N*HV,NV,NK).

The actual launch sequence, including CUDA Graph padding, is identical across ranks:

Effective launch N	Graphs per rank	Share
1	2	0.54%
8	1	0.27%
16	2	0.54%
24	5	1.35%
32	361	97.30%

Only 2/371 baseline shapes satisfy this PR's N*HV <= 32 two-stage predicate. This is counterfactual branch eligibility, not a candidate execution trace. The wider BV=16 change would still apply to larger shapes. The dominant N32/T1/H8 microbenchmark below saves 0.224 microseconds per call (1.41%); the best 20.86% result is N1/T4/H8 and does not represent this MTP-off workload.

Across ranks, the median per-graph target-duration sum is 0.480–0.482 ms, while median graph GPU span is 15.315–15.326 ms. Summed target durations divided by summed graph spans are 3.103–3.125%. These are profiled baseline timing descriptors, not critical-path fractions, E2E speedup bounds, or candidate timing; rank durations are not combined. This narrows the opportunity for the current launch-only proposal and supplies no basis for another C32 stage-threshold sweep. MTP3 behavior remains outside these traces. No kernel code changed in this update. Retained analysis SHA256: 8c2a856abb4810c6f2ba77f37b0b8dd55a6562bf8cd003315f511272cc226064.

Follow-up serving validation (2026-10-25)

A second isolated A0 / K / A1 run on the FW1 image, retaining the MAIN79 runtime and the same MTP-off, APC-off, external-cache-off TP8/EP configuration, again found no stable E2E improvement. Each arm completed 391 requests: 12 natural-stop correctness questions (all passed), 67 warmups, and three blocks of 8 C1 plus 96 C32 performance requests. Performance inputs/outputs were fixed at 2048/128 tokens. No preemptions were observed. This smaller synthetic follow-up is separate from the stock benchmark below and is not pooled with it.

Output tok/s, median of three blocks	A0	K: this PR	A1
C1	111.036	111.136	111.158
C32	755.981	755.571	756.372

At C32, K/A0 is -0.05% and K/A1 is -0.11% by this follow-up's block-median definition. The all-token/all-wall sensitivity calculation gives -0.247%/+0.610%, with -0.851% baseline drift; the slow first A1 block is retained. C32 p95 first-output latency is +0.20%/+3.13% versus A0/A1. This is not a statistically established benefit. The exact candidate source was verified on all eight ranks; candidate-specific recurrent Triton variants were generated, including the two-stage variant. That verifies generation, not per-layer critical-path attribution. An initial A1 readiness-check timeout and the subsequent successful readiness check were retained before its formal measurement.

This does not change the local-kernel evidence below, but does not justify adding this patch to the measured production configuration or continuing launch-parameter sweeps without a new hypothesis.

Standard end-to-end measurement (2026-10-23)

8x H20, GLM-5.3-Flash FP8 checkpoint, TP8/DP1/EP8, max_num_seqs=32, batch-token budget 8192, long-prefill threshold 2048, BF16 KV with 30 GiB/GPU, context 1M. MTP, native APC and external KV transfer were disabled. All arms used the same MAIN79 (79d5c9379) runtime plus the existing #62479 producer, model/tokenizer bytes, dependencies and auto backend; #62603 was absent. Only the relevant PR's Python file was overlaid for each candidate. This validates those source overlays, not a rebuilt current-main binary.

Stock vllm bench serve was used with fixed seeded custom JSONL inputs produced using its RandomDataset, --save-detailed, --request-rate inf, temperature 0, --ignore-eos, fixed 128-token output, --ready-check-timeout-sec 0 and --num-warmups 0. Primary: three 512-request C32 blocks, each 2048 input tokens. Reference: 32 requests at C1. Secondary: three 128-request C32 blocks, a fixed 3:1 mix of 2048/32768-token inputs. Separate validated shape warmups (2 C1, 64 C32 and 64 mixed requests) preceded formal measurement. Each arm also passed the same 12 synthetic gold questions, including 8K/32K inputs, with a separate natural-stop 4096-token cap.

The prespecified primary is sum of reported output tokens / sum of all three stock durations, not the median block or a sum across different workloads. All blocks, pauses and tails are retained. Each arm completed all 1952 formal requests with zero failures and matching prompt/generation/request counters, cache-hit zero and MTP disabled. Input generator/tokenizer fingerprints matched across arms. A configured torch profiler was inactive during formal measurement; separate diagnostic traces are excluded.

The fixed actual order was A0b -> P -> K -> A1, not randomized or adjacent independent A/B pairs; this table displays only this PR's candidate and the two shared baselines. Initial compiler caches were separate copies from stopped owned runs: A0b from the earlier standard baseline, P from its earlier single-PR run, and K from an earlier baseline run. A1 copied A0b's cache after A0b completed formal measurement and its separate profile. Only seven approved compiler-cache directories were copied, not KV/model state, result files or private venvs. Source hashes and common model settings were verified, but equal initial JIT-variant coverage is not proven.

Arm	Primary C32 output tok/s (blocks 0 / 1 / 2)	Pooled primary	C1 reference	Mixed C32 blocks 0 / 1 / 2 (pooled)
Baseline before	750.823 / 730.712 / 752.492	744.543	110.467	162.091 / 162.069 / 162.052 (162.071)
K: this PR only	751.344 / 753.063 / 750.250	751.551	110.316	162.254 / 158.641 / 162.475 (161.104)
Baseline return	751.397 / 752.103 / 751.518	751.673	110.203	161.476 / 162.333 / 162.284 (162.030)

K versus the before / return baseline: primary +0.941% / -0.016%, C1 -0.137% / +0.102%, mixed -0.596% / -0.571%. The return-baseline comparison is essentially flat at primary C32; mixed throughput regressed. The local kernel gain must not be presented as stable serving acceleration.

Pooled primary C32 request latencies, with every block retained:

Arm	TTFT p95 (s)	TPOT p50 / p95 (ms)	E2EL p99 (s)
Baseline before	2.075338	30.752 / 40.422	7.831840
K: this PR only	2.050587	31.334 / 39.019	6.796972
Baseline return	2.063535	30.732 / 39.343	6.791837

K's mixed block 1 includes an approximately 2.48-second shared pause. A same-window HC cache-key/size/timestamp pattern was observed, but K artifact contents were not hashed: neither identical content nor compilation causation is established. No pause time is subtracted.

Stock TTFT can include an empty first choices chunk; ITLs are SSE choices-chunk gaps, not individual-token timing. Detailed raw JSON, CLI stdout/stderr, counter acceptances and all block durations are retained. The stock result does not independently expose each request's server token IDs or finish/DONE/usage provenance. Three blocks within one sequential run are not independent experiments or confidence intervals. There is no SLA or statistically established general serving speedup. The return baseline reused a mature owned compilation-cache copy, so the before/return difference is not attributable solely to hardware drift. The approximately 2.4-second pauses in the first baseline and P run remain included; a same-window HC compilation artifact is a diagnostic clue, not established causation.

The executed stock CLI form below uses archived fixture/output paths supplied through variables; these are not new upstream benchmark targets:

uv run --no-project --offline --no-sync --python .venv/bin/python .venv/bin/python -B -m vllm.entrypoints.cli.main bench serve \
  --backend openai --host 127.0.0.1 --port 28401 --endpoint /v1/completions \
  --model "$MODEL" --tokenizer "$MODEL" --served-model-name GLM-5.3-Flash \
  --dataset-name custom --custom-dataset-path "$DATASET" --custom-output-len 128 \
  --skip-chat-template --no-oversample --disable-shuffle --request-rate inf \
  --num-prompts 512 --max-concurrency 32 --num-warmups 0 --ready-check-timeout-sec 0 \
  --ignore-eos --temperature 0 --seed 20261023 --trust-remote-code \
  --save-result --save-detailed --result-dir "$RESULT_DIR" --result-filename "$RESULT_FILE" \
  --disable-tqdm --percentile-metrics ttft,tpot,itl,e2el,client_queue_time --metric-percentiles 50,95,99

Correctness and tests

The complete existing recurrent suite imported the actual installed candidate module and passed 23 tests, 0 failures/errors/skips on H20. Output and full recurrent state were checked. Original result SHA256: 6a18ee83163d491fec7db3e8672226a637ab1114823ea50fc033c5b23ae7f477. Runtime: PyTorch 2.14.0+cu130, Triton 3.8.0, FlashInfer 0.7.2, pytest 9.1.1; native/dependency files were unchanged. The executed isolated UV runner invoked this complete suite with frozen conftest/rootdir/JUnit settings; an earlier lazy-SM90-cache initialization error stopped that runner before pytest and is retained as a harness failure.

Reviewer command in a configured checkout:

uv run --no-project --offline --no-sync --python .venv/bin/python .venv/bin/python -B -m pytest tests/models/glm5next/test_kda_recurrent.py -v

Applicable changed-file hooks, including Python 3.12 mypy, passed. The Windows test-tethering encoding failure and successful UTF-8 retry are retained.

Microbenchmark and earlier serving limits

Historical unchanged-patch H20 microbenchmark: BF16 inputs, FP32 recurrent state, K=V=128, N=1/4/32, T=1/4, H=8/16; strict CUPTI graph replay/cold L2, untimed warmup, three ABBA blocks. All 12 shapes matched the default launch bit-for-bit and passed reference checks. Eleven improved 1.41-20.86%, but N32/T1/H16 was 0.92% slower in every block. No universal SM90/all-shape benefit is claimed. The micro harness is separate from the complete pytest suite and is not a repository benchmark added by this PR. Result SHA256: ae8a46fd8293f898ca78eb929bb9125a4de23b3db9253c5ceb404d8e44c956ca.

A prior single checkpoint A/B combined #57585 with #57617 and MTP3: C1/C32 throughput +2.51%/+1.89%, but C32 first-output median/p95 worsened 11.54%/0.57%, and MTP acceptance/output hashes varied. That experiment did not isolate this PR or establish a serving benefit and is not pooled with the new standard benchmark.

Related work and disclosure

Narrow open-PR/body searches refreshed on 2026-10-23 found related work, not another implementation of this SM90 launch change: #56974 is the merged grid-correctness fix; #58377 targets AMD Kimi, #52364 another model, #62778 final-state allocation, #61814 non-power-of-two gate-load masking, and #61662 an opt-in ROCm MI355X monolithic decode layer. Search results are not proof of exhaustive absence of overlap. This updates the existing PR.

AI assistance was used for implementation, experiment preparation and analysis; the human author remains accountable for reviewing every changed line and the tests. The recorded published-head pre-run admission, DCO, pre-commit and documentation checks passed, while review remained required. No CI admission label or rerun was requested; these checks do not establish complete GPU CI or maintainer approval.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill.
• AI assistance was used during the creation of this PR.
• Design Fit: Local SM90 launch change; no new serving configuration or core scheduling changes.
• Testing and Validation: Correctness and measured workload coverage are documented, including negative E2E results and the slower microbenchmark case.
• Code Quality and Style: Existing source and changed-file checks remain unchanged by this evidence update.
• Pull Request Contents: Claims, commands, source identity, comparison limits and AI disclosure are explicit.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[positive666]

Reworked: split the paired gate-projection change out to a follow-up PR; large grids now keep the default num_stages=3 so non-H20 SM90 parts see no untuned stage change (the gain comes from BV=16); added a bit-identity test locking the SM90 dispatch against the default config.

[NolenLiang]

Thanks for noting the relation to #56974 — agreed the two changes are orthogonal, and they compose: with this branch's two files normalised to LF, a three-way merge with #56974 applies without conflicts and both new tests in test_kda_recurrent.py survive.

One thing to fix before merge: both files in this PR are converted to CRLF line endings (kernels.py has 1399 \r, the test file 208). That is why the diff shows the whole files rewritten (+1399/−1377 and +208/−186) instead of the ~30 changed lines, and it will conflict textually with any other change to these files — including the launch-grid change in fused_recurrent_kda_fwd and the added test in #56974 — whichever merges second. Re-saving the two files with LF endings should shrink the diff to the actual change.

[positive666]

Thanks ~. I've normalized both files to LF.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @positive666.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @positive666.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[positive666]

Thanks @qianlihuang for reviewing!

I've rebased onto current main (38f0e37), preserving the launch-grid fix and regression test from #56974. The SM90 tuning and bit-identity test remain unchanged, and local pre-commit checks pass. GPU tests and benchmarks have not been rerun after this rebase.