[zcnrex]

Independent PRs

The first three independent changes are split into draft PRs, each based directly on main:

• [Perf] Enable fast-math for JIT activations on Blackwell #43236 — small-batch BF16 SiLU launch tuning on SM103.
• [Perf] Fuse BF16 writes to native HND KV cache storage #43237 — BF16 writes to native HND KV cache storage.
• [Perf] Route BF16 mixed-batch decode tails through decode attention #43238 — mixed-batch BF16 decode-tail routing.

These PRs do not depend on one another. This aggregate PR still contains their changes and the remaining work; the split PRs are the focused review units for those three changes.

Motivation

Qwen3-VL-4B's small-batch BF16 projections and separate QK normalization/rotation kernels add decode overhead on Blackwell. Mixed prefill batches also send one-token decode tails through context attention. At concurrency 128 in the measured workload, that routing reduces throughput from 3765 to 3201 output tokens/s when mixed chunking is enabled.

Modifications

• Tune the SM103 BF16 SiLU-and-multiply launch for hidden width 9728 and 1–128 tokens. Use 128 threads with four elements per thread through 32 tokens and eight thereafter, preserving precise arithmetic, BF16 rounding, PDL and existing fallbacks.

• Fuse the batch-one, batch-two, batch-four and batch-eight BF16 QKV GEMM epilogue with Q/K RMSNorm, multimodal RoPE and HND page-32 cache writes on SM103. Preserve the original GEMM BF16 round, norm/rotation rounding and PDL dependency behavior. The shared GEMM hook is optional; other shapes, execution modes and existing epilogues retain their paths.

• Autotune the batch-four and batch-eight BF16 vocabulary projections for the tied Qwen3-VL weight on SM103 at TP1. Use the public FlashInfer cuBLASLt API at startup, verify cached tactics before dispatch, retain existing fallbacks and preserve downstream logits processing. Readiness now includes the row count so existing M128 projections remain separate.

• Pack cached-prefix and fresh BF16 K/V for eligible SM103 prefill batches and run FA4 with a per-call CLC scheduler override. Reuse the existing pack kernel, retain TRT decode tails, bound scratch space to 128 MiB, and fall back for unsupported shapes, streams, graph capture and execution modes. Default FA4 scheduler behavior remains unchanged.

• Use the existing BF16 direct GEMM for the measured single-row gate/up projection on SM103 (M1/N19456/K2560). Guard shape, backend, dtype, layout and inference mode, preserve output buffers and fallbacks, and warm the kernel before graph capture.

• Autotune the M128 BF16 gate/up and down projections on SM103 through the public FlashInfer cuBLASLt API. Verify cached tactics before enabling dispatch, preserve BF16 input/output and caller-owned output buffers, and retain existing fallbacks.

• Add 13 measured BF16 split-K tactics: QKV, output and down projections at batches 1/2/4/8, plus the batch-16 down projection.

• Initialize the BF16 backend in sglang.benchmark.one_batch. This runner bypasses the scheduler's initialization; without these two lines it benchmarks a different GEMM path from serving.

• Route trailing one-token requests in ordinary BF16 causal prefill batches through decode attention, retaining context attention for the prefill prefix and preserving output-buffer reuse.

• Fuse BF16 QK RMSNorm and full-width NeoX multimodal RoPE for batches up to 128 tokens on Blackwell using the existing QK normalization warp kernel and an optional rotary/cache epilogue. Keep the original normalization path as the default; remove the separate standalone CUDA kernel and Python wrapper. Preserve the BF16 rounding between normalization and rotation and retain existing fallbacks for unsupported configurations.

• Fuse the existing QK normalization/rotation with BF16 HND cache writes during ordinary decode. Restrict this path to the tested page-32 TRT backend and plain cache pool; preserve physical-slot checks and cache-transfer synchronization.

• Support native HND strides in the fused cache writer. The existing opt-in HND layout uses it for BF16 CUDA writes; other dtypes and compiled execution retain their previous writer.

Weights, KV cache and attention queries remain BF16 in the evaluated configuration. Serving defaults and the default cache layout are unchanged. The optional deployment recipe is in #42914.

Validation

• Shared-kernel scheduling: retain the existing normalization engine and shared per-head body, but use a single guarded invocation for the nonpersistent rotary path and compute independent addresses before the PDL wait. Generated code removes nine collective synchronization pairs and duplicated alternate-path loads. Cache-writing kernel latency (microseconds, previous shared → optimized shared) is B1 2.1621→1.7080, B8 2.2829→1.8313, B16 2.3422→1.8885, and B128 3.7206→3.4314. The former standalone kernel measures 1.7074/1.8307/1.9420/3.4171 respectively. Most of the consolidation overhead is recovered; the interleaved B2 case remains 2.8% slower than the former standalone kernel. These are kernel measurements, not serving throughput gains. Serving validation of this final scheduling change was stopped at the author's request before launch.

• Shared-kernel correctness: 32 three-way standalone cases and eight K/V-alias cases pass bitwise, including changed-input graphs, both multimodal axis maps, negative slots and BF16 cache writes. All 560 existing normalization tests pass. Full checked model logits match bitwise for two seeds at B1/B16 (context 8192) and B128 (context 1024), with verified matching public GEMM tactics during startup; that external diagnostic override is restored before evaluation. The original normalization loop and early PDL wait remain the default. Validation scripts and evidence remain outside this code PR. Raw evidence for this scheduling update is retained locally; the linked published archive below covers preceding changes.

• Activation high-concurrency regression screen: C128 N128/warm128 normal-startup serving improves throughput by 0.230% and 0.189% (geometric mean 0.209%); mean TTFT changes 4688.49→4628.70 ms and 4607.25→4684.09 ms, while TPOT changes 28.586→28.564 ms and 28.570→28.416 ms. Both pairs select different public M128 down-projection tactics (control/candidate: 1/4 and 6/4), so these results cannot isolate the activation change. All runs complete exact request/token counts; every server scores 19/20 on the short GSM8K sanity check. This is not a full accuracy evaluation.

• Activation normal-startup serving: two same-GPU swapped pairs improve output throughput by a geometric mean of 0.797% at concurrency 1 and 0.266% at concurrency 8. C1 mean TTFT increases 73.52→76.10 ms and 75.02→80.45 ms, while TPOT decreases 2.352→2.333 ms and 2.362→2.335 ms. C8 mean TTFT decreases 364.46→362.36 ms and 360.13→358.84 ms, with TPOT 3.818→3.816 ms and 3.834→3.820 ms. These short screens use 8/40 measured requests and 2/8 warmups, exact 8192-input/1024-output token accounting, and normal public tuning. They do not establish a general TTFT improvement or statistical noninferiority.

• Activation launch: all 65,536 BF16 gate bit patterns across 11 multiplier cases match bitwise, including special values; changed-input graphs and boundary/fallback batches also pass. Existing activation tests: 452 passed, 6 skipped. Full-model checked logits match bitwise for two seeds at batches 1/8/16 (8192-token context) and 128 (1024-token context). The batch-128 check uses a startup-only external diagnostic to match verified public GEMM tactics, restored before evaluation; independent normal startups selected different M128 down tactics. Standalone activation latency improves from 1.9915 to 1.1967 microseconds at batch 8, 1.9934 to 1.2200 at batch 16, and 3.5934 to 3.0746 at batch 128. These are kernel measurements, not end-to-end throughput claims.

• Batch-one/two QKV extension: integrated standalone full QKV/cache/dependent-attention outputs match bitwise across all 36 layers, including changed-input stride128 graphs and all-valid cache slots. Batch-four/eight regression checks also match bitwise with sequence timing changes below 0.05%. The public model matches full checked logits and all generated tokens for two seeds at each new batch size; all 36 layers activate at eager/capture, and graph buffers remain unchanged. There are 27 guard checks and 67 existing namespace tests. One image request followed by two concurrent requests matches returned text, token/logprob objects and usage exactly, with 32 graph replays at each batch size after completed startup warmup; no MIXED forward or broad multimodal coverage is claimed.

• Full normal-startup GSM8K: C1 scores 1217/1314 in both arms with identical extracted answers/correctness (three response-text differences). C2 scores 1213/1314 candidate versus 1216/1314 control, with eight control-only and five candidate-only correct. C2 startup tactics match; the three-question loss remains an empirical limitation. A same-loaded-model diagnostic on four control-only prompts, in fixed pairs, then matches full prefill and all 512 decode-step logits bitwise and every greedy token. These steps cover the observed serving first-divergence tokens 97/116/264/219. The diagnostic reconstructs the exact previous eligibility method during baseline capture and restores public candidate dispatch with shared weights/tactics/inputs. It does not establish scheduling causality or broad accuracy equivalence.

• Batch-four QKV extension: 108 standalone layer comparisons pass bitwise against the existing production path and measured prototype, including changed-input graph replay. Existing batch-eight outputs also match bitwise; its measured sequence time changes by +0.045%. The public batch-four model matches full checked logits and all 64 generated tokens for each of two seeds; all 36 layers capture and shared graph buffers remain unchanged. Serving admission directly verifies positions stride128 for all 36 batch-four layers. Existing namespace tests: 67 passed. Four requests using one image fixture match text, returned token/logprob objects and usage; 24 batch-four graph replays were observed. Startup warmup overlapped this bounded image test, and it contains no MIXED forward.

• Full normal-startup C4 GSM8K for this extension scored 1220/1314 candidate versus 1217/1314 control: 10 control-only and 13 candidate-only correct, with 43 extracted-answer differences. Public M128 down tactics differ (control1/candidate4) and execute in 9/10 mixed forwards. This single result does not isolate a causal accuracy improvement or establish numerical equivalence across serving schedules.

• Graph-buffer integration: the QKV operator accepts row-strided multimodal positions from a larger decode graph buffer. Contiguous and stride-128 views pass 36-layer initial/changed-input QKV/cache/attention comparisons and 20 guard checks, including row-gap alias rejection. A real batch-eight model with graph cap 128 verifies eager warmup and capture of all 36 fused layers, bitwise checked logits/tokens for two seeds, and unchanged shared buffers. Earlier throughput crossovers used graph cap 8; they are not measurements of this larger-graph configuration.

• Full C8 GSM8K for QKV fusion scored 1218/1314 candidate versus 1220/1314 control with differing active M128 gate/up tactics. A separate matched-startup diagnostic scored 1213/1314 versus 1216/1314 (13 control-only and 10 candidate-only correct), with common tactics validated and the temporary startup override restored before evaluation. The net three-question loss is retained as an empirical limitation; no accuracy improvement or statistical equivalence is claimed. A fixed-batch diagnostic on eight control-only-correct disagreement prompts then matched full prefill and all 128 decode-step logits bitwise, with every greedy token identical, using the public production dispatch. This isolates the tested computation from serving batch formation but does not establish the cause of full-evaluation disagreement.

• Fused QKV preparation: all 36 actual-weight standalone QKV/cache/TRT attention outputs match bitwise initially and after changed-input graph replay, including distinct position axes, negative slots and caller-owned output. The public production model matches full checked logits and all generated tokens for two batch-eight input seeds. All 36 layers dispatch during capture. Existing shared-kernel/namespace tests: 77 passed; external guard probes: 27 passed (model-guard probes use isolated AST state). Eight concurrent requests using one existing image fixture match responses, token logprobs and usage exactly; this is bounded image coverage.

• Batch-8 production vocabulary dispatch: two fresh 8K input seeds retained bitwise-equal prefill, first/fifteenth decode logits and all generated tokens. Startup READY, actual capture and graph replay were verified on the audited two-file production overlay. Local AST-isolated probes covered 12 logits cases and 11 startup exclusions; these are metadata checks, not GPU tests.

• Full production C8 GSM8K scored 1219/1314 versus 1213/1314 with differing M128 down tactics. A separate matched-startup diagnostic using the same production sources scored 1225/1314 versus 1217/1314 (8 control-only and 16 candidate-only correct), with common M128/M4 tactics verified and actual M128 forwards observed. The temporary startup-only tuner override was restored before evaluation; candidate M8 tuning remained public and normal. This single comparison shows no aggregate regression, not statistical equivalence or a causal accuracy improvement.

• Batch-16 down projection: the standalone production dispatch selects the measured split-K tactic and matches the external winner bitwise, including changed-input graph replay. A real-text B16 probe confirmed all 36 down weights during graph capture. An external full GSM8K pair with normal startup tuning scored 1222/1314 versus 1213/1314, but used different existing M128 tactics. A separate matched-startup diagnostic scored 1220/1314 versus 1221/1314 (12 control-only and 11 candidate-only correct), with identical validated tactics and the temporary tuner override restored before evaluation. This is limited empirical accuracy evidence, not numerical or statistical equivalence.

• M4 vocabulary projection: 26 helper, 10 logits-branch and 11 startup/lifecycle checks passed (AST-isolated CPU probes). On the audited production source, actual optimized graph capture/replay was verified; prefill and first/fifteenth decode logits were bitwise equal, with all 64 generated tokens matching. Four image requests retained their 32-token greedy outputs; cross-server log probabilities differed by up to 0.101319 with different prefill grouping. A separate same-input diagnostic matched original and tuned BF16 vocabulary outputs bitwise on all 24 observed image decode steps. These bounded checks do not establish broad multimodal equivalence.

• Full C4 GSM8K for the vocabulary change scored 1218/1314 candidate versus 1216/1314 control (9 control-only and 11 candidate-only correct), with the same existing M128 tactics. Normal public autotuning was used, with no external algorithm replacement or private tuning override. This is a single paired accuracy check, not a claim of statistical or numerical equivalence.

• Packed prefill: existing layout/namespace tests passed (5 tests and 4 subtests), as did four existing SM103 FA4 cases. Standalone token-major/HND packing was bitwise correct, and default/explicit CLC dispatch checks passed. Two short mixed-model checks improved latency by 1.37–1.46% and retained four greedy tokens per request; logits are not bitwise equal.

• Full c128 GSM8K for packed prefill initially scored 1218/1314 control versus 1207/1314 candidate, with different M128 down-projection tactics. A separate startup-policy diagnostic held both GEMM tactics fixed, verified captured optimized dispatch, and scored 1217/1314 versus 1216/1314 (10 control-only and 9 candidate-only correct). This diagnostic used an external startup-only tuner override, not a production change. It recorded exactly 36 pack calls—one 36-layer context batch—so shared-prefix accuracy coverage is limited. These results do not establish accuracy equivalence.

• Additional M1 dispatch: 33 existing tests and 9 subtests passed. Three changed-input graph replays matched the direct kernel bitwise; normalized RMS error versus Torch was 7.15e-5–1.20e-4. A repository-image smoke test retained all 32 greedy tokens, with maximum selected-logprob difference 0.015082. This is one image fixture, not broad multimodal accuracy validation.

• Full concurrency-one GSM8K scored 1217/1314 for the M1 candidate versus 1219/1314 for control. A GPU-swapped production repeat reproduced the same 14 control-only and 12 candidate-only correct answers. This is a reproducible net loss of two questions (0.15 percentage points), not numerical equivalence or run noise. Weights, KV cache and attention queries remain BF16.

• Additional M128 dispatch: 33 existing tests and 9 subtests passed, along with GPU fallback guards and graph-capture/replay checks. Two full GSM8K runs scored 1213/1314 and 1223/1314, versus controls 1217/1314 and 1219/1314 with GPUs swapped (concurrency 128). Equal aggregate scores do not establish numerical or statistical equivalence.

• All 12 tuned GEMM shapes passed prior numerical validation: 36 shape/seed cases, normalized RMSE approximately 0.00165, matching ordinary BF16 GEMM; cosine similarity at least 0.99999994.

• QK norm/MRoPE: 24 distinct GPU cases were bitwise equal, repeated after source formatting. Shapes cover tokens 1/8/128, Q heads 4/32, KV heads 1/8 and both multimodal axis layouts.

• Mixed attention: 18 cases covering heterogeneous prefill lengths, one-token tails and output aliasing passed; maximum absolute difference 0.001953125.

• Cache writes: 20 layout/negative-slot/sliding-window/noncontiguous cases and six dtype/scale regressions passed.

• Existing CPU namespace/fused-op/dispatch tests: 123 passed. Repository pre-commit checks passed.

• Full text GSM8K: fusion plus mixed routing 1216/1314 (92.54%), versus BF16 reference 1215/1314 (92.47%). Optional HND configuration: 1220/1314 (92.85%). Five-shot dataset-order protocol, temperature 0, max output 2048, concurrency 32. These are single-run empirical checks.

• Image smoke: original main and candidate generated identical 32-token greedy descriptions from the repository image fixture. The trace confirmed the fused kernel executed during image decode. Selected-token logprobs differed by up to 0.04090; this checks basic multimodal behavior, not multimodal accuracy equivalence.

The additional cache fusion scored 1217/1314 and 1222/1314 in two full GSM8K runs, versus 1222/1314 and 1221/1314 for the preceding HND implementation with GPUs swapped. These empirical checks show run variation and do not establish statistical accuracy equivalence.

Additional cache fusion validation: 44 standalone GPU cases were bitwise equal across cache layouts, token counts, tensor-parallel head counts and multimodal axis layouts. All 24 same-GPU greedy response pairs retained identical tokens; maximum selected-logprob difference was 0.0000224. Existing CPU regression rerun: 122 passed, one skipped.

Validation scripts and benchmark artifacts are kept outside this code PR.

Performance

Measured snapshot of exact PR commit c76cfaba1a: one B300 per point, BF16 weights/queries/KV, HND page32, mixed chunk16384, explicit20 decode buckets through128, KV capacity1,600,000, serverseed0/clientseed42. Original N30/40/80/80/160/320/640 counts and max(64,C) warmups followed by flush are preserved. All exact counts and before/after10,245-file manifests pass. All36 fused layers activate at B1/B2/B4/B8 with stride128 positions.

Comparison baseline: original SGLang f70e8c68fc from the 2026-10-07 handoff, using FlashInfer attention, page size 1 and mixed chunking disabled. Baseline numbers are rounded as recorded in that handoff. The measured snapshot combines code and configuration changes, including TRT attention, HND page 32 and mixed chunking; this historical comparison does not isolate code-only gains. The snapshot predates subsequent activation and shared-kernel scheduling changes and is not a measurement of the latest PR head. No new serving validation was run.

Concurrency	Baseline output tokens/s	PR snapshot output tokens/s	Throughput change	Baseline median TTFT ms	PR snapshot median TTFT ms	Baseline median TPOT ms	PR snapshot median TPOT ms
1	329	411.63	+25.12%	122	85.58	2.9	2.347
4	962	1276.59	+32.70%	322	254.64	3.8	2.889
8	1497	1907.64	+27.43%	515	380.05	4.8	3.828
16	2076	2582.38	+24.39%	901	635.04	6.8	5.585
32	2613	3139.82	+20.16%	1672	808.90	10.6	9.363
64	2896	3540.85	+22.27%	3244	687.28	18.9	17.311
128	3043	3832.18	+25.93%	6328	599.64	35.9	32.684
128 (second GPU)	3043	3845.56	+26.37%	6328	591.80	35.9	32.532

Primary C128 mean/p95/p99 TTFT is1475.70/6803.68/8642.01 ms. The two C128 workers select different public M128 down tactics4/6 and differ in throughput by0.349%. These concurrent distinct-GPU measurements describe a pinned configuration, not causal per-change attribution or a repeatability bound. Full per-request arrays, telemetry and exact commands are retained in results/final_qkv_full_sweep.

Batch-one/two QKV fusion: two same-GPU C1/N30 pairs improve throughput by 4.897% and 4.814% (geometric mean 4.856%), from 391.50/394.36 to 410.67/413.34 output tokens/s. Two C2/N40 pairs improve by 4.143% and 4.227% (geometric mean 4.185%), from 716.63/716.83 to 746.32/747.12. All exact counts pass after 64 warmups and flush, with uniform cap128 graph buckets, KV capacity 1,600,000 and normal startup. No M128 forwards execute during warmup or measurement despite different startup tactics. C1 mean TTFT increases 1.675/0.071 ms; median changes +4.372/-1.957 ms. C2 mean TTFT increases 2.315/1.096 ms; median changes -0.446/+0.667 ms. These results support throughput gains, not a general TTFT improvement. Median TPOT falls from 2.475/2.457 to 2.355/2.341 ms at C1 and 2.675/2.676 to 2.563/2.562 ms at C2.

Batch-four QKV fusion, public production dispatch: two same-GPU C4/N40 crossover pairs improve output throughput by3.641% and3.921% (geometric mean3.781%). Control/candidate rates are1235.13/1280.10 and1249.68/1298.68 output tokens/s. Mean TTFT changes200.90→198.59 and202.22→201.37 ms; median TTFT245.58→245.43 and239.88→239.78 ms; median TPOT2.997→2.891 and2.967→2.855 ms. All exact request/token counts pass after64 warmups and cache flush, with graph cap128, explicit20 buckets, KV capacity1,600,000 and normal startup. M128 tactics differ but no M128 forwards execute during either warmup or measurement. The standalone QKV/preparation/dependent-attention sequence improves7.834%; these microbenchmarks are separate from the serving measurement.

• Integrated QKV fusion: two same-GPU C8/N80 crossover pairs improve throughput by 1.021% and 1.122% (geometric mean 1.071%). Rates are 1895.15 → 1914.50 and 1912.55 → 1934.01 output tokens/s. Mean TTFT changes 356.06 → 355.40 ms and 353.46 → 339.37 ms; median TTFT changes 345.16 → 379.22 ms and 370.68 → 366.01 ms. This establishes a measured throughput gain, not a general TTFT improvement. Runs use 64 warmups, cache flush, nominal 8192/1024, graph cap 8, verified KV capacity 1,600,000, normal public tuning and exact request/token counts. No M128 forwards execute during measurement. Per-request timings are retained. The full 36-layer standalone dependent-attention sequence improves by 1.393%.

One B300 (SM103), TP1, Qwen/Qwen3-VL-4B-Instruct, BF16 weights/KV/queries, TRTLLM MHA, page size 32, chunk size 16384. Torch 2.14.1+cu130 and FlashInfer 0.7.0.post1. Implementation base: upstream 43b4abd plus the changes in this PR.

• Production M16 down tuning: two same-GPU c16/N80 crossover pairs improved throughput by 0.523% and 0.364% (geometric mean 0.443%). Control/candidate output tokens/s were 2562.44/2575.85 and 2555.87/2565.17. All runs completed exact request/input/output counts after 64 warmups and a cache flush. Median TTFT changed 599.14 → 601.58 ms and 601.04 → 607.43 ms; TPOT changed 5.65738 → 5.62964 ms and 5.70315 → 5.64546 ms. The one-file production change used normal startup tuning and the existing dispatcher; observers verified actual split-K capture and measured decode execution, with no runtime M128 forwards. The common control already includes M8 vocabulary tuning.
• Production M8 vocabulary tuning: two same-GPU c8/N80 crossover pairs improved throughput by 0.335% and 0.302% (geometric mean 0.318%). Candidate rates were 1894.24 and 1907.46 output tokens/s. All four runs completed exact request/token counts after 64 warmups and a cache flush. Median TTFT changed by −11.41% and +3.95%; TPOT changed by +0.20% and −0.69%. Existing M128 and M4 tactics matched across all runs, with normal public startup tuning and no external algorithm replacement.
• Production M4 vocabulary tuning: two same-GPU c4/N40 crossover pairs improved throughput by 0.160% and 0.221% (geometric mean 0.191%). Candidate rates were 1240.50 and 1246.50 output tokens/s. Each run used 64 warmups followed by a cache flush and completed exact request/input/output counts. Median TTFT changed 240.83 → 215.67 ms and 213.04 → 209.95 ms; median TPOT changed 3.00026 → 3.00554 ms and 3.01603 → 3.00015 ms. Two pairs do not establish a general latency improvement. A complete post-flush prefill-row audit excludes M128 execution during measurement, so differing startup M128 down-projection tactics do not affect these serving intervals.
• Packed prefill production serving: four same-GPU c128/N640 crossover pairs improved throughput by a mean of 0.446%, with all four positive (+0.398% to +0.479%). All eight runs completed exact request/input/output counts; the two GEMM tactics matched within each pair, and no external dispatch hook was used. Paired median TTFT changes were -1.83%/-1.74%/+21.94%/-1.70%; no general TTFT improvement is established.
• Packed-prefill lower-concurrency checks used the exact measured request counts but reduced warmups equal to concurrency, followed by cache flush. Single same-GPU pairs at concurrency 1/4/8/16/32/64 changed throughput by +0.101%/+0.189%/+0.237%/+0.355%/+0.538%/+0.433%. Median TTFT increased 7.5% at concurrency 4, 12.5% at concurrency 32 and 22.0% at concurrency 64. These short checks do not establish statistical noninferiority or a latency improvement. At concurrency 64, mean TTFT changed 985.20 → 992.47 ms (+0.74%), while p90/p95/p99 improved; the 22.0% median increase reflects a distribution shift. Server logs show changed queue occupancy but do not establish its cause.
• Production M1 direct GEMM: four same-GPU c1/N30 crossover pairs improved output throughput by a geometric mean of 1.692%; all four pairs were positive. Candidate throughput was 390.93–393.82 output tokens/s. The descriptive paired bootstrap 95% interval was [+1.612%, +1.771%]. Mean run-median TTFT was 85.71 → 85.41 ms, but paired changes ranged from -10.50% to +3.95%; no general TTFT improvement is established. Fresh tuning caches and shared compiled kernels were used, without an external dispatch hook.
• M1-path regression checks at concurrency 4/8/16/32/64/128 changed throughput by +0.157%/-0.165%/+0.132%/-0.054%/-0.041%/-0.269%. These single paired checks used 40/80/80/160/320/128 measured requests respectively, with exact counts verified. No material regression was observed; these checks do not establish statistical noninferiority.
• Production M128 autotuning: eight same-GPU c128/N640 crossover pairs improved output throughput by a geometric mean of 0.336%; all eight pairs were positive. Mean throughput was 3809.82 → 3822.62 output tokens/s. Each worker used a fresh isolated tuning cache; compiled kernels were shared. The descriptive paired bootstrap 95% interval was [+0.280%, +0.403%], covering these two periods rather than repeated-day variation.
• Lower-concurrency production checks used the same-device swapped protocol and exact full-sweep request counts. Throughput changes at concurrency 1/4/8/16/32/64 were -0.075%/-0.116%/+0.208%/-0.061%/-0.056%/-0.024%. No material regression was observed in these single pairs; they do not establish statistical noninferiority.
• Selected cold-weight GEMMs: 1.26–1.30x faster. Short model decode: batch 1 3.126 → 2.987 ms, batch 4 3.333 → 3.201 ms, batch 8 3.694 → 3.588 ms.
• Representative mixed attention batch (8192 prefill tokens plus 128 decode rows): approximately 3.79 → 1.00 ms.
• Exact serving sweep at concurrency 128: mixed routing improves 3201.14 → 3777.27 output tokens/s; QK norm/MRoPE fusion increases this to 3801.87. At concurrency 1, fusion increases 375.15 → 383.81 output tokens/s.
• With the same fusion-enabled source, enabling mixed chunking changes concurrency-128 median TTFT from 4120.29 → 577.62 ms, throughput from 3763.54 → 3801.87 output tokens/s, and median TPOT from 29.97 → 32.95 ms. The lower TTFT comes with higher TPOT.
• Combining QK normalization/rotation with HND writes: eight-GPU same-device crossovers averaged +1.41%, +1.73% and +1.37% output throughput at concurrency 1, 4 and 8; all 24 pairs were positive. These short screens used 3/40/80 measured requests and 1/64/64 warmups. High-concurrency gains are much smaller.
• Production HND full-sweep ablation for the additional fusion: concurrency 1 379.67 → 383.44 output tokens/s, concurrency 8 1870.41 → 1886.09, concurrency 128 3790.59 → 3797.51. At concurrency 128, median TTFT was 611.07 → 610.25 ms. The gain is concentrated at low concurrency.
• Optional HND cache writes: approximately 10.2 → 1.45 microseconds at 64 tokens and 86.64 → 10.18 microseconds at 8192 tokens. These are writer microbenchmarks, not serving speedups.

Full sweeps used nominal 8192-input/1024-output text prompts at concurrency 1/4/8/16/32/64/128, respectively 30/40/80/80/160/320/640 measured requests, and max(64, concurrency) warmups followed by a cache flush. Each sweep completed all 1350 requests. Results are single sweeps; small differences can include run/GPU variation.

Raw measurements and source provenance.

Checklist

• Format code with repository pre-commit hooks.
• Provide numerical, accuracy and performance results.
• Keep benchmark/test additions out of the code PR.
• Upstream CI and maintainer review. Current CI gates require the repository's run-ci label; runtime tests have not run there.

---

CI States

Latest PR Test (Base): ⏳ Run #37855516054
Latest PR Test (Extra): ⏳ Run #37855515554
Latest PR Test (AMD ROCm 10): ⏳ Run #37855516098