[positive666]

Purpose and source

Pair the BF16 f_b_proj / g_b_proj projections into one cached batched matmul. LoRA-configured models retain the original Linear path, including adapters installed after postload; non-BF16 weights also retain the fallback. Compatible postload weight refreshes update the cache in place to preserve CUDA-graph addresses. First load or incompatible shape/dtype/device allocates new storage and requires graph recapture.

Published head: 47895f99d0b9e52a850285e223f597327691e6a8 on fixed main 32551991b9aff4d8af837d640e3adcd9609f7d75. The final source has a substantial small-M paired-operation microbenchmark benefit, but the latest standalone standard serving run does not establish a stable C32 end-to-end gain.

Controlled follow-up, 2026-10-26 (local candidate; code not pushed)

A new A0 -> P0 -> P1 -> A1 cycle tests the paired projection with a local compile-compatible view (narrow -> unflatten -> transpose, source SHA256 c4b9ade1bfc00e33ecdd7dafdd43c0fb36cc482258a65babfd8bcf5c28118550). The published code head above remains unchanged. The view change fixes this PR's padded/nonzero-offset M=1 fullgraph regression: the old source gives 1 failure / 1 pass for the two added tests; the local full suite gives 9 passes. This is not evidence of a baseline upstream serving defect.

All four arms used the same fixed MAIN79 image and DeepGEMM #448 correctness extension, #60567 mHC warmup and the local NVIDIA KPool warmup. Each had a separate container and compile-cache copy, with the same initial seed and the relevant mHC/KPool entries removed before startup. All 16 mHC and 8 KPool keys existed at readiness, with zero new keys during requests. These common changes are not attributed to paired projection. The startup mnnvl all-reduce probe fell back to the same single-node trtllm backend in all arms; this is a condition of this measurement, not a complete communication optimization certification.

The same stock protocol described below was kept: 1952 formal requests per arm, C1 reference, three 512-request 2K/128 C32 blocks, and three 128-request mixed 2K/32K blocks. All 7808 formal requests, 48 natural-stop gold questions, and 384 candidate real-weight finite/bitwise output comparisons passed. The 520 workload warmup requests and 32 first/repeat probes are checked separately and excluded from formal aggregates. All formal durations and tails are retained.

Arm	Primary C32 output tok/s	Primary TTFT p95 (s)	Primary E2EL p95 (s)	C1 output tok/s	Mixed C32 output tok/s
A0 baseline	744.548	1.957995	5.860297	110.180	160.739
P0 local candidate	747.045	2.089758	5.830634	111.399	160.890
P1 local candidate	746.509	2.098413	5.875038	111.393	160.906
A1 baseline return	745.658	2.090994	5.861095	110.206	160.786

Across the four candidate/baseline comparisons, primary throughput changes range from +0.114% to +0.335%, and primary TTFT p95 changes from -0.059% to +7.171%. Baseline A1/A0 drift is +0.149%. The prespecified throughput hurdle was >1.000% for each candidate against both baselines, with no >5% TTFT/E2EL p95 regression in any workload. The gate was not met; a stable comprehensive serving gain is still not established.

Model compilation mode was NONE, with CUDA graphs enabled; the standalone fullgraph tests are separate evidence and do not certify whole-model piecewise compilation. The shared native extension differs from older experiments, so old/new percentages are not a one-factor estimate of warmup's effect. This is one ordered ABBA cycle, not independent-trial significance, a broad model evaluation, latest-main binary acceptance or production TP16/cache integration. The local compile fix is not published as part of this evidence-only update.

Earlier published-head standard end-to-end measurement

8x H20, GLM-5.3-Flash FP8 checkpoint, TP8/DP1/EP8, max_num_seqs=32, batch-token budget 8192, long-prefill threshold 2048, BF16 KV with 30 GiB/GPU, context 1M. MTP, native APC and external KV transfer were disabled. All arms used the same MAIN79 (79d5c9379) runtime plus the existing #62479 producer, model/tokenizer bytes, dependencies and auto backend; #62603 was absent. Only the relevant PR's Python file was overlaid for each candidate. This validates those source overlays, not a rebuilt current-main binary.

Stock vllm bench serve was used with fixed seeded custom JSONL inputs produced using its RandomDataset, --save-detailed, --request-rate inf, temperature 0, --ignore-eos, fixed 128-token output, --ready-check-timeout-sec 0 and --num-warmups 0. Primary: three 512-request C32 blocks, each 2048 input tokens. Reference: 32 requests at C1. Secondary: three 128-request C32 blocks, a fixed 3:1 mix of 2048/32768-token inputs. Separate validated shape warmups (2 C1, 64 C32 and 64 mixed requests) preceded formal measurement. Each arm also passed the same 12 synthetic gold questions, including 8K/32K inputs, with a separate natural-stop 4096-token cap.

The prespecified primary is sum of reported output tokens / sum of all three stock durations, not the median block or a sum across different workloads. All blocks, pauses and tails are retained. Each arm completed all 1952 formal requests with zero failures and matching prompt/generation/request counters, cache-hit zero and MTP disabled. Input generator/tokenizer fingerprints matched across arms. A configured torch profiler was inactive during formal measurement; separate diagnostic traces are excluded.

The fixed actual order was A0b -> P -> K -> A1, not randomized or adjacent independent A/B pairs; this table displays only this PR's candidate and the two shared baselines. Initial compiler caches were separate copies from stopped owned runs: A0b from the earlier standard baseline, P from its earlier single-PR run, and K from an earlier baseline run. A1 copied A0b's cache after A0b completed formal measurement and its separate profile. Only seven approved compiler-cache directories were copied, not KV/model state, result files or private venvs. Source hashes and common model settings were verified, but equal initial JIT-variant coverage is not proven.

Arm	Primary C32 output tok/s (blocks 0 / 1 / 2)	Pooled primary	C1 reference	Mixed C32 blocks 0 / 1 / 2 (pooled)
Baseline before	750.823 / 730.712 / 752.492	744.543	110.467	162.091 / 162.069 / 162.052 (162.071)
P: this PR only	732.831 / 749.974 / 751.854	744.787	111.493	162.302 / 162.340 / 162.508 (162.383)
Baseline return	751.397 / 752.103 / 751.518	751.673	110.203	161.476 / 162.333 / 162.284 (162.030)

P versus the before / return baseline: primary +0.033% / -0.916%, C1 +0.929% / +1.170%, mixed +0.193% / +0.218%. C1 shows an approximately 1% observation in this MTP-off 2K/128 setting; the primary C32 comparison does not support a consistent net gain.

Pooled primary C32 request latencies, with every block retained:

Arm	TTFT p95 (s)	TPOT p50 / p95 (ms)	E2EL p99 (s)
Baseline before	2.075338	30.752 / 40.422	7.831840
P: this PR only	2.059919	33.661 / 40.199	7.790771
Baseline return	2.063535	30.732 / 39.343	6.791837

P's median TPOT is higher than both baselines; its p95 TPOT and p99 end-to-end latency are also worse than the return baseline. The C1 throughput observation does not override these C32 tradeoffs.

Stock TTFT can include an empty first choices chunk; ITLs are SSE choices-chunk gaps, not individual-token timing. Detailed raw JSON, CLI stdout/stderr, counter acceptances and all block durations are retained. The stock result does not independently expose each request's server token IDs or finish/DONE/usage provenance. Three blocks within one sequential run are not independent experiments or confidence intervals. There is no SLA or statistically established general serving speedup. The return baseline reused a mature owned compilation-cache copy, so the before/return difference is not attributable solely to hardware drift. The approximately 2.4-second pauses in the first baseline and P run remain included; a same-window HC compilation artifact is a diagnostic clue, not established causation.

The executed stock CLI form below uses archived fixture/output paths supplied through variables; these are not new upstream benchmark targets:

uv run --no-project --offline --no-sync --python .venv/bin/python .venv/bin/python -B -m vllm.entrypoints.cli.main bench serve \
  --backend openai --host 127.0.0.1 --port 28401 --endpoint /v1/completions \
  --model "$MODEL" --tokenizer "$MODEL" --served-model-name GLM-5.3-Flash \
  --dataset-name custom --custom-dataset-path "$DATASET" --custom-output-len 128 \
  --skip-chat-template --no-oversample --disable-shuffle --request-rate inf \
  --num-prompts 512 --max-concurrency 32 --num-warmups 0 --ready-check-timeout-sec 0 \
  --ignore-eos --temperature 0 --seed 20261023 --trust-remote-code \
  --save-result --save-detailed --result-dir "$RESULT_DIR" --result-filename "$RESULT_FILE" \
  --disable-tqdm --percentile-metrics ttft,tpot,itl,e2el,client_queue_time --metric-percentiles 50,95,99

Correctness and tests

7 H20 tests passed, 0 skipped, using real installed modules and constructor paths: four BF16 shapes, non-BF16 fallback, CUDA-graph replay after refit, and a nonzero LoRA adapter installed after postload. The graph test first produced 6 passes/1 failure against an earlier revision of this PR that replaced cache storage, then passed with the in-place storage fix. This fixes a regression introduced during this PR's development, not a newly discovered baseline upstream defect.

The serving candidate additionally passed 192 finite/bitwise comparisons after real checkpoint loading: first/middle/last of 34 KDA layers, 8 ranks, M=1/32/2048/8192, both outputs. These use real checkpoint weights with deterministic synthetic activations; they are not measurements of actual-request intermediate activations or whole-model bit identity. All arms' 12 synthetic gold answers passed; this is not a general model evaluation suite.

Reviewer command in a configured checkout:

uv run --no-project --offline --no-sync --python .venv/bin/python .venv/bin/python -B -m pytest tests/models/glm5next/test_kda_paired_projection.py -v

The actual isolated UV runners used frozen source/dependency hashes and the complete test file. Changed-file hooks, including Python 3.12 mypy in the pre-commit environment, passed; an additional wrong-environment mypy failure is retained separately. Native, LMCache and producer files remained unchanged.

Microbenchmark and earlier serving limits

Final-source H20 paired-operation measurement used real BF16 ColumnParallelLinear modules, strict CUPTI graph replay/cold L2, three ABBA blocks and untimed warmup. It measures the full call_fn GPU activity span, not CPU dispatch or whole-layer/model latency. Both outputs were bitwise equal before and after timing.

M	Two Linear GPU span (us)	Paired GPU span (us)	Time reduction
1	5.872	2.720	+53.678%
4	5.848	2.752	+52.941%
32	5.880	2.816	+52.111%
2048	17.424	17.720	-1.699%
8192	48.800	40.624	+16.755%

M2048 regressed in two blocks; M8192 improved in all three. These results do not predict compiled serving performance. Original micro result SHA256: e1106c548e0497dd430acc29f7a96fe71816bf94fc442b621318901050e8a2f1.

Historical custom-client MTP-off A0/P/A1 serving is retained separately: the P C32 block rates were 649.138 / 758.354 / 759.621 output tok/s. Pooling every block's output and full wall time gave P 718.460 versus baselines 754.070/757.330, -4.72%/-5.13%. Its predeclared block-median results were much closer (+0.61%/+0.06%), while p95 first-output and median TPOT worsened. These older custom-client definitions differ from the new stock benchmark; they are not replaced, pooled with it or used to discard its slow block. Neither episode establishes stable comprehensive serving acceleration.

Related work and disclosure

Narrow open-PR/body searches refreshed on 2026-10-23 found no direct replacement for paired f_b_proj/g_b_proj: #62956's SM121 skinny-GEMM scope excludes these projections; #56327 concerns LoRA mappings and #57638/#57678 loader/cache lifecycle work. #61662 is an opt-in ROCm MI355X decode layer. Search results are not proof of exhaustive absence of overlap. This updates the existing PR.

AI assistance was used for implementation, experiment preparation and analysis; the human author remains accountable for reviewing every changed line and the tests. The recorded published-head pre-run admission, DCO, pre-commit and documentation checks passed, while review remained required. No CI admission label or rerun was requested; these checks do not establish complete GPU CI or maintainer approval.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[qianlihuang]

These profiles were collected on my commit 0b874994c3.
Setup: 8× NVIDIA H200, GLM-5.3-Flash FP8 checkpoint with BF16 KDA projections, TP1/DP8/EP, DecodeBenchConnector, ISL131072/OSL512, concurrency128, MTP3 with synthetic unconditional acceptance rates [0.8, 0.6, 0.5].

baseline-dp0.pt.trace.json.gz
bmm-dp0.pt.trace.json.gz

The DP0 profile shows 34 fewer BF16-linear kernel launches per step (431 → 397, one fewer per KDA layer). Summed BF16-linear kernel time decreased from 5.457 to 5.363 ms/step, saving approximately 94 µs (~1.7% of this category).

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @positive666.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork