[AMD] [GLM5] Add opt-in PTPC FP8 KDA projections on gfx950

[Raiden-Makoto]

Motivation

GLM-5.3-Flash keeps its KDA qkv, low-rank f/g, beta, and output projections in BF16. The PTPC path repacks selected weights to per-channel FP8 and quantizes activations per token, but o_proj previously required a separate BF16 gated RMSNorm output followed by a standalone quantization kernel.

This change adds an opt-in no-copy packed PTPC prefill path while preserving the upstream fused BF16 projection modules for decode. Prefill uses one FP8 [qkv | f_a | g_a] GEMM plus BF16 beta and passes qkv/beta/fg directly to their consumers without reconstructing the fused output; decode remains one fused GEMM.

Modifications

• Preserve the default-off SGLANG_OPT_GLM53_KDA_PTPC_MODULES selector and reject it unless AITER is active on gfx950.
• Move GLM-specific repacking and FP8/BF16 dispatch into dedicated subclasses; leave UnquantizedLinearMethod tensor-in/tensor-out for every other model and backend.
• Preserve upstream fused_qkvbfg_a_proj and fused_bfg_a_proj construction instead of unfusing projections when PTPC is selected.
• Repack [qkv | f_a | g_a] into one FP8 weight, quantize hidden_states once, and return qkv/beta/fg views directly without a full-width output concatenation.
• Keep fused BF16 decode and fallback. Enable no-copy packed PTPC only for measured prefill windows: TP4 M=4096-16384, TP8 M=8192-16384.
• Reject unsupported fused qkv layouts with a named ValueError instead of a bare lookup failure.
• Add a gfx950 Triton bridge that performs independent 128-wide RMSNorm, applies the sigmoid gate, and emits FP8 directly for PTPC o_proj.
• Register the gfx950 kernel suites on the MI35X runner.

Enable with:

SGLANG_OPT_GLM53_KDA_PTPC_MODULES=qkv_proj,f_a_proj,g_a_proj,o_proj

Accuracy Tests

Focused validation passed 17 CPU contracts and 3 gfx950 projection tests. Coverage includes TP4/TP8 no-copy packed projection shapes, decode BF16 parity, prefill PTPC numerics, selector/backend failures, supported-layout errors, crossover boundaries, retained BF16 weights, finite outputs, and CUDA-graph capture.

Full GLM-5.3-Flash TP8 GSM8K on snapshot 03eb5366286afd40d2221b1d9c63a6dd1ba4832e: 1,319 examples, 1,200 evaluator threads, 4,096 maximum output tokens, thinking enabled, temperature 1.0, top-p 0.95, sampling seed unset.

Configuration	Score	Stop rate	Truncated rate	Error rate
BF16 fused projections — selector unset	96.82%	99.62%	0.38%	0.00%
No-copy packed PTPC — qkv_proj,f_a_proj,g_a_proj,o_proj	97.04%	99.70%	0.30%	0.00%

Speed Tests and Profiling

Environment:

• Hardware: AMD Instinct MI355X (gfx950)
• Docker image: rocm/sgl-dev:v0.5.19-rocm724-mi35x-20260910
• Model: zai-org/GLM-5.3-Flash, snapshot 03eb5366286afd40d2221b1d9c63a6dd1ba4832e
• Exact PR head: e8a2d9d2ab

No-copy packed first-stage latency

Real GLM-5.3-Flash checkpoint weights. Decode compares the same fused BF16 GEMM with the selector unset/set; prefill compares fused BF16 against one packed qkv/f/g PTPC GEMM plus BF16 beta, returning views without fused-output reconstruction.

TP	M	BF16 fused (ms)	Selector enabled (ms)	Delta
4	1	0.0257	0.0267	+3.89%
4	8	0.0246	0.0248	+0.65%
4	64	0.0301	0.0302	+0.53%
4	4,096	0.1987	0.1479	-25.58%
4	8,192	0.3614	0.2516	-30.40%
4	16,384	0.6749	0.4742	-29.74%

Decode retains one fused BF16 projection launch. Shapes outside the enabled prefill window stay on fused BF16.

Fused o-path latency

Complete path is gated RMSNorm plus output projection. The candidate replaces the BF16 norm output and standalone quantization with the fused bridge:

TP	M	BF16 path (ms)	Fused PTPC path (ms)	Delta
4	4,096	0.1111	0.0790	-28.95%
4	8,192	0.1612	0.1153	-28.45%
4	16,384	0.2720	0.2036	-25.17%
4	131,072	1.9188	1.6744	-12.74%
8	4,096	0.0861	0.0758	-12.01%
8	8,192	0.1144	0.0912	-20.26%
8	16,384	0.1693	0.1490	-12.00%
8	131,072	1.0713	1.0688	-0.23%

Every enabled TP/M cell beat the complete BF16 path. TP4 keeps BF16 below 256 tokens because intermediate small-batch shapes were neutral or slower.

End-to-end serving

GLM-5.3-Flash snapshot 03eb5366286afd40d2221b1d9c63a6dd1ba4832e, TP4, random 8K input / 1K output. The first concurrency-4 run in each repetition is the warmup; the tables average the two measured runs per concurrency. Both arms use the same source and differ only by the PTPC selector.

A. BF16 fused projections — selector unset:

Concurrency	TTFT (ms)	ITL (ms)	E2EL (ms)	Output tok/s
4	680.62	9.190	10082.12	406.08
8	1130.06	10.325	11691.83	700.36
16	1985.76	12.240	14510.19	1128.71
32	3651.16	15.885	19901.03	1645.98
64	7252.82	22.765	30542.19	1991.18

B. No-copy packed PTPC — qkv_proj,f_a_proj,g_a_proj,o_proj (delta versus BF16 fused projections):

Concurrency	TTFT (ms)	Delta	ITL (ms)	Delta	E2EL (ms)	Delta	Output tok/s	Delta
4	665.48	-2.22%	9.185	-0.05%	10061.90	-0.20%	406.90	+0.20%
8	1105.44	-2.18%	10.320	-0.05%	11664.72	-0.23%	702.00	+0.23%
16	1938.57	-2.38%	12.205	-0.29%	14426.81	-0.57%	1135.23	+0.58%
32	3568.69	-2.26%	15.820	-0.41%	19752.45	-0.75%	1658.36	+0.75%
64	7095.01	-2.18%	22.635	-0.57%	30248.42	-0.96%	2009.50	+0.92%

No-copy packed PTPC improves TTFT by 2.18-2.38%, ITL by 0.05-0.57%, E2E latency by 0.20-0.96%, and output throughput by 0.20-0.92% across the complete TP4 matrix.

TP8 end-to-end serving

GLM-5.3-Flash TP8, 131,072 context, random 8K input / 1K output. The first concurrency-4 run in each repetition is the warmup; the tables average the two measured runs per concurrency. Both arms use source 05fcea11c0 and differ only by the PTPC selector.

A. BF16 fused projections — selector unset:

Concurrency	TTFT (ms)	ITL (ms)	E2EL (ms)	Output tok/s
4	540.38	8.370	9105.14	449.65
8	906.39	9.355	10474.33	781.75
16	1584.41	10.765	12595.11	1300.28
32	2913.73	13.610	16833.84	1945.80
64	5566.28	18.145	24132.36	2714.67

B. No-copy packed PTPC — qkv_proj,f_a_proj,g_a_proj,o_proj (delta versus BF16 fused projections):

Concurrency	TTFT (ms)	Delta	ITL (ms)	Delta	E2EL (ms)	Delta	Output tok/s	Delta
4	538.35	-0.37%	8.350	-0.24%	9077.96	-0.30%	450.99	+0.30%
8	902.17	-0.47%	9.330	-0.27%	10447.53	-0.26%	783.76	+0.26%
16	1572.00	-0.78%	10.745	-0.19%	12564.34	-0.24%	1303.46	+0.24%
32	2885.02	-0.99%	13.540	-0.51%	16737.91	-0.57%	1956.95	+0.57%
64	5510.10	-1.01%	18.030	-0.63%	23954.41	-0.74%	2734.93	+0.75%

No-copy packed PTPC improves TP8 TTFT by 0.37-1.01%, ITL by 0.19-0.63%, E2E latency by 0.24-0.74%, and output throughput by 0.24-0.75% across the complete matrix.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations. No user-facing API is added; the opt-in selector remains documented in code and this PR.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Scoped syntax, isort, Ruff, Ruff format, codespell, registered-test validation, and git diff --check pass.

---

CI States

Latest PR Test (Base): ❌ Run #37683185870
Latest PR Test (Extra): ❌ Run #37683185566
Latest PR Test (AMD ROCm 10): ❌ Run #37683185864

[Raiden-Makoto]

/tag-and-rerun-ci

[Jacob0226]

Thanks for the restructure. With the old GSM8K/serving tables removed, the packed path has no full-model evidence yet. Could you add GSM8K and an i8k TP4/TP8 serving A/B (selector unset vs set) on a checkpoint where do_fuse_qkvbfg is True, and state which checkpoint was used?

[Jacob0226]

LGTM.
Since you edited the TP8 path as well. Please submit TP8 perf and gsm8k as well.

[Raiden-Makoto]

I'll fix merge conflict and update the PR body