[Triton/Gluon] [Config] Add gfx950 gemm_a16w16 config for N=288, K=4096

[Jacob0226]

Motivation

GLM-5.3-Flash's MoE router (288 experts over hidden size 4096) runs a bf16 GEMM of N=288, K=4096 in each of its 42 MoE layers, and at decode batch 1–8 each call takes 27 us where a spill-free kernel takes 5 us. gfx950 gemm_a16w16 has no file for this shape, so it falls back to DEFAULT.json, whose M_LEQ_8 and M_LEQ_32 entries set waves_per_eu=8. That caps the kernel at 64 VGPRs and it spills 16–21 registers. The new GEMM-A16W16-N=288-K=4096.json drops that hint and .cg for M ≤ 64 and keeps one kernel (no split-K). M_LEQ_128 and above are copied verbatim from DEFAULT.json.

Test Plan

MI355X (gfx950), ROCm 10.0.0, Triton 3.8.0, rocm/sgl-dev:v0.5.20-rocm10-mi35x-20260928.
Step 1 — a NUM_KSPLIT=1 sweep picks the entries. 8,640 configs over M=1–64, timed in a HIP graph over 42 rotating weights (one per MoE layer, so the weight read misses L2). Among configs within 1% of the fastest, the one with waves_per_eu=0 was kept.
Step 2 — bench_gemm_a16w16.py A/Bs the file on upstream main, moving the JSON in and out of aiter/ops/triton/configs/gfx950/triton/gemm/gemm_a16w16/. Median of 3 runs per point:
python3 op_tests/op_benchmarks/triton/bench_gemm_a16w16.py --shape 288 4096 --metric time
Correctness: gemm_a16w16 vs torch.nn.functional.linear, M ∈ {1…256}, layouts TN/TT/NN/NT, with and without bias — 128 cases, 0 failures. End to end: SGLang bench_serving, TP4, same JSON in/out of the image's config dir.

Test Result

Kernel, us/call. "bench" is the Step 2 command; "graph" is the Step 1 harness with 42 rotating weights.

M	bucket	bench stock	bench tuned	graph stock	graph tuned	VGPRs / spills stock → tuned
1	M_LEQ_4	24.57	5.42
2	M_LEQ_4	24.57	5.43
4	M_LEQ_4	24.60	5.44	27.19	5.37	64 / 16 → 94 / 0
8	M_LEQ_8	24.70	4.94	27.34	5.03	64 / 16 → 98 / 0
16	M_LEQ_16	4.96	4.95	7.12	5.10	80 / 0 → 98 / 0
32	M_LEQ_32	29.57	5.02	35.00	5.18	64 / 21 → 98 / 0
64	M_LEQ_64	9.19	5.29	13.49	5.39	88 / 0 → 90 / 0
128	M_LEQ_128 (unchanged)	14.52	14.52
256	M_LEQ_256 (unchanged)	49.10	49.09

No M is slower. M=16 is flat on the bench because a single hot weight hides DEFAULT's cost there; with rotating weights it is 7.12 → 5.10 us. M=128/256 resolve to the same entries as before, field for field.

End to end, GLM-5.3-Flash-Quark-MXFP4, TP4, input 8192 / output 1024:

concurrency 4	stock	tuned
median ITL (ms)	9.59	8.67 (−9.6%)
mean TPOT (ms)	10.15	9.22(−9.1%)
output throughput (tok/s)	330.2	406.2 (+23%)
median TTFT (ms)	241.0	237.2
GSM8K (1,319)	–	96.59%

In SGLang, decode batches above 16 route the router GEMM through aiter.tuned_gemm to hipBLASLt, so the gain is at concurrency ≤ 16; the M_LEQ_32/M_LEQ_64 entries matter only for callers that reach gemm_a16w16 directly.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5933 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[Copilot]

Copilot review overview

🟢 Approval recommended

The configuration is valid, complete, correctly located, and preserves the documented defaults.

Review effort: Balanced
Findings: None

What changed in this PR

Adds a gfx950 Triton tuning profile for GLM-5.3-Flash’s BF16 router GEMM.

Changes:

• Tunes M ≤ 64 for N=288, K=4096.
• Preserves default configurations for larger M values.

File	Description
aiter/​ops/​triton/​configs/​gfx950/​triton/​gemm/​gemm_a16w16/​GEMM-A16W16-N=288-K=4096.json	Adds shape-specific GEMM tuning tiers.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[azaidy]

LGTM!