[Triton/Gluon] [JIT] # RDNA3 fp8 enablement + tuned DSV4 A8W8 block-scale GEMM configs

[amd-xavierwang]

Motivation

• gfx11 has no default configs nor shape specific config files for fp8-gemm-blockscale in aiter now.
• gfx11 uses fnuz for dtype which should be changed to torch.float8_e4m3fn.
• the GRID_MN is a tl.constexpr which makes Triton to trigger re-compile for its dependencies, M and N. For LLM, cold compiling for each M in GEMM is not accepted.

Changes

1. aiter/ops/triton/utils/types.py — gfx11 fp8 dtype

Add gfx1100, gfx1101, gfx1102, gfx1151 to the non-fnuz arch tuple in both
get_fp8_dtypes() and get_fp8_e4m3_dtype() → e4m3 resolves to
torch.float8_e4m3fn on gfx11.

2. aiter/ops/triton/configs/.../gemm_a8w8_blockscale/ — tuned configs

Six JSON files in the nested layout (#4927):

file	shape (N, K)	role
DEFAULT.json	—	fallback covering all (N, K)
GEMM-A8W8_BLOCKSCALE-N=1536-K=4096.json	1536, 4096	DSV4 TP=8 linear
GEMM-A8W8_BLOCKSCALE-N=512-K=4096.json	512, 4096	DSV4 TP=8 linear
GEMM-A8W8_BLOCKSCALE-N=4096-K=1024.json	4096, 1024	DSV4 TP=8 linear
GEMM-A8W8_BLOCKSCALE-N=4096-K=256.json	4096, 256	DSV4 TP=8 linear
GEMM-A8W8_BLOCKSCALE-N=8192-K=1024.json	8192, 1024	DSV4 TP=8 linear

3. aiter/ops/triton/_triton_kernels/gemm/basic/gemm_a8w8_blockscale.py — remove M-dependent GRID_MN

GRID_MN causes M, N to become constexpr, causing a new JIT compile on every new M. This is not acceptable for LLM inference. The only usecase, remap_xcd(pid, GRID_MN) is also dead code.

4. RDNA3/3.5 fp8 enablement

file	change
aiter/ops/triton/utils/types.py	add gfx1150 to the non-fnuz arch tuple → e4m3 resolves to torch.float8_e4m3fn
aiter/utility/dtypes.py	add "gfx1150": {"fp8": torch.float8_e4m3fn} to defaultDtypes (see #4764 )
aiter/jit/utils/chip_info.py	add gfx1150 to _LDS_CAPACITY_BYTES at 64 KiB/WGP (also fills the missing gfx1101/gfx1102/gfx1200 entries)

Exercised through vLLM DeepSeek-V4-Flash serving on gfx1100 (TP=8, fp8 KV cache). Inference runs after adapting Aiter's backend.

Test Result

Metric	Baseline	aiter blockscale	Δ
Output throughput (tok/s)	21.09	30.96	+46.8%
Benchmark duration (s)	1006	660.7	−34.3%
Median TPOT (ms)	200	113	−43.5%
Median ITL (ms)	421	191	−54.6%

# tuned config: GEMM-A8W8_BLOCKSCALE-N=8192-K=1024.json
pytest op_tests/triton_tests/gemm/basic/test_gemm_a8w8_blockscale.py \
  -k "False and triton and 8192-1024"
# -> 6 passed, 6 skipped (gluon)

# tuned config: GEMM-A8W8_BLOCKSCALE-N=1536-K=4096.json
pytest op_tests/triton_tests/gemm/basic/test_gemm_a8w8_blockscale.py \
  -k "False-triton and 1536-4096"
# -> 6 passed

# DEFAULT.json fallback (no per-shape file for 4096x4096)
pytest op_tests/triton_tests/gemm/basic/test_gemm_a8w8_blockscale.py \
  -k "False-triton and 4096-4096"
# -> 7 passed

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

*** Honouring contribution to @saeid-rostami for identifying the default dtype and compilation regression.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5217 --add-label <label>

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to opt this PR out.

[amd-xavierwang]

Agreed. In addition, kpack and matrix_instr_nonkdim are essentially no-ops on RDNA. I will do the clean up with existing files. There are gfx1201 files in upstream with kpack as well. I can clean those up as well upon demand.

[Copilot]

🟡 Changes recommended

Three specialized configurations are never selected by the existing unit-test matrix.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Architecture-specific dtype behavior and GPU tuning require human confirmation, and one duplicated dispatch list remains.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Balanced

[brunomazzottiamd]

LGTM!

@amd-xavierwang, can you please rebase on top of latest main branch? I think there are some CI failures in Aiter Test / Standard Tests suite that have already been fixed.

By this way, we can trigger CI again and then merge when everything is green.

[amd-xavierwang]

CI looks good now!

[brunomazzottiamd]

CI looks good now!

I was trying to merge this in the last hour or so, but couldn't find an unresolved discussion... Now it's merged thanks to Satya!