[Triton/Gluon] [Config] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes by yuychang · Pull Request #6409 · ROCm/aiter

yuychang · 2026-10-12T03:38:40Z

Motivation

Kimi-K3 runs its shared-expert FP4 GEMMs on gfx950 at N=1536 K=7168 and N=7168 K=768. Neither shape has a Triton config, so both use DEFAULT.json. This PR adds tuned configs for them. It contains config data only and no code changes.

Technical Details

New gfx950 configs:
- gemm_afp4wfp4: GEMM-AFP4WFP4-N=1536-K=7168.json and GEMM-AFP4WFP4-N=7168-K=768.json
- gemm_a16wfp4: GEMM-A16WFP4-N=1536-K=7168.json and GEMM-A16WFP4-N=7168-K=768.json
M buckets that were slower than DEFAULT.json are left out, so those M values keep the default config.

Test Plan

A/B against main on one MI355X for M=1..8192: the same script on both trees, run back to back, for 2-3 rounds.
Timing uses run_perftest.
Each output is checked against a torch reference with checkAllclose.

Test Result

Setup: MI355X (gfx950), ROCm 7.2, torch 2.11.0+rocm7.2, Triton 3.7.0.

GPU kernel time, this PR vs main, best of the rounds:

kernel	M	shapes	geomean	range
`gemm_a16wfp4`	<= 256	16	1.68x	1.01x-3.07x
`gemm_a16wfp4`	> 256	8	1.00x	1.00x-1.01x
`gemm_afp4wfp4`	<= 256	16	1.13x	0.99x-1.44x
`gemm_afp4wfp4`	> 256	8	1.21x	0.98x-2.30x

Shapes below 1.00x use the same config as main, so those gaps are run-to-run noise.
All outputs match the reference.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

…apes - Add gemm_afp4wfp4 and gemm_a16wfp4 configs for N=1536 K=7168 and N=7168 K=768. - M buckets that were slower than DEFAULT.json are left out, so they keep the default config.

github-actions · 2026-10-12T03:39:06Z

🏷️ CI Guide

Runs automatically on every PR:

✅ Pre-checks (submodule verification, code formatting)
✅ Aiter op tests (gfx942 + gfx950)
✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
`ci:gfx1250-ffm-triton`	Run the five-shard gfx1250 FFM Triton test suite
`ci:triton-300x`	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
`ci:triton-355`	Run the full Triton test suite on MI35X, not only the tests the change affects
`ci:multi-gpu`	Aiter multi-GPU tests on the 8-GPU runner
`ci:sglang`	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
`ci:atom`	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
`ci:atom_full`	ATOM accuracy suite for PR and main models from ATOM `models_accuracy.json`
`ci:vllm`	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
`ci:all`	All standard extended tests (excludes `ci:atom_full`)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6409 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Copilot

🟢 Approval recommended

All configurations are valid, complete, correctly named, and preserve default fallback behavior.

0 open findings

What changed in this PR

Adds gfx950 Triton tuning data for Kimi-K3 shared-expert FP4 GEMMs without changing runtime code.

Changes:

Adds tuned AFP4×WFP4 and A16×WFP4 configurations for two (N, K) shapes.
Preserves default configurations for untuned larger-M buckets.

File	Description
`gemm_afp4wfp4/GEMM-AFP4WFP4-N=7168-K=768.json`	Adds AFP4×WFP4 tuning buckets.
`gemm_afp4wfp4/GEMM-AFP4WFP4-N=1536-K=7168.json`	Adds AFP4×WFP4 tuning buckets.
`gemm_a16wfp4/GEMM-A16WFP4-N=7168-K=768.json`	Adds A16×WFP4 tuning buckets.
`gemm_a16wfp4/GEMM-A16WFP4-N=1536-K=7168.json`	Adds A16×WFP4 tuning buckets.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…CPU test - test_pa_prefill: restore the torch default device after each test; input_helper set it to CUDA and never reset it. - test_fused_qk_norm_rope_gate_fp8_quant: build the CPU-input case under torch.device("cpu") so it ignores leaked defaults. - Fixes the order-dependent failure of the rejects_cpu_inputs test in Triton Tests (MI35X) shard 7.

Copilot

🟡 Changes recommended

The independent test-isolation fix should be moved to a dedicated follow-up PR.

1 open finding

Separate test-isolation changes from gfx950 GEMM retuning · New

🧠 Review effort: Balanced

- Take main's versions of test_pa_prefill.py and test_fused_qk_norm_rope_gate_fp8_quant.py. - Main fixed the same default-device leak in ROCm#6406 and ROCm#6288, so this branch's test-only fix is no longer needed.

Copilot

🔵 Needs a closer look

The configurations are structurally valid, but their hardware-specific performance and correctness results require final human verification.

0 open findings

1 resolved since last review

Separate test-isolation changes from gfx950 GEMM retuning

🧠 Review effort: Balanced

azaidy

LGTM!

perf: add gfx950 Triton FP4 GEMM configs for Kimi-K3 shared-expert sh…

a49ae91

…apes - Add gemm_afp4wfp4 and gemm_a16wfp4 configs for N=1536 K=7168 and N=7168 K=768. - M buckets that were slower than DEFAULT.json are left out, so they keep the default config.

yuychang requested review from a team and a balanced review from Copilot October 12, 2026 03:38

github-actions Bot changed the title ~~[Triton/Gluon] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes~~ [Triton/Gluon] [Config] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes Oct 12, 2026

github-actions Bot added Config Triton/Gluon labels Oct 12, 2026

Copilot started reviewing on behalf of yuychang October 12, 2026 03:39

View session

yuychang mentioned this pull request Oct 12, 2026

[Triton/Gluon] Kimi-K3 GEMM tuning, caller-owned outputs and large-M bpreshuffle reuse #6262

Closed

1 task done

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 12, 2026 08:18

github-actions Bot changed the title ~~[Triton/Gluon] [Config] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes~~ [Triton/Gluon] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes Oct 12, 2026

github-actions Bot removed the Config label Oct 12, 2026

Copilot started reviewing on behalf of yuychang October 12, 2026 08:19

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread op_tests/triton_tests/attention/test_pa_prefill.py Resolved

chore: merge main into k3/triton-fp4-configs

9a47065

- Take main's versions of test_pa_prefill.py and test_fused_qk_norm_rope_gate_fp8_quant.py. - Main fixed the same default-device leak in ROCm#6406 and ROCm#6288, so this branch's test-only fix is no longer needed.

Copilot AI balanced review requested due to automatic review settings October 12, 2026 08:22

github-actions Bot changed the title ~~[Triton/Gluon] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes~~ [Triton/Gluon] [Config] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes Oct 12, 2026

github-actions Bot added the Config label Oct 12, 2026

Copilot started reviewing on behalf of yuychang October 12, 2026 08:23

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

azaidy self-assigned this Oct 12, 2026

azaidy self-requested a review October 12, 2026 16:05

azaidy approved these changes Oct 12, 2026

View reviewed changes

azaidy merged commit b06e606 into ROCm:main Oct 12, 2026

yuychang mentioned this pull request Oct 13, 2026

[Config] perf: Publish Kimi-K3 BF16 and shared-expert FP4 GEMM tuning #6217

Closed

1 task done

Navigation Menu

Repository navigation

[Triton/Gluon] [Config] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes - #6409

[Triton/Gluon] [Config] Add gfx950 FP4 GEMM configs for Kimi-K3 shared-expert shapes#6409
azaidy merged 3 commits into
ROCm:mainfrom
yuychang:k3/triton-fp4-configs

yuychang commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

azaidy left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

yuychang commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

github-actions Bot commented Oct 12, 2026

🏷️ CI Guide

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

azaidy left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuychang commented Oct 12, 2026 •

edited

Loading