Skip to content

feat(gemm_a8w8_blockscale_bpreshuffle): add GLM-5.2 tuned configs (gfx950) - #4243

Merged
yifehuan merged 1 commit into
ROCm:mainfrom
Raiden-Makoto:glm52-blockscale-bpreshuffle-gfx950
Jul 21, 2026
Merged

feat(gemm_a8w8_blockscale_bpreshuffle): add GLM-5.2 tuned configs (gfx950)#4243
yifehuan merged 1 commit into
ROCm:mainfrom
Raiden-Makoto:glm52-blockscale-bpreshuffle-gfx950

Conversation

@Raiden-Makoto

@Raiden-Makoto Raiden-Makoto commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Motivation

The a8w8_blockscale_bpreshuffle GEMM has per-model tuned configs under
aiter/configs/model_configs/ for dsv3, mm2.5, and qwen — but none for GLM-5.2.
GLM-5.2's two 128-aligned MLA dense projections on gfx950 — q_b_proj
(N=4096, K=2048) and o_proj (N=6144, K=4096) — therefore miss the tuned-config
lookup and fall back to the default solution (aiter logs
not found tuned config in ... will use default config).

Note aiter already ships a8w8_bpreshuffle_tuned_gemm_glm5.2.csv, but that is the
per-tensor bpreshuffle path (schema has q_dtype_w). These projections use the
block-scale (128x128) path (a8w8_blockscale_bpreshuffle), which has no GLM-5.2
config. This PR adds one. Config coverage only — no code changes.

Technical Details

  • Adds one file:
    aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_glm5.2.csv
    • 120 tuned rows for gfx950 / cu_num=256 covering both shapes
      (q_b_proj 4096,2048 and o_proj 6144,4096).
    • M coverage: prefill token counts (1024–65536) and decode / cuda-graph batch
      sizes (1–512).
    • Schema matches the base a8w8_blockscale_bpreshuffle_tuned_gemm.csv:
      gfx,cu_num,M,N,K,libtype,kernelId,splitK,us,kernelName,tflops,bw,errRatio
      (libtype ∈ {asm, ck, cktile}).
  • No code changes. AiterConfigs.get_config_file(...) globs
    model_configs/*a8w8_blockscale_bpreshuffle_tuned_gemm*.csv and
    update_config_files(...) merges + dedups (lowest us per shape) with the base
    at runtime, so the new file is picked up automatically.
  • Rows produced by the standard tuner
    (csrc/ck_gemm_a8w8_blockscale/gemm_a8w8_blockscale_tune.py --preshuffle), no
    manual edits.

Test Plan

  • Run the tuner for both shapes across the M range on gfx950 (MI355X, cu256).
  • Load GLM-5.2 and confirm the two shapes no longer emit the default-config
    fallback warning (i.e. get_config_file merges the new model_config).
  • op_tests/test_gemm_a8w8_blockscale.py on the added shapes.

Test Result

  • Tuner completed with 0 failures across all M for both shapes (gfx950, cu256).
  • With the file merged, the not found tuned config … will use default warnings
    for 4096,2048 and 6144,4096 no longer appear at load; both resolve to their
    tuned entries.

Submission Checklist

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4243 --add-label <label>

@Raiden-Makoto
Raiden-Makoto marked this pull request as ready for review July 15, 2026 01:22
@Raiden-Makoto
Raiden-Makoto requested a review from a team July 15, 2026 01:22
@zufayu
zufayu requested a review from yifehuan July 15, 2026 02:20
Comment thread aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_glm5.2.csv Outdated
Oseltamivir added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Jul 18, 2026
The cookbook-verified v0.5.13.post1-rocm720-mi35x-20260618 build segfaults
in AITER gemm_a8w8_blockscale_bpreshuffle mid-profiling under the agentic
corpus (KV usage 0.95+, 5.3M pending tokens, untuned GLM-5.2 GEMM shapes
falling back to default configs - gfx950 tuned configs are not upstream
yet, ROCm/aiter#4243). Move to the newest rocm720 build for the latest
AITER fallback fixes.

Cap the conc list at 32 (was 64): the B300 sibling frontier showed
no-offload points beyond ~c48 are KV-cliff-dominated on the 131k-token
mean corpus, and MI355X prefill is slower still - c48/c64 only added
crash surface and sweep time for dominated points.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ig (gfx950)

Adds tuned a8w8_blockscale_bpreshuffle configs for GLM-5.2's o_proj (N=6144,
K=4096) on gfx950 — the one 128-aligned MLA dense-projection shape missing from
the GLM5 block-scale configs merged in ROCm#4236 (which covers q_b_proj 4096,2048 and
other GLM5 shapes but not o_proj 6144,4096). Config coverage only, no code changes;
merges via the existing *a8w8_blockscale_bpreshuffle_tuned_gemm*.csv glob.

60 rows, gfx950/cu_num=256, M = prefill (1024-65536) + decode/cuda-graph (1-512).
@Raiden-Makoto
Raiden-Makoto force-pushed the glm52-blockscale-bpreshuffle-gfx950 branch from 889a79f to 6aa9329 Compare July 20, 2026 15:50
@Raiden-Makoto

Raiden-Makoto commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@yifehuan following up on your review — the duplicated a8w8_blockscale cases you flagged (the shapes overlapping qwen3.5_397b) were removed in f3130ae, so the merge is dedup-clean now and CI is fully green: 29 checks passing, 13 skipped, 0 failures, and the PR is mergeable.

This is the same tuned-config pattern you already reviewed and merged in #4236 (and #4264), just the remaining GLM-5.2 o_proj rows. Could you re-review and approve/merge when you get a chance? Thanks!

@Raiden-Makoto
Raiden-Makoto requested a review from yifehuan July 20, 2026 19:16

@yifehuan yifehuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, no more duplicate shape now

@yifehuan
yifehuan merged commit 9f0a99b into ROCm:main Jul 21, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants