Skip to content

[HIP] [Feature] a8w8 blockscale asm: out_zeroed skips the split-K output memset - #6786

Draft
kevin-mii wants to merge 3 commits into
ROCm:mainfrom
kevin-mii:kevin-mii/asm-blockscale-out-zeroed
Draft

kevin-mii wants to merge 3 commits into
ROCm:mainfrom
kevin-mii:kevin-mii/asm-blockscale-out-zeroed

Conversation

@kevin-mii

@kevin-mii kevin-mii commented Oct 21, 2026 •

Copy link
Copy Markdown

Summary

Add out_zeroed=False to the blockscale bpreshuffle asm entrypoint and tuned dispatcher. With out_zeroed=True, the caller promises the output already contains zeros, allowing the split-K launcher to skip its redundant clear. This lets a fused producer clear the output without requiring a copied custom GEMM launcher.

The default preserves the existing clear. The opt-in flag preserves upstream’s output alignment checks, row-stride support, and fp16 fallback. A row-padded output keeps its padding untouched. The dispatcher requires an explicit output when the flag is enabled.

Validation

Against main 1dd826c42, on gfx950 / ROCm 10 / PyTorch 2.11:

  • All 22 cases in op_tests/test_gemm_a8w8_blockscale_bpreshuffle_out.py passed.
  • Added coverage exercises direct asm and tuned dispatch, contiguous and row-padded pre-zeroed outputs, sentinel preservation, a deliberately nonzero output proving that the clear is skipped, and alignment rejection with the flag enabled.
  • Existing CK/CK-Tile/asm output-layout and fp16 fallback cases passed.
  • Black, CI-pinned Ruff 0.16.0, and whitespace checks passed.

The flag is still absent upstream. This remains a draft; the original producer-integrated performance result has not been re-benchmarked on this updated branch.

…put memset

gemm_a8w8_blockscale_bpreshuffle_asm zero-fills `out` with hipMemsetAsync
before every split-K launch, because the splits accumulate into it
atomically. A caller whose producer already writes those zeros (a fused
norm + quant kernel, say) pays for a second fill and one more stream / graph
node per GEMM. out_zeroed=True tells the launcher the output already holds
zeros, so it launches the kernel alone.

- C launcher: an `int out_zeroed` argument; the memset is skipped when set.
- gemm_a8w8_blockscale_bpreshuffle_asm and the tuned dispatcher
  gemm_a8w8_blockscale_bpreshuffle take out_zeroed=False; the dispatcher
  forwards it to asm rows only (every other row overwrites `out`) and
  requires the caller's `out` with it.
- op test: the zeroed launch matches the default one at split-K 2 and 3, and
  a non-zero `out` shows through, so the memset really is skipped.

Defaults keep today's behaviour for every existing caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kevin Mi <mikevin920@yahoo.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355 Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6786 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

@kevin-mii

Copy link
Copy Markdown
Author

Updated against main while preserving the new fp16 fallback, output-row stride handling and split-K alignment checks. out_zeroed=True only skips the clear; it does not bypass validation.

On gfx950 / ROCm 10, the output-layout suite passed all 22 cases. The added cases cover the direct asm entry and tuned dispatcher, contiguous and row-padded pre-zeroed outputs, sentinel preservation, deliberate nonzero input proving the clear was skipped, and rejection of misaligned outputs even when the flag is set. Black, Ruff and whitespace checks pass.

The flag is still absent upstream and remains useful for a producer that clears the output. Keeping the existing draft status; the original end-to-end performance claim was not re-benchmarked in this pass.

kevin-mii added a commit to kevin-mii/sglang that referenced this pull request Nov 2, 2026
…tput

The block-fp8 GEMM after a gfx950 Gemma norm launched aiter's asm split-K
kernel into an output the norm zero-filled, saving aiter's memset launch.
End to end it measured within run-to-run noise, and ROCm/aiter#6786 adds
out_zeroed to aiter's own launcher. This removes the local launcher
(aiter_asm_blockscale, hipcc's 2-D grid), the norm's zero fill, the GEMM
shape each norm was bound to, and their tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kevin-mii added a commit to kevin-mii/sglang that referenced this pull request Nov 2, 2026
…tput

The block-fp8 GEMM after a gfx950 Gemma norm launched aiter's asm split-K
kernel into an output the norm zero-filled, saving aiter's memset launch.
End to end it measured within run-to-run noise, and ROCm/aiter#6786 adds
out_zeroed to aiter's own launcher. This removes the local launcher
(aiter_asm_blockscale, hipcc's 2-D grid), the norm's zero fill, the GEMM
shape each norm was bound to, and their tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant