[xytpai]

Summary

Parallelize the cross-wave amax reduction in the gfx1250 two-stage fused
all-reduce + residual + RMSNorm + per-token FP8 epilogue.

Instead of thread 0 folding the wave partials serially, the first wave loads
the nonnegative amax partials and reduces them with wave shuffles. Unused
lanes contribute zero; at most 32 wave32 partials fit in that wave.

• Preserve square-sum reduction order, rank-order summation and all BF16/FP16 rounding steps.
• Keep residual semantics, FP8 conversion, scales and the optional pre-quantization mirror unchanged.
• Keep communication barriers, staging buffers, launch bounds, grid sizes, one-stage dispatch and no-quant normalization unchanged.
• No model-specific dispatch or new environment variables.
• Production changes are confined to custom_all_reduce_gfx1250.cuh. gfx942/gfx950 implementations are untouched; no runtime validation on those architectures is claimed.

Tracking:

• https://github.com/wuhuikx/LLM-Booster/issues/188
• https://github.com/wuhuikx/LLM-Booster/issues/340

This is a partial FP8 second-stage improvement, not a claim to close the entire model-level performance gap.

Performance

Measured on 2026-10-28 UTC, 4 x gfx1250, 256 CU/device, TP4.
Baseline: 9b7bb36ad9bead63ff0931bbd098a27aac2453bb.
Candidate: 7d9ae81a08ef81ad29cc9b0f4b387030c380d6d4.

Image: rocm/fw-bringup-mlperf:gfx1250-atom-20261023.
PyTorch: 2.11.0+rocm10.1.0a20260811-bkc.20260820; reported HIP: 7.16.26315.

Separate source checkouts, containers and JIT caches; GPU process checks
before runs. Three runs per side, interleaved base/opt, opt/base, base/opt.
Each graph contains 16 calls; three profiled replays provide 48 second-stage
events per rank. Reported latency is the median across three runs of the
median of the four rank-local kernel medians.

Shape	Output	Base (us)	Candidate (us)	Speedup
[2517,6144]	BF16	47.490	47.270	1.005x
[2517,6144]	FP8	48.422	46.088	1.051x
[4097,6144]	BF16	69.032	69.513	0.993x
[4097,6144]	FP8	73.709	69.974	1.053x
[2048,8192]	BF16	43.995	44.145	0.997x
[2048,8192]	FP8	43.794	43.945	0.997x
[512,6144]	BF16	9.884	10.024	0.986x
[512,6144]	FP8	12.287	10.855	1.132x

For MiniMax-M3's [2517,6144] BF16-input/FP8-output case:

• Base run medians: 48.411 / 48.422 / 48.561 us.
• Candidate run medians: 45.818 / 46.088 / 46.298 us.

The large 6144-hidden FP8 cases improve by about 5%; 8192-hidden is effectively
unchanged. The 512-row measurements have more relative variability and are
not the primary performance claim. No-quant BF16 is an unchanged control.

These are second-stage kernel timings, not full-collective or model
throughput speedups. No full-model throughput or GSM8K validation is claimed.
Rejected grid/thread-count candidates are not included.

Correctness

The new regression is added to the existing
op_tests/multigpu_tests/test_fused_ar_rms.py, using public dispatching APIs,
@benchmark, run_perftest, checkAllclose, and per-rank markdown tables
with latency, logical TFLOPS/TB/s and error columns.

The identical test passed on baseline and candidate (one pytest test,
internally sweeping 8 shapes x 2 input dtypes x 2 norm modes x 2 output modes
on all four ranks):

• FP16/BF16 and ordinary/Gemma RMSNorm with distinct rank inputs.
• Shapes: [12,6144], [511,6144], [512,6144], [513,6144], [2517,6144],
  [513,6176], [513,6160], [2048,8192].
• Grid-cap boundaries, partial waves and one-stage/small-message controls.
• FP32 rank-order reference with AR rounded to input dtype before residual addition.
• Exact residual, eager/graph and cross-rank agreement checks.
• FP8 exactly matches separate quantization of the norm output, including
  scales and the optional BF16/FP16 mirror.

An additional 7-shape x 2-output-mode sweep passed. Saved rank-0 norm/FP8,
residual and scale tensors matched baseline bit-for-bit in all 14 cases;
each run independently checked cross-rank agreement.

Reproduction

In the described ROCm environment:

OMP_NUM_THREADS=1 GPU_ARCHS=gfx1250 PYTORCH_ROCM_ARCH=gfx1250 \
NCCL_IB_DISABLE=1 RCCL_IB_DISABLE=1 \
python3 -m pytest -v -s \
  op_tests/multigpu_tests/test_fused_ar_rms.py::test_prefill_quant_tp4

python3 -m black --check op_tests/multigpu_tests/test_fused_ar_rms.py
python3 -m ruff check --no-cache op_tests/multigpu_tests/test_fused_ar_rms.py
git diff --check

All listed checks passed. Raw traces, timing samples and exploratory drivers
remain outside the repository; no generated artifacts or benchmark reports
are added under docs.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7080 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[TennyWang1223]

LGTM

[aiter-bot]

WAITING ON AUTHOR

PR Self-Review

Revision 7d9ae81a08ef · mode shadow

Waiting for the PR author. The readiness report will appear after the self-review answers are complete for this revision.

Questions to answer in the PR description

1. What behavior does this PR change, and why is the existing behavior insufficient?
2. What did you validate, and what remains unverified? Include commands, hardware/configuration and results.
3. Which architecture, dtype and shape boundaries select the changed kernel path, and what invariant prevents unsupported inputs from reaching it?
4. For every changed op_tests/ file, what behavior does it cover and why is it not redundant? Write None if no op test changed.

Edit the PR Self-Review section in the PR description. Include the current revision SHA; this comment refreshes automatically.

Missing or invalid answers

• FAIL — missing ### Reviewed revision in the PR description
• FAIL — missing ### Behavior and intent in the PR description
• FAIL — missing ### Validation and unverified scope in the PR description
• FAIL — missing ### PR-specific question in the PR description
• FAIL — missing ### Changed op tests in the PR description
• FAIL — Reviewed revision must contain at least the first 7 characters of the current head SHA
• FAIL — Behavior and intent has no substantive answer
• FAIL — Validation and unverified scope has no substantive answer
• FAIL — PR-specific question has no substantive answer
• FAIL — changed op test not explained in Changed op tests: op_tests/multigpu_tests/test_fused_ar_rms.py
• FAIL — Changed op tests must explain why each changed op test is necessary and non-redundant

_{Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.}