Skip to content

[HIP] [BUG fix] Initialize reverse route indices before Opus sorting - #6467

Merged
zufayu merged 4 commits into
mainfrom
cguo/fixm3
Oct 14, 2026
Merged

zufayu merged 4 commits into
mainfrom
cguo/fixm3

Conversation

@charlieguo1106

Copy link
Copy Markdown
Contributor

Motivation

Opus sorting can leave reverse_sorted entries unwritten when routing repeats an expert with zero weight. The scatter reduction then reads stale allocator contents as sorted row indices, which can cause an illegal GPU memory access.

Initialize all reverse route indices to -1 on the current HIP stream before sorting. Valid routes overwrite their entries, while unwritten slots retain the sentinel that scatter reduction skips. Remove the redundant Python-side initialization for EP routing.

Validated locally:

  • Reproduced the illegal access before the fix and verified the original tuned scatter kernel passes afterward.
  • Checked shared-expert routing with duplicate zero-weight slots at 256 and 32768 tokens.
  • Verified torch.compile, non-default stream execution, CUDA Graph replay with changed routing, and existing Opus sorting checks.

Technical Details

Test Plan

Test Result

Submission Checklist

@charlieguo1106
charlieguo1106 requested a review from a team October 13, 2026 05:51
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355 Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6467 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

@github-actions github-actions Bot changed the title [BUG fix][Flydsl MOE] Initialize reverse route indices before Opus sorting [HIP] [BUG fix] Initialize reverse route indices before Opus sorting Oct 13, 2026
@github-actions github-actions Bot added the HIP label Oct 13, 2026
@zufayu
zufayu self-requested a review October 14, 2026 01:28
@zufayu zufayu self-assigned this Oct 14, 2026
@zufayu
zufayu merged commit dcb7c7f into main Oct 14, 2026
@zufayu
zufayu deleted the cguo/fixm3 branch October 14, 2026 07:09
Klaud-Cold pushed a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 14, 2026
aiter 9205e105 in nightly_202610131505 picks the scatter-epilog GEMM2 for
MiniMax-M3 TP4 MoE at M > 8192, which hits an illegal memory access on
unwritten Opus reverse routes (fixed upstream in ROCm/aiter#6467). Cap the
TP4 points' max-num-batched-tokens at 8192 so they stay on the reduce rows.

nightly_202610131505 中的 aiter 9205e105 会在 M > 8192 时为 MiniMax-M3 TP4
MoE 选择 scatter epilog GEMM2,因 Opus 反向路由未初始化而触发非法内存访问
(上游已在 ROCm/aiter#6467 修复)。将 TP4 点位的 max-num-batched-tokens
限制为 8192,使其使用 reduce 配置。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants