Skip to content

[Don't Merge] Isolate MLA failure: CK pre-#7331 (do not merge) - #3246

Closed
DDEle wants to merge 1 commit into
mainfrom
yiding12/ci-debug-ck-pre7331
Closed

[Don't Merge] Isolate MLA failure: CK pre-#7331 (do not merge)#3246
DDEle wants to merge 1 commit into
mainfrom
yiding12/ci-debug-ck-pre7331

Conversation

@DDEle

@DDEle DDEle commented May 18, 2026

Copy link
Copy Markdown
Contributor

Why

PR #3150 sees a test_mla_persistent.py (shard 5/2 MI35X) failure that's hard to attribute. PR #3150's own aiter code changes only touch FMHA BWD; CK is also bumped from 33b62ed083566edb0 (10 CK PRs including #7331).

This draft bumps CK to 4d852e80f (= PR #6983, the commit immediately before #7331) and leaves everything else identical to main. CI here exercises the same 9 CK PRs as #3150 minus only #7331's FMHA BWD async-prepare changes.

Outcome interpretation

  • If shard 5/2 MLA fails here too → cause is one of the other 9 CK PRs (or pre-existing flakiness on MI35X). [CK_TILE] FMHA BWD: stream-async workspace prepare for group mode #3150 is innocent.
  • If shard 5/2 MLA passes here → would need to investigate #7331 further (very unlikely since #7331 only touches FMHA BWD launcher code paths that MLA does not call).

Not for merge

Pure CI diagnostic; will be closed once we have the result.

Temporary debug branch to isolate the source of MI35X shard 5/2
test_mla_persistent.py failure observed on PR #3150.

PR #3150 bumps CK from 33b62ed0 -> 83566edb0 (10 CK PRs incl. #7331).
This commit bumps to the immediately preceding commit (4d852e80f =
PR #6983), so CI runs the exact same 9 CK PRs as #3150 minus only
the FMHA BWD async-prepare changes that #3150 actually depends on.

If MLA still fails here, the cause is one of the other 9 CK PRs
(or pre-existing flakiness) — proving the failure is unrelated to
PR #3150's own changes.

Not for merge.
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 3246 --add-label <label>

@DDEle DDEle changed the title [CI-DEBUG] Isolate MLA failure: CK pre-#7331 (do not merge) [Don't Merge] Isolate MLA failure: CK pre-#7331 (do not merge) May 18, 2026
@DDEle
DDEle marked this pull request as ready for review May 18, 2026 08:59
@DDEle
DDEle requested review from a team and Copilot May 18, 2026 08:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Pins the composable_kernel submodule to the commit immediately preceding ROCm/rocm-libraries#7331 to help bisect and attribute the observed test_mla_persistent.py failure (MI35X shard 5/2) without introducing any aiter-side code changes.

Changes:

  • Updates 3rdparty/composable_kernel submodule revision to 4d852e80f (pre-#7331) for CI/bisect isolation.
  • Leaves the rest of the repository identical to main to keep the experiment controlled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@DDEle DDEle closed this May 19, 2026
@zufayu
zufayu deleted the yiding12/ci-debug-ck-pre7331 branch August 11, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants