Skip to content

[ROCm][CI] Stabilize MI355 FlyDSL MoE accuracy test - #53024

Merged
njhill merged 2 commits into
mainfrom
akaratza_flydsl
Aug 21, 2026
Merged

njhill merged 2 commits into
mainfrom
akaratza_flydsl

Conversation

@AndreasKaratzas

Copy link
Copy Markdown
Member

This PR stabilizes the two MI355 FlyDSL W4A16 MoE cases that failed in AMD CI build 12250. The test was introduced in #44400, and its test, kernel, and conversion code are unchanged between recent passing and failing builds; #41100 changed test collection and exposed the test's dependence on inherited RNG state. The failures contained only a few BF16 accumulation outliers among as many as 117 million output values, so a strict torch.allclose made the largest cases brittle. This change seeds the inputs and checks both the mismatch rate and worst normalized error, retaining sensitivity to broad or severe accuracy regressions.

  • Fix: Seed every parameterized case, require at least 99.999% of values to satisfy the original atol=0.5, rtol=0.1 tolerance, and bound every permitted outlier to 3x its elementwise tolerance.
  • Repeated validation: 30/30 independent full-file runs passed across two gfx950 MI355 GPUs, for 900/900 collected test cases.
  • Input sweep: 30 distinct seeds passed both formerly failing shapes in fresh processes across two additional MI355 GPUs, for 60/60 exact-shape runs.
  • Control: With the original unseeded strict assertion restored, the two cases failed in 3/10 fresh processes; deterministic diagnostics observed only 0-3 mismatches out of 117,440,512 values, with a worst error of 1.086x the original elementwise tolerance.

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review August 20, 2026 00:49

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@mergify mergify Bot added the rocm Related to AMD ROCm label Aug 20, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Aug 20, 2026
@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84715 for commit b7a77e2ea2de.

@njhill
njhill merged commit a0af854 into main Aug 21, 2026
37 checks passed
@njhill
njhill deleted the akaratza_flydsl branch August 21, 2026 20:17
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Aug 21, 2026
am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
mikeshawcode pushed a commit to mikeshawcode/vllm that referenced this pull request Sep 1, 2026
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: mikeshawcode <michaelwshaw2@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants