Skip to content

ci(miopen-dbsync): make gfx950 blocking - #9768

Merged
JonathanLichtnerAMD merged 1 commit into
mainfrom
users/jlichtne/miopen-dbsync-gfx950-blocking
Nov 6, 2026
Merged

JonathanLichtnerAMD merged 1 commit into
mainfrom
users/jlichtne/miopen-dbsync-gfx950-blocking

Conversation

@JonathanLichtnerAMD

Copy link
Copy Markdown
Contributor

JIRA ID: ALMIOPEN-2674

Motivation

miopen-dbsync on gfx950-dcgpu was re-added as advisory (#8813, expect_failure) so a regression during re-qualification would not block unrelated PRs. ALMIOPEN-2674 asks to drop that exemption once the job has proven reliable. It has: no gfx950 failures in any run since it began running on PRs and nightlies.

Technical Details

  • Remove the expect_failure: {"linux": ["gfx950"]} declaration (and its TODO(ALMIOPEN-2674)) from the miopen-dbsync entry in fetch_test_configurations.py, and the comment that referred to it.
  • gfx950 then behaves like gfx942: a failure fails the job and the CI summary. The generic expect_failure plumbing and its other users (WSL entries) are unchanged.
  • Tests: the two tests that asserted gfx950 is advisory now assert it is blocking (no expect_failure declared; the scheduled job is not xfail). test_miopen_dbsync_gfx942_stays_blocking reads the flag with .get, since no job in this entry declares it any more.

Because the key is no longer emitted, test_component.yml sees expect_failure as absent, which evaluates as not-xfail, and the job name loses its (xfail) suffix.

Test Plan

  • python -m unittest tests.fetch_test_configurations_test.
  • Run the real fetch_test_configurations.py for gfx94X-dcgpu and gfx950-dcgpu at standard and confirm miopen-dbsync is requested on the CPU runner with no expect_failure.

Test Result

  • Unit tests: 107 pass (the 107 include the renamed and updated dbsync tests).
  • Selector at TEST_TYPE=standard: miopen-dbsync is requested for both gfx94X-dcgpu and gfx950-dcgpu on the CPU runner (aws-linux-scale-rocm-prod).

Evidence that gfx950 is reliable

Scan of rocm-libraries Multi-Arch CI runs created 2026-11-03 18:58 UTC to 2026-11-05 15:33 UTC (since the pin picked up the miopen -> miopen-dbsync selection), plus the Multi-Arch Nightly runs:

gfx950 miopen-dbsync jobs success cancelled queued failed
PRs touching projects/miopen (15 PRs) 19 1 1 0
PRs touching other projects 28 0 0 0
develop pushes 27 0 0 0
PR runs not mapped to a PR 11 0 0 0
Total 85 1 1 0
  • Because the job is advisory, a failure could be hidden behind continue-on-error. I checked all 85 completed gfx950 jobs: each log has StaticFDBSync gfx950 @ 256 CU and [ PASSED ] 1 test, no [ FAILED ] line and no failed step.
  • Scheduled nightlies (comprehensive) with a gfx950 dbsync job: 10-24 through 11-05, all green. gfx950 also passed in a 10-22 dispatch. The 10-23 nightly was cancelled before tests.
  • Runtime: about 3.7 to 5.2 minutes per gfx950 job (median 3.9), measured over the first 46 jobs.
  • gfx94X over the same window: 210 success, 3 cancelled, 2 queued, 0 failures.
  • One run (41963576281) never returned its job list and is not counted.

Risk Assessment

Risk Level: Low-Medium. CI selection only, but it makes a previously advisory job blocking. After this lands and rocm-libraries bumps its pin, a gfx950 dbsync failure turns the Multi-Arch CI Summary red on MIOpen PRs (and on PRs of projects that select miopen). The job runs on the CPU runner pool under a pinned rocjitsu, so a rocjitsu or runner-image regression would now block merges. PR-side history is about two days; the nightly series is about two weeks.

Drop the gfx950 expect_failure exemption on miopen-dbsync (ALMIOPEN-2674); gfx950 now blocks like gfx942.
@therock-pr-bot

therock-pr-bot Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Nov 5, 2026

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@ScottTodd ScottTodd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you link to a few workflow runs where the job is successful? The PR description is pretty long but I don't see that detail in there.

@JonathanLichtnerAMD

Copy link
Copy Markdown
Contributor Author

@ScottTodd Sure. Here are a few gfx950 miopen-dbsync jobs that passed. All are in ROCm/rocm-libraries, and each ran with the current advisory config, so the job name ends in (xfail). Since a failed xfail job could still show as green, I read each job's log: all have StaticFDBSync gfx950 @ 256 CU and [ PASSED ] 1 test, no [ FAILED ], and no failed step.

PRs touching projects/miopen (standard tier):

develop push (standard tier), 37f616f6: https://github.com/ROCm/rocm-libraries/actions/runs/42042911197/job/127337338841

Scheduled nightlies (comprehensive tier):

For the full picture: from 2026-11-03 18:58 UTC to 2026-11-05 15:33 UTC there were 85 completed gfx950 miopen-dbsync jobs on PRs and develop pushes, all green, none failed (19 of them on the 15 PRs that touch projects/miopen). The scheduled nightlies have been green on gfx950 from 2026-10-24 on. The PR-side history is only about two days, since the job only started running on PRs when the rocm-libraries pin picked up #9451.

@ScottTodd
ScottTodd self-requested a review November 6, 2026 19:52
@JonathanLichtnerAMD
JonathanLichtnerAMD merged commit 1a29cfe into main Nov 6, 2026
394 of 402 checks passed
@JonathanLichtnerAMD
JonathanLichtnerAMD deleted the users/jlichtne/miopen-dbsync-gfx950-blocking branch November 6, 2026 23:43
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants