Repository navigation
ci(miopen): run miopen-dbsync on gfx950 as advisory - #8813
Merged
JonathanLichtnerAMD merged 5 commits intoOct 20, 2026
Merged
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Pre-commit check failed⛔ pre-commit failed Please run locally:
This repo uses |
|
🎉 All checks passed! This PR is ready for review. |
Contributor
Author
Validation update (ALMIOPEN-2604)
Sweep: multi_arch_ci run 38498125276, dispatched with
gfx942 stays blocking and gfx950 is advisory via Not covered by CI:
Other CI on the previous run: pre-commit is fixed (black, |
JonathanLichtnerAMD
marked this pull request as ready for review
October 14, 2026 13:56
geomin12
reviewed
Oct 14, 2026
Replace the hard-coded `if key == "miopen-dbsync" and amdgpu_families in
{"gfx950-dcgpu"}` with a declarative "expect_failure_family" key, so a
component's advisory arches are visible in its matrix entry instead of in a
conditional further down the file.
The key is a platform-keyed map of family entries, mirroring include_family /
exclude_family, and is matched with the same _family_matches helper. That also
fixes an asymmetry in the old check: it compared only the family group string
("gfx950-dcgpu"), while the entry's own include_family lists the gfx target
("gfx950"). Both forms now work.
The declaration is popped from the merged job config so only the boolean the
workflows read ("expect_failure" -> continue-on-error plus an "(xfail)" job-name
suffix in test_component.yml) is emitted.
Also drops an inaccurate comment claiming this mirrored "the benchmark
components" -- nothing else in this repo sets expect_failure.
Known gap: the Windows hip-tests PAL/ROCR branch builds its entries from an
explicit field allowlist and returns before this plumbing runs, so
expect_failure_family would not apply there. No component needs it today.
Co-Authored-By: Claude <noreply@anthropic.com>
geomin12
reviewed
Oct 16, 2026
geomin12
left a comment
geomin12
left a comment
Contributor
There was a problem hiding this comment.
one more thing: does this even work? you can use labels in the PR (https://github.com/ROCm/TheRock/blob/main/docs/development/ci_behavior_manipulation.md) to test.
Per review:
- Rename "expect_failure_family" to "expect_failure". The matrix entry is
self-explanatory without the suffix.
- Drop the prose comment above the declaration; the key and the remaining
TODO(ALMIOPEN-2674) say everything it did.
- Use get() rather than pop().
The pop was not incidental: test_component.yml compares the emitted value
with `== true`, and `{**test_matrix[key]}` copies the declaration verbatim,
so under the new name a non-matching family would have emitted
`"expect_failure": {"linux": [...]}` -- a map sitting under the key the
workflow reads. Resolving the declaration to a boolean in place keeps get()
and guarantees the emitted value is always a bool.
Jobs on a non-matching family now carry `"expect_failure": false` instead of
omitting the key; components that never declare it are unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
JonathanLichtnerAMD
added
ci:gfx950-dcgpu
Contributor
Author
|
Yes — validated on this PR with labels, as you suggested. Added
and Two things this proved that the earlier
First attempt did fail, but on infra — gfx950 sanity died on On the label question from earlier: the recipe above works, so The three nits are addressed in f1b9c1c. |
geomin12
approved these changes
Oct 20, 2026
1 task done
JonathanLichtnerAMD
deleted the
users/jlichtne/ALMIOPEN-2604-gfx950-dbsync
branch
October 20, 2026 22:08
yassinsolim
added a commit
that referenced
this pull request
Oct 22, 2026
The miopen-dbsync component is opted in to gfx942 and gfx950, so the five Navi system find/perf DB pairs that rocm-libraries ships (gfx110060, gfx110220, gfx115128, gfx120020, gfx120140) have no gate at all. StaticFDBSync has a param for each, but they are skipped everywhere the lane currently lands: in rocm-libraries develop run 37944725308 all four Test miopen shards of Windows::release / Test gfx110X-all ran on gfx1101 runners and params 7, 8, 9, 10 and 11 all reported SKIPPED. Across the last 40 completed develop runs, 90 of 90 gfx110X-all MIOpen jobs landed on gfx1101. That is how ROCm/rocm-libraries#13394 reached develop. Add gfx1100, gfx1102, gfx1151, gfx1200 and gfx1201 to the linux include_family list. gfx1151 is declared advisory through the expect_failure plumbing added by #8813, with a TODO on ROCm/rocm-libraries#13429: its dbsync passes under the rocjitsu interposer but has never run on real gfx1151 silicon, where CK is_applicable aborts with invalid device function, and the interposer CU correction is load-bearing there (the config declares 32 CU while gfx115128 is a 40 CU database). gfx1100, gfx1102, gfx1200 and gfx1201 are blocking. This costs no new builds. amdgpu_family_matrix.py already builds gfx110x, gfx1151 and gfx120x on linux for presubmit, postsubmit, submodule_bump and nightly, and the component is linux_cpu_runner, so it takes no GPU test capacity. Those families carry tests_on_trigger ["nightly"], so this lands as a nightly gate. Unit tests cover selection by family group string and by individual gfx target, the linux-only scoping, CPU-runner scheduling when the GPU runner is gated, and the emitted expect_failure boolean for every included target. Requires ROCm/rocm-libraries#13411, which adds the matching FAMILY_MAP entries to run_dbsync_rocjitsu.py.
amd-justchen
added a commit
that referenced
this pull request
Oct 23, 2026
main now resolves a component's "expect_failure" from a map keyed by platform (#8813). Declare {"wsl": ["gfx110X-all"]} on hip-tests and rocrtst instead of setting expect_failure inline for platform == "wsl", so the xfail sits with the component entry (review: easier to find).
JonathanLichtnerAMD
added a commit
that referenced
this pull request
Oct 27, 2026
JIRA ID: ALMIOPEN-3021 ## Motivation `miopen-dbsync` (GPU-free rocjitsu StaticFDBSync, #8813 added gfx950 as advisory) is never scheduled for PR-style runs. A rocm-libraries PR touching `projects/miopen` selects `projects_to_test=hipdnn-integration-tests,hipdnn-samples,miopen,miopenprovider` at `test_type=standard`; `fetch_test_configurations.py` only schedules a job whose key is in that list (or `*`), so `miopen-dbsync` only runs in nightlies (`*`) or manual dispatch. Verified on a throwaway rocm-libraries PR against TheRock `b0958085`: the gfx94X "Configure test matrix" log requested sanity, miopen, hipdnn-integration-tests, hipdnn-samples, miopenprovider and no `miopen-dbsync`. ## Technical Details Add `"miopen": ["miopen", "miopen-dbsync"]` to `_CI_TEST_SELECTOR_ALIASES` in `test_tools/determine_rocm_test_dependencies.py` (same pattern as `tensilelite` -> `tensilelite-common`). `miopen` stays selected; `miopen-dbsync` is added whenever `miopen` is in the result. The job still self-limits to standard/comprehensive/full tiers, gfx942 (blocking) and gfx950 (xfail) on the CPU runner, so quick-tier and other families are unchanged. This is the same alias #8990 adds, but #8990 merged into the stacked branch of #8757 and so is not on `main`; this PR does not depend on #8757. If #8757 lands later it should carry the same alias (its reachability guard already requires it). ## Test Plan - Unit test `test_cli_selects_miopen_dbsync_with_miopen`: `--changed-projects miopen` outputs exactly `{miopen, miopen-dbsync}`. - Run the real selection scripts locally for gfx94X-dcgpu and gfx950-dcgpu at standard and quick tiers. - End to end in rocm-libraries CI: a PR pinned to a TheRock commit containing this change, with a MIOpen source change. ## Test Result - `cd test_tools && python3 -m unittest tests.determine_rocm_test_dependencies_test`: 75 tests OK (74 before). - `build_tools/github_actions/tests/fetch_test_configurations_test.py`: passes unchanged. - Local scripts: `determine_rocm_test_dependencies.py --changed-projects projects/miopen --gha-output` now emits `hipdnn-integration-tests,hipdnn-samples,miopen,miopen-dbsync,miopenprovider`. `fetch_test_configurations.py --platform=linux` at `TEST_TYPE=standard` requests `miopen-dbsync` on the CPU runner for both gfx94X-dcgpu and gfx950-dcgpu; at `quick` it logs `Excluding job miopen-dbsync` for both. - End to end: throwaway rocm-libraries PR #14288 (closed) pinned TheRock to `2104ff3c` and touched a `projects/miopen` source file. `2104ff3c` was this change plus a prototype that treats `.github/actions/ci-env/*` as a workflow change in `configure_external_repo_ci.py`. That prototype is NOT part of this PR. It was needed only because a pin bump edits `ci-env/action.yml`, which otherwise forces run-all at quick tier, where `miopen-dbsync` is skipped by design. Run: https://github.com/ROCm/rocm-libraries/actions/runs/39985392063 - `Test miopen-dbsync (gfx94X-dcgpu)`: passed. StaticFDBSync gfx942 @ 304 CU and @ 228 CU, each `[ PASSED ] 1 test`. - `Test miopen-dbsync (gfx950-dcgpu) (xfail)`: passed. StaticFDBSync gfx950 @ 256 CU, `[ PASSED ] 1 test`. - Both ran on the CPU runner `aws-linux-scale-rocm-prod`; the gfx94X "Configure test matrix" log showed `TEST_TYPE: standard` and `Requesting job miopen-dbsync`. - Other jobs in that run were not waited on; the run was cancelled after the dbsync jobs finished. - Not run: `label_consistency_test` (module did not import in my environment, before or after this change), pre-commit. ## Risk Assessment **Risk Level: Low.** CI selection only; adds one job to MIOpen-touching PR runs, CPU runner, gfx950 non-blocking.
JonathanLichtnerAMD
added a commit
that referenced
this pull request
Nov 6, 2026
JIRA ID: ALMIOPEN-2674 ## Motivation `miopen-dbsync` on `gfx950-dcgpu` was re-added as advisory (#8813, `expect_failure`) so a regression during re-qualification would not block unrelated PRs. ALMIOPEN-2674 asks to drop that exemption once the job has proven reliable. It has: no gfx950 failures in any run since it began running on PRs and nightlies. ## Technical Details - Remove the `expect_failure: {"linux": ["gfx950"]}` declaration (and its `TODO(ALMIOPEN-2674)`) from the `miopen-dbsync` entry in `fetch_test_configurations.py`, and the comment that referred to it. - gfx950 then behaves like gfx942: a failure fails the job and the CI summary. The generic `expect_failure` plumbing and its other users (WSL entries) are unchanged. - Tests: the two tests that asserted gfx950 is advisory now assert it is blocking (no `expect_failure` declared; the scheduled job is not xfail). `test_miopen_dbsync_gfx942_stays_blocking` reads the flag with `.get`, since no job in this entry declares it any more. Because the key is no longer emitted, `test_component.yml` sees `expect_failure` as absent, which evaluates as not-xfail, and the job name loses its `(xfail)` suffix. ## Test Plan - `python -m unittest tests.fetch_test_configurations_test`. - Run the real `fetch_test_configurations.py` for `gfx94X-dcgpu` and `gfx950-dcgpu` at `standard` and confirm `miopen-dbsync` is requested on the CPU runner with no `expect_failure`. ## Test Result - Unit tests: 107 pass (the 107 include the renamed and updated dbsync tests). - Selector at `TEST_TYPE=standard`: `miopen-dbsync` is requested for both `gfx94X-dcgpu` and `gfx950-dcgpu` on the CPU runner (`aws-linux-scale-rocm-prod`). ## Evidence that gfx950 is reliable Scan of rocm-libraries Multi-Arch CI runs created 2026-11-03 18:58 UTC to 2026-11-05 15:33 UTC (since the pin picked up the `miopen` -> `miopen-dbsync` selection), plus the Multi-Arch Nightly runs: | gfx950 `miopen-dbsync` jobs | success | cancelled | queued | failed | |---|---|---|---|---| | PRs touching `projects/miopen` (15 PRs) | 19 | 1 | 1 | 0 | | PRs touching other projects | 28 | 0 | 0 | 0 | | `develop` pushes | 27 | 0 | 0 | 0 | | PR runs not mapped to a PR | 11 | 0 | 0 | 0 | | **Total** | **85** | **1** | **1** | **0** | - Because the job is advisory, a failure could be hidden behind `continue-on-error`. I checked all 85 completed gfx950 jobs: each log has `StaticFDBSync gfx950 @ 256 CU` and `[ PASSED ] 1 test`, no `[ FAILED ]` line and no failed step. - Scheduled nightlies (`comprehensive`) with a gfx950 dbsync job: 10-24 through 11-05, all green. gfx950 also passed in a 10-22 dispatch. The 10-23 nightly was cancelled before tests. - Runtime: about 3.7 to 5.2 minutes per gfx950 job (median 3.9), measured over the first 46 jobs. - gfx94X over the same window: 210 success, 3 cancelled, 2 queued, 0 failures. - One run (41963576281) never returned its job list and is not counted. ## Risk Assessment **Risk Level: Low-Medium.** CI selection only, but it makes a previously advisory job blocking. After this lands and rocm-libraries bumps its pin, a gfx950 dbsync failure turns the Multi-Arch CI Summary red on MIOpen PRs (and on PRs of projects that select `miopen`). The job runs on the CPU runner pool under a pinned rocjitsu, so a rocjitsu or runner-image regression would now block merges. PR-side history is about two days; the nightly series is about two weeks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA ID: ALMIOPEN-2604
Fixes ROCm/rocm-libraries#2266
Motivation
Add gfx950 (MI355X, 256 CU) coverage for the
miopen-dbsyncGHA test component. The runner script and gfx950 FAMILY_MAP entry already exist in rocm-libraries (run_dbsync_rocjitsu.py). This relies on TheRock #8500/#8575 (merged), which let CPU-only components run when GPU tests are gated.Technical Details
In
fetch_test_configurations.py, addgfx950to themiopen-dbsynclinuxinclude_familylist, and declare"expect_failure_family": {"linux": ["gfx950"]}on the same entry sotest_component.ymltreats the gfx950 job ascontinue-on-error(advisory) while gfx942 stays blocking. A TODO references ALMIOPEN-2674 to remove it once gfx950 is stable.expect_failure_familyis new generic plumbing (per review feedback): any component can declare the families it is advisory on, matched with the same_family_matcheshelper asinclude_family/exclude_family, instead of a per-component conditional inrun(). It is documented in the key reference block abovetest_matrix, and the declaration is popped from the merged job config so only theexpect_failureboolean the workflows read is emitted. Matching via_family_matchesalso accepts both the family group string (gfx950-dcgpu) and the individual target (gfx950).Add unit tests in
fetch_test_configurations_test.pycovering job selection for gfx942/gfx950 including the gated-GPU case, plus theexpect_failure_familyplumbing itself (target/group matching, per-platform scoping, key not emitted, matrix not mutated).Risk Assessment
Risk Level: 🟢 Low
CI config only; gfx950 is non-blocking and gfx942 behavior is unchanged.
Test Plan
Run
fetch_test_configurations_test.pylocally. Then validate in CI: a multi-arch dispatch for gfx950 (needstest_filter:full; atest:*label alone leaves the tier atquick, whichmiopen-dbsyncskips), and a rocm-libraries PR-event run (no gfx950 label) to confirm CPU-pool scheduling.Test Result
fetch_test_configurations_test.pypass locally (blackclean).expect_failure_familyrefactor, the emitted component JSON was re-checked directly:AMDGPU_FAMILIES=gfx950-dcgpuemits"expect_failure": truewith noexpect_failure_familykey,gfx94X-dcgpuemits neither (stays blocking), andAMDGPU_FAMILIES=gfx950(target form) now also emits"expect_failure": true— the previousin {"gfx950-dcgpu"}check did not match that form.multi_arch_cidispatch (gfx950,test:miopen-dbsync,test_filter:full, rocm-librariesdevelop@ 7943b63): https://github.com/ROCm/TheRock/actions/runs/37647187163.miopen-dbsync (gfx950-dcgpu) (xfail)passed: StaticFDBSync gfx950 @ 256 CU[ OK ]in ~92s,[ PASSED ] 1 test, job ~4 min. Note a dispatch routes the job to the family GPU runner, so this does not exercise the CPU pool.develop@ 58179e4b:multi_arch_cidispatch (gfx94x,gfx950,test:miopen-dbsync,test_filter:full) https://github.com/ROCm/TheRock/actions/runs/38498125276 succeeded.miopen-dbsync (gfx94X-dcgpu)(blocking): StaticFDBSync gfx942 @ 304 CU and @ 228 CU[ OK ].miopen-dbsync (gfx950-dcgpu) (xfail): gfx950 @ 256 CU[ OK ]. Dispatch jobs run on the family GPU runners, so the GPU-gated CPU-pool path is covered by unit tests only.