feat(ci): dividing CPU vs GPU `tests_run_on` for more granular testing ability by geomin12 · Pull Request #8575 · ROCm/TheRock

geomin12 · 2026-09-28T18:30:17Z

Currently (from issue #8500), when GPU tests are not scheduled, it auto-excludes CPU tests. However, there may be scenarios where CPU tests are asked for, but GPU tests are not required

Changes:

adding tests_enabled as the main component of determining if test jobs are required (cpu or gpu)
adding test_runs_on_cpu to distinguish CPU test runner
when CPU and GPU tests run separately, so if GPU label is not provided, CPU tests still run. vice-versa too

Working in CI! Notice that GPU tests are disabled for gfx120X but it's working for CPU only tests here: https://github.com/ROCm/TheRock/actions/runs/36922378265/job/110602256870?pr=8575

Closes #8500

therock-pr-bot · 2026-09-28T18:31:17Z

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-28T18:31:36Z

🎉 All checks passed! This PR is ready for review.

Add separate CPU runner label (test-runs-on-cpu) so CPU-only test components can run even when GPU testing is gated for a family. Changes: - Add test-runs-on-cpu to family_info in configure_multi_arch_ci.py - Add test_runs_on_cpu input to test_artifacts.yml - Update gate condition to allow CPU tests when GPU is gated - Modify fetch_test_configurations.py to skip GPU runner assignment for CPU-only components and exclude GPU components when no runner - Update CI summary to show CPU Runner column This fixes #8500 where CPU-only components (hipfile, rocgdb-cpu, etc.) were blocked when a family used trigger_test_label_only. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add a boolean tests_enabled flag to family_info that is true when any test runner (GPU or CPU) is available. This provides a cleaner single flag for workflows to gate test jobs instead of checking multiple runner strings. The workflow gate now uses tests_enabled as the primary check, with fallback to runner string checks for backward compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When tests are disabled via _should_run_tests_for_family (tests_on_trigger), both GPU and CPU tests should be disabled. Previously, test_runs_on_cpu was always set regardless of trigger gating, causing CPU tests to run even when the tests_on_trigger check indicated tests should be skipped. This was problematic because tests_enabled was set to bool(test_runs_on or test_runs_on_cpu), so CPU tests would always enable the test jobs even when GPU tests were gated. Now we track tests_gated_by_trigger separately and clear test_runs_on_cpu when trigger gating is active. Also enhanced test_real_families_trigger_gating to verify that when tests are gated: test-runs-on, test-runs-on-cpu, and tests_enabled are all correctly disabled. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ive" This reverts commit caeafba.

therock-pr-bot · 2026-09-30T18:16:07Z

Pre-commit check failed

⛔ pre-commit failed

Please run locally:

python -m pip install pre-commit
pre-commit install
pre-commit run --all-files --show-diff-on-failure

This repo uses .pre-commit-config.yaml.

When GPU tests are gated (e.g., via tests_on_trigger), the policy decision was being lost because fetch_test_configurations.py re-read the runner label from the static family matrix instead of using the policy-gated value from configure_multi_arch_ci.py. This fix: 1. Passes TEST_RUNS_ON env var from workflow to fetch_test_configurations.py 2. When TEST_RUNS_ON is empty, GPU tests are gated but CPU-only components (linux_cpu_runner: True) can still run 3. fetch_test_configurations.py now respects the policy decision instead of overriding it with the static matrix value This allows CPU-only tests (hipfile, rocgdb-cpu, miopen-dbsync) to run even when a family's GPU testing is gated by trigger conditions. Fixes: #8500 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move CPU runner assignment from hardcoded workflow fallback to fetch_test_configurations.py, using the TEST_RUNS_ON_CPU environment variable passed from configure_multi_arch_ci.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Change logging.info() to print() for CPU-only runner selection messages to match the GPU runner selection output format in amdgpu_family_matrix.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

geomin12 · 2026-10-01T23:30:00Z

cancelling due to gfx110X queues

ScottTodd

Behavior changes LGTM. Logging could use some tweaks.

ScottTodd · 2026-10-02T22:21:43Z

-            --platform=${{ contains(inputs.test_runs_on, 'windows') && 'windows' || 'linux' }}
+            --platform=${{ contains(inputs.test_runs_on, 'windows') && 'windows' || contains(inputs.test_runs_on_cpu, 'windows') && 'windows' || 'linux' }}


there should be some more concise way to say --platform=[linux,windows] 🤔

ScottTodd · 2026-10-02T22:31:18Z

+            if is_cpu_only:
+                if test_runs_on_cpu:
+                    component["test_runner"] = test_runs_on_cpu
+                    print(f"  {job_name}: CPU-only, using runner: {test_runs_on_cpu}")
+                else:
+                    print(
+                        f"Excluding job {job_name}: CPU runner required but none configured"
+                    )
+                    continue


Can you add some section headings / newlines / etc. between some of these logs? May also want to sort how the logic is run differently.

I'm looking here: https://github.com/ROCm/TheRock/actions/runs/36922378265/job/110601872675?pr=8575#step:7:112 and where it flips from includes to excludes is hard to spot:

INFO:root:Including job tensilelite with test_type quick INFO:root:Including job tensilelite-common with test_type quick INFO:root:Including job origami with test_type quick gfx90a/linux: test-runs-on-labels = [{'label': 'linux-gfx90a-gpu-rocm', 'weight': 0.1, 'count': 2}, {'label': 'linux-gfx90a-1gpu-ossci-rocm', 'weight': 0.5, 'count': 10}, {'label': 'linux-mi210-1gpu-ossci-rocm', 'weight': 0.4, 'count': 8}] gfx90a/linux: test-runs-on-multi-gpu = linux-gfx90a-8gpu-ossci-rocm gfx90a/linux: test-runs-on-multi-gpu-labels = [{'label': 'linux-gfx90a-8gpu-ossci-rocm', 'weight': 0.5, 'count': 2}, {'label': 'linux-mi210-8gpu-ossci-rocm', 'weight': 0.5, 'count': 2}] gfx950/linux: test-runs-on-labels = [{'label': 'linux-gfx950-1gpu-ccs-ossci-rocm', 'weight': 1.0, 'count': 3}] gfx950/linux: test-runs-on-sandbox = linux-gfx950-8gpu-asan-sandbox-rocm gfx950/linux: test-runs-on-sandbox-labels = [{'label': 'linux-gfx950-8gpu-asan-sandbox-rocm', 'weight': 1.0, 'count': 4}] gfx950/linux: test-runs-on-multi-gpu-labels = [{'label': 'linux-gfx950-8gpu-ccs-ossci-rocm', 'weight': 1.0, 'count': 2}] hipfile: CPU-only, using runner: aws-linux-scale-rocm-prod rocgdb-cpu: CPU-only, using runner: aws-linux-scale-rocm-prod profiler-hub: CPU-only, using runner: aws-linux-scale-rocm-prod Setting github output: { "sanity_component": "null", "components": "[{\"additional_requirements_files\": [], \"job_name\": \"hipfile\", \"fetch_artifact_args\": \"--hipfile --tests\", \"timeout_minutes\": 15, \"test_script\": \"python build_tools/github_actions/test_executable_scripts/test_hipfile.py\", \"platform\": [\"linux\"], \"linux_cpu_runner\": true, \"total_shards_dict\": {\"linux\": 1}, \"test_type\": \"quick\", \"shard_arr\": [1], \"total_shards\": 1, \"test_runner\": \"aws-linux-scale-rocm-prod\", \"container_options\": \"--ipc host --user 0:0 --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 --security-opt seccomp=unconfined\"}, {\"additional_requirements_files\": [], \"fetch_artifact_args\": \"--debug-tools --tests\", \"timeout_minutes\": 30, \"platform\": [\"linux\"], \"total_shards\": 1, \"container_image\": \"ghcr.io/rocm/no_rocm_image_ubuntu24_04_rocgdb@sha256:aa3f8966fcdefca04d4c04fb10ae7f8b654d1bb1cc6a894ea7089e5a01953197\", \"container_options\": \"--ipc host --user 0:0 --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE\", \"job_name\": \"rocgdb-cpu\", \"test_script\": \"python ./build/tests/rocgdb/test_rocgdb.py --parallel -f 0.25 --tests gdb.dwarf2\", \"linux_cpu_runner\": true, \"test_type\": \"quick\", \"shard_arr\": [1], \"test_runner\": \"aws-linux-scale-rocm-prod\"}, {\"additional_requirements_files\": [], \"job_name\": \"profiler-hub\", \"timeout_minutes\": 5, \"test_script\": \"python build_tools/github_actions/test_executable_scripts/test_profiler_hub_install.py\", \"platform\": [\"linux\"], \"linux_cpu_runner\": true, \"total_shards_dict\": {\"linux\": 1}, \"test_type\": \"quick\", \"shard_arr\": [1], \"total_shards\": 1, \"test_runner\": \"aws-linux-scale-rocm-prod\", \"container_options\": \"--ipc host --user 0:0 --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 --security-opt seccomp=unconfined\"}]", "platform": "linux" } OUTPUT sanity_component=null OUTPUT components=[{"additional_requirements_files": [], "job_name": "hipfile", "fetch_artifact_args": "--hipfile --tests", "timeout_minutes": 15, "test_script": "python build_tools/github_actions/test_executable_scripts/test_hipfile.py", "platform": ["linux"], "linux_cpu_runner": true, "total_shards_dict": {"linux": 1}, "test_type": "quick", "shard_arr": [1], "total_shards": 1, "test_runner": "aws-linux-scale-rocm-prod", "container_options": "--ipc host --user 0:0 --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 --security-opt seccomp=unconfined"}, {"additional_requirements_files": [], "fetch_artifact_args": "--debug-tools --tests", "timeout_minutes": 30, "platform": ["linux"], "total_shards": 1, "container_image": "ghcr.io/rocm/no_rocm_image_ubuntu24_04_rocgdb@sha256:aa3f8966fcdefca04d4c04fb10ae7f8b654d1bb1cc6a894ea7089e5a01953197", "container_options": "--ipc host --user 0:0 --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE", "job_name": "rocgdb-cpu", "test_script": "python ./build/tests/rocgdb/test_rocgdb.py --parallel -f 0.25 --tests gdb.dwarf2", "linux_cpu_runner": true, "test_type": "quick", "shard_arr": [1], "test_runner": "aws-linux-scale-rocm-prod"}, {"additional_requirements_files": [], "job_name": "profiler-hub", "timeout_minutes": 5, "test_script": "python build_tools/github_actions/test_executable_scripts/test_profiler_hub_install.py", "platform": ["linux"], "linux_cpu_runner": true, "total_shards_dict": {"linux": 1}, "test_type": "quick", "shard_arr": [1], "total_shards": 1, "test_runner": "aws-linux-scale-rocm-prod", "container_options": "--ipc host --user 0:0 --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 --security-opt seccomp=unconfined"}] OUTPUT platform=linux INFO:root:Including job hipblas with test_type quick INFO:root:Including job amdsmi with test_type quick ... INFO:root:Including job rocdecode with test_type quick INFO:root:Including job rocjpeg with test_type quick INFO:root:Including job rpp with test_type quick INFO:root:Including job aqlprofile with test_type quick INFO:root:Including job rocrtst with test_type quick INFO:root:Including job hiptensor with test_type quick INFO:root:Excluding job sanity: GPU runner required but none configured INFO:root:Excluding job hip-tests: GPU runner required but none configured INFO:root:Excluding job rocblas: GPU runner required but none configured INFO:root:Excluding job rocroller: GPU runner required but none configured INFO:root:Excluding job tensilelite: GPU runner required but none configured INFO:root:Excluding job tensilelite-common: GPU runner required but none configured INFO:root:Excluding job origami: GPU runner required but none configured

This would be easier to read:

INFO:root:Determining jobs to include based on components affected INFO:root:Including job rocdecode with test_type quick INFO:root:Including job rocjpeg with test_type quick INFO:root:Including job rpp with test_type quick INFO:root:Including job aqlprofile with test_type quick INFO:root:Including job rocrtst with test_type quick INFO:root:Including job hiptensor with test_type quick INFO:root: INFO:root:Determining jobs to exclude based on CPU/GPU runner availability INFO:root:Excluding job sanity: GPU runner required but none configured INFO:root:Excluding job hip-tests: GPU runner required but none configured INFO:root:Excluding job rocblas: GPU runner required but none configured INFO:root:Excluding job rocroller: GPU runner required but none configured INFO:root:Excluding job tensilelite: GPU runner required but none configured INFO:root:Excluding job tensilelite-common: GPU runner required but none configured INFO:root:Excluding job origami: GPU runner required but none configured

and this is also confusing:

# include then exclude later INFO:root:Including job libhipcxx_hiprtc with test_type quick ... INFO:root:Excluding job libhipcxx_hiprtc: GPU runner required but none configured # Flipped order (include then include again) rocgdb-cpu: CPU-only, using runner: aws-linux-scale-rocm-prod ... INFO:root:Including job rocgdb-cpu with test_type quick

Could do something like:

INFO:root:Requesting job libhipcxx_hiprtc with test_type quick INFO:root: Rejecting job libhipcxx_hiprtc: GPU runner required but none configured INFO:root:Including job rocgdb-cpu with test_type quick INFO:root: Including job rocgdb-cpu (CPU-only, using runner: aws-linux-scale-rocm-prod)

- Change "Including job" to "Requesting job" in first pass to clarify that jobs are candidates, not yet confirmed - Add section header "Assigning runners to requested jobs..." before runner assignment phase for better log readability - Use logging.info() consistently instead of print() for CPU runner logs - Indent exclusion messages for visual grouping under the section header - Simplify --platform detection to only check test_runs_on Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JIRA ID: ALMIOPEN-2604 Fixes ROCm/rocm-libraries#2266 ## Motivation Add gfx950 (MI355X, 256 CU) coverage for the `miopen-dbsync` GHA test component. The runner script and gfx950 FAMILY_MAP entry already exist in rocm-libraries (`run_dbsync_rocjitsu.py`). This relies on TheRock #8500/#8575 (merged), which let CPU-only components run when GPU tests are gated. ## Technical Details In `fetch_test_configurations.py`, add `gfx950` to the `miopen-dbsync` linux `include_family` list, and declare `"expect_failure_family": {"linux": ["gfx950"]}` on the same entry so `test_component.yml` treats the gfx950 job as `continue-on-error` (advisory) while gfx942 stays blocking. A TODO references ALMIOPEN-2674 to remove it once gfx950 is stable. `expect_failure_family` is new generic plumbing (per review feedback): any component can declare the families it is advisory on, matched with the same `_family_matches` helper as `include_family`/`exclude_family`, instead of a per-component conditional in `run()`. It is documented in the key reference block above `test_matrix`, and the declaration is popped from the merged job config so only the `expect_failure` boolean the workflows read is emitted. Matching via `_family_matches` also accepts both the family group string (`gfx950-dcgpu`) and the individual target (`gfx950`). Add unit tests in `fetch_test_configurations_test.py` covering job selection for gfx942/gfx950 including the gated-GPU case, plus the `expect_failure_family` plumbing itself (target/group matching, per-platform scoping, key not emitted, matrix not mutated). ## Risk Assessment **Risk Level: 🟢 Low** CI config only; gfx950 is non-blocking and gfx942 behavior is unchanged. ## Test Plan Run `fetch_test_configurations_test.py` locally. Then validate in CI: a multi-arch dispatch for gfx950 (needs `test_filter:full`; a `test:*` label alone leaves the tier at `quick`, which `miopen-dbsync` skips), and a rocm-libraries PR-event run (no gfx950 label) to confirm CPU-pool scheduling. ## Test Result - All 83 unit tests in `fetch_test_configurations_test.py` pass locally (`black` clean). - After the `expect_failure_family` refactor, the emitted component JSON was re-checked directly: `AMDGPU_FAMILIES=gfx950-dcgpu` emits `"expect_failure": true` with no `expect_failure_family` key, `gfx94X-dcgpu` emits neither (stays blocking), and `AMDGPU_FAMILIES=gfx950` (target form) now also emits `"expect_failure": true` — the previous `in {"gfx950-dcgpu"}` check did not match that form. - `multi_arch_ci` dispatch (gfx950, `test:miopen-dbsync,test_filter:full`, rocm-libraries `develop` @ 7943b63): https://github.com/ROCm/TheRock/actions/runs/37647187163. `miopen-dbsync (gfx950-dcgpu) (xfail)` passed: StaticFDBSync gfx950 @ 256 CU `[ OK ]` in ~92s, `[ PASSED ] 1 test`, job ~4 min. Note a dispatch routes the job to the family GPU runner, so this does not exercise the CPU pool. - Sweep with this PR's head + rocm-libraries `develop` @ 58179e4b: `multi_arch_ci` dispatch (`gfx94x,gfx950`, `test:miopen-dbsync,test_filter:full`) https://github.com/ROCm/TheRock/actions/runs/38498125276 succeeded. `miopen-dbsync (gfx94X-dcgpu)` (blocking): StaticFDBSync gfx942 @ 304 CU and @ 228 CU `[ OK ]`. `miopen-dbsync (gfx950-dcgpu) (xfail)`: gfx950 @ 256 CU `[ OK ]`. Dispatch jobs run on the family GPU runners, so the GPU-gated CPU-pool path is covered by unit tests only. --------- Co-authored-by: Claude <noreply@anthropic.com>

github-project-automation Bot added this to TheRock Triage Sep 28, 2026

github-project-automation Bot moved this to TODO in TheRock Triage Sep 28, 2026

geomin12 and others added 2 commits September 30, 2026 09:10

geomin12 force-pushed the users/geomin12/fix-cpu-test-gate branch from 76da4a0 to b49f5c3 Compare September 30, 2026 16:13

geomin12 and others added 2 commits September 30, 2026 11:14

Revert "fix(ci): gate CPU tests when trigger-based test gating is act…

25a6a70

…ive" This reverts commit caeafba.

geomin12 force-pushed the users/geomin12/fix-cpu-test-gate branch from 1c36564 to 9f3e68f Compare October 1, 2026 16:20

geomin12 force-pushed the users/geomin12/fix-cpu-test-gate branch from 9f3e68f to 496c41f Compare October 1, 2026 16:25

fix(ci): use print() for CPU runner selection logs for consistency

5ef86e0

Change logging.info() to print() for CPU-only runner selection messages to match the GPU runner selection output format in amdgpu_family_matrix.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

geomin12 requested a review from JonathanLichtnerAMD October 1, 2026 23:31

geomin12 marked this pull request as ready for review October 1, 2026 23:31

geomin12 requested review from ScottTodd and jayhawk-commits as code owners October 1, 2026 23:32

ScottTodd approved these changes Oct 2, 2026

View reviewed changes

geomin12 force-pushed the users/geomin12/fix-cpu-test-gate branch 2 times, most recently from c22d185 to 58ed623 Compare October 5, 2026 17:14

geomin12 merged commit 3f95c88 into main Oct 5, 2026
162 of 167 checks passed

geomin12 deleted the users/geomin12/fix-cpu-test-gate branch October 5, 2026 21:33

JonathanLichtnerAMD mentioned this pull request Oct 6, 2026

ci(miopen): run miopen-dbsync on gfx950 as advisory #8813

Merged

amd-justchen mentioned this pull request Oct 8, 2026

feat(ci): run hip-tests and rocrtst on WSL GPU runners alongside native Linux #8457

Merged

Navigation Menu

Repository navigation

feat(ci): dividing CPU vs GPU `tests_run_on` for more granular testing ability - #8575

feat(ci): dividing CPU vs GPU `tests_run_on` for more granular testing ability#8575
geomin12 merged 8 commits into
mainfrom
users/geomin12/fix-cpu-test-gate

geomin12 commented Sep 28, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 28, 2026

Uh oh!

therock-pr-bot Bot commented Sep 30, 2026

Uh oh!

geomin12 commented Oct 1, 2026

Uh oh!

ScottTodd left a comment

Uh oh!

ScottTodd Oct 2, 2026

Uh oh!

ScottTodd Oct 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		--platform=${{ contains(inputs.test_runs_on, 'windows') && 'windows' \|\| 'linux' }}
		--platform=${{ contains(inputs.test_runs_on, 'windows') && 'windows' \|\| contains(inputs.test_runs_on_cpu, 'windows') && 'windows' \|\| 'linux' }}

Conversation

geomin12 commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

therock-pr-bot Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Checks Passed — Ready for Review

Uh oh!

therock-pr-bot Bot commented Sep 28, 2026

Uh oh!

therock-pr-bot Bot commented Sep 30, 2026

Pre-commit check failed

Uh oh!

geomin12 commented Oct 1, 2026

Uh oh!

ScottTodd left a comment

Choose a reason for hiding this comment

Uh oh!

ScottTodd Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

ScottTodd Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

geomin12 commented Sep 28, 2026 •

edited

Loading

therock-pr-bot Bot commented Sep 28, 2026 •

edited

Loading