feat(ci): run TensileLite common GEMM tests in a tensilelite-common job by tony-davis · Pull Request #8459 · ROCm/TheRock

tony-davis · 2026-09-23T19:33:09Z

Motivation

TensileLite's Tensile/Tests/common suite is the one that actually generates, builds, and runs GEMM kernels against a CPU reference. Right now it only runs in Math CI's preliminary job. TheRock runs the TensileLite unit tree, but not this. This PR adds a tensilelite-common job so TheRock covers it too, against the installed artifact, on the same four architectures preliminary uses.

Companion rocm-libraries PR (adds the hw-common category this job runs): ROCm/rocm-libraries#12492

Technical Details

pytest_runner.py: a new TEST_CATEGORY env var overrides TEST_TYPE, for jobs that always run one category regardless of tier. Adds tensilelite-common to INSTALLED_COMPONENTS (same install tree as tensilelite), so TEST_COMPONENT resolves and the JUnit report gets its own name.
fetch_test_configurations.py: a new tensilelite-common entry running TEST_CATEGORY=hw-common through pytest_runner.py, with the same tensilelite requirements file. It's a separate job rather than another stage on tensilelite, so a unit-test failure can't hide the GEMM result. test:tensilelite now selects both jobs.
Opt-in by family: gfx90a, gfx94X-dcgpu, gfx950-dcgpu, gfx120X-all. This has to be opt-in. Selection inside the suite works by each config declaring skip-gfxNNNN, and a family with no declarations (gfx1103, gfx115X) would try to run all ~417 configs.
determine_rocm_test_dependencies.py: the tensilelite graph key now fans out to both CI selectors via _CI_TEST_SELECTOR_ALIASES, like rocgdb does. I put it there instead of a test_include in test_policies.toml, because test_include only fires when tensilelite is the changed project itself. The alias fires wherever the key lands in the result. So TensileLite, shared/origami, shared/stinkytofu, and shared/mxdatagenerator changes all select it, and hipBLASLt-only or rocroller-only changes select neither.
Unsharded, 180 min timeout. With current family flags only gfx942 (~1 hour) is on the PR path. gfx950 (~2 hours) and gfx90a are postsubmit, and gfx120X is nightly. The timeout is sized for gfx950. If gfx950 test machines come back per-PR ([ci] Re-enable gfx950-dcgpu test machines for rocm-libraries and TheRock when enough capacity is back #3288), I think we'd want to shard.
Graceful skip until the component catches up. TheRock's pinned rocm-libraries doesn't define hw-common yet, and submodule bumps follow their own governance. So when a job pins a category with TEST_CATEGORY and the installed component doesn't define it, pytest_runner.py exits 0 with a ::warning:: annotation instead of failing. A missing tier from TEST_TYPE is still an error. The job starts running for real once the pin includes ci(tensilelite): add hw-common category for real-hardware common GEMM tests rocm-libraries#12492.

Test Plan

pytest build_tools/github_actions/tests test_tools/tests
Mutation check on the new tests: dropped gfx950 from include_family, emptied include_family, removed the TEST_LABEL_GROUPS entry, made resolve_test_category() ignore TEST_CATEGORY, and removed the selector alias. Each one turns at least one new test red.
Local dry run of fetch_test_configurations.py with the selector output for a projects/hipblaslt/tensilelite change, per family.
pre-commit run on the changed files.
GPU bring-up: this draft with the test:tensilelite-common label.

Test Result

1046 passed, 2 skipped. The one failure (post_stage_upload_test.py::test_archives_ccache_subdirectory) is pyzstd missing from my local venv and is unrelated.
The dry run emits tensilelite-common for gfx94X, gfx950, gfx90a, and gfx120X, each on its 1-GPU runner label with /dev/kfd and /dev/dri mapped. It isn't emitted for gfx1151.
pre-commit passes.
GPU bring-up: pending.
Ran hw-common on gfx942 via a rocm-libraries dispatch pinned to this PR (run 35915539444). 136 tests were selected, matching Math CI. Every failure came from a worker-to-GPU bug in TensileLite's conftest, now fixed in ci(tensilelite): add hw-common category for real-hardware common GEMM tests rocm-libraries#12492; every test on a correctly assigned worker passed.
Ran pytest_runner.py against the test_categories.yaml TheRock main pins today (2e8f62c): hw-common skips with exit 0 and a warning, and an undefined tier still exits 1.

Device / architecture coverage

This is test infrastructure with no product-code change. The new job itself covers gfx942 on PRs, gfx950 and gfx90a on postsubmit, and gfx120X on nightly. I think PR CI plus the labelled bring-up run is enough; I don't see a need for a sweep.

Risk level

3 of 5. It's test-dispatch infra with real integration surface, and it adds about an hour of gfx942 runner time to TensileLite PRs.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/CONTRIBUTING.md.

Related: #5183, #7851, #3288

Add a tensilelite-common test job that runs TensileLite's Tensile/Tests/common suite on real hardware through pytest_runner.py and the hw-common category, matching Math CI's preliminary stage. - pytest_runner.py: TEST_CATEGORY overrides TEST_TYPE; register the tensilelite-common install path. - fetch_test_configurations.py: new job, opt-in to gfx90a, gfx94X-dcgpu, gfx950-dcgpu, gfx120X-all; test:tensilelite selects both jobs. - determine_rocm_test_dependencies.py: the tensilelite key fans out to both CI selectors. Co-authored-by: Cursor <cursoragent@cursor.com>

therock-pr-bot · 2026-09-23T19:33:50Z

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-23T19:34:08Z

🎉 All checks passed! This PR is ready for review.

Copilot

Copilot review overview

🔵 Needs a closer look

Address the reproduction environment propagation and add full-tier regression coverage.

Review effort: Lite
Findings: None

What changed in this PR

Adds a dedicated TheRock CI job for TensileLite hardware GEMM tests with architecture-specific selection.

Changes:

Adds tensilelite-common job and category override support.
Extends TensileLite dependency and label selection.
Adds coverage for runner, job, and selector behavior.

File	Summary
`test_tools/tests/determine_rocm_test_dependencies_test.py`	Tests TensileLite selector fan-out.
`test_tools/determine_rocm_test_dependencies.py`	Maps TensileLite dependencies to both CI jobs.
`build_tools/github_actions/tests/pytest_runner_test.py`	Tests category override and component resolution.
`build_tools/github_actions/tests/fetch_test_configurations_test.py`	Tests job selection and configuration; full-tier coverage is missing.
`build_tools/github_actions/test_executable_scripts/pytest_runner.py`	Supports category overrides and shared install mapping.
`build_tools/github_actions/fetch_test_configurations.py`	Defines the hardware GEMM job; reproduction environment propagation is incomplete.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

endurthiabhilash

LGTM. This is not affecting any other components, only adds tensilelite-common so, we should be good. Thanks for the PR!

TheRock's pinned rocm-libraries doesn't define the hw-common category yet, and submodule bumps land separately, so the new job would fail in TheRock CI until then. When TEST_CATEGORY pins a category the installed component doesn't define, pytest_runner.py now exits 0 with a ::warning:: annotation instead of failing. A missing tier from TEST_TYPE is still an error. Co-authored-by: Cursor <cursoragent@cursor.com>

geomin12

lgtm but let's make sure this passes and matches the test_filter

tony-davis · 2026-09-24T04:16:49Z

lgtm but let's make sure this passes and matches the test_filter

@geomin12 Yup. I want to see this full CI finish here first, and I also have https://github.com/ROCm/rocm-libraries/actions/runs/35954780246 running to see how it will work in rocm-libraries once the corresponding PR merges there. So it should be safe with and without that libraries-side PR, and we'll see that in CI here prior to merging.

… tests (#12492) ISSUE ID : #12491 ## Motivation `Tensile/Tests/common` (real codegen, build, and execution against a CPU reference) only runs in Math CI's `preliminary` job today. It doesn't run anywhere in TheRock Multi-Arch CI or nightly. TESTING.md already lists this as a known gap (Improvement Roadmap item 5). This PR adds the category TheRock needs to run it. The TheRock side is a companion PR, linked below. ## Technical Details - Adds a `hw-common` category to `projects/hipblaslt/tensilelite/test_categories.yaml`. It runs `Tensile/Tests/common` with `-m common` and the prebuilt `tensilelite-client` from the artifact. - Arch selection is opt-out, the same way `preliminary` does it. `configMarks()` tags every config `common` and turns each config's `skip-gfxNNNN` marks into skips for whatever arch `rocm_agent_enumerator` reports. So this category deliberately doesn't use `{AMDGPU_TARGETS}` (that would match almost nothing on real hardware), and it doesn't pass `--gpu-targets`. Hardware detection also sets `PyTestBuildArchNames`, which is what lets a gfx120X runner build for whichever of gfx1200/gfx1201 it actually has. - `parallel_workers: 4`, which is what Math CI runs. I didn't copy the 16 from the FFM categories. `assign_gpu_to_worker()` maps workers with `worker_num % num_gpus` and hands out per-worker locks, so on TheRock's 1-GPU runners every worker shares GPU 0. - Per-test timeout is 5400s, as a hang guard only. Math CI runs this suite with no per-test timeout, and the slowest config I found in recent `preliminary` logs (`streamk/sk_I8gemm_quick.yaml`) took 3173s. - Updates `tensilelite/TESTING.md`: the CI-gates section, the TheRock row in the lanes table (it still pointed at the retired `test_tensilelite.py`), two Known Risks rows, and roadmap item 5 (now the deferred sharding follow-up). - Worker-to-GPU assignment on TheRock's 1-GPU runners is fixed by #11898 (merged separately), which this PR now picks up from develop. The first TheRock run of `hw-common` hit exactly that bug: all 76 failures were on xdist workers gw1-gw3 with `no ROCm-capable device is detected`. With a worker-assignment fix in place, a later run passed 135 of 136. The one failure was `gemm/xfp32.yaml`, which #12569 fixes. Nothing selects `hw-common` until the TheRock PR lands, so this is inert on its own. ## Test Plan - Loaded the YAML and ran TheRock's `pytest_runner.build_marker_expression()` against `hw-common` for each target family, to confirm the `-m` expression it builds. - Sized the timeouts from four recent green `preliminary` builds (PR-12472 #2, PR-12469 #1, PR-12462 #1, PR-12461 #1). - Real collection and execution need a HIP runtime, so that happens on the TheRock PR's GPU run. ## Test Result - Marker expressions come out as `(common) and not skip-<arch>` for gfx90a, gfx94X, gfx950, and gfx120X. - Measured `-m common` wall clock in `preliminary` at 4 workers: gfx950 1:43 to 2:03 (182 tests), gfx942 49 to 64 min (136), gfx90a 16 to 24 min (43), gfx12 13 to 25 min (38). - Ran `hw-common` on gfx942 through TheRock Multi-Arch CI ([run 35915539444](https://github.com/ROCm/rocm-libraries/actions/runs/35915539444)): 136 tests selected, matching Math CI's `preliminary`. Every test on gw0 passed (46/46). All 76 failures were on gw1-gw3 with `no ROCm-capable device is detected`, which is what the conftest fix addresses. The suite ran in 35 minutes. - With #12569 (the msgpack LibraryClient fix) below it in the stack and #12526's TheRock pins on top, `tensilelite-common` passed all 136 tests on gfx942 in 46 minutes on the stack's testing PR, #12573 ([job](https://github.com/ROCm/rocm-libraries/actions/runs/36062518748/job/107875479911)). That includes `gemm/xfp32.yaml`, the one test that failed before #12569. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests. ## Risk level Low. It's a new category plus docs, and nothing selects it until the TheRock PR merges. Related: ROCm/TheRock#8459 (companion), ROCm/TheRock#5183, ROCm/TheRock#3288 Co-authored-by: Cursor <cursoragent@cursor.com>

github-project-automation Bot added this to TheRock Triage Sep 23, 2026

github-project-automation Bot moved this to TODO in TheRock Triage Sep 23, 2026

tony-davis mentioned this pull request Sep 23, 2026

ci(tensilelite): add hw-common category for real-hardware common GEMM tests ROCm/rocm-libraries#12492

Merged

1 task done

tony-davis requested a lite review from Copilot September 23, 2026 19:37

Copilot started reviewing on behalf of tony-davis September 23, 2026 19:38

View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

test(ci): cover the full tier in the tensilelite-common category test

e329dc8

Co-authored-by: Cursor <cursoragent@cursor.com>

tony-davis mentioned this pull request Sep 23, 2026

reproduce_test_failure.py doesn't export TEST_COMPONENT, so printed repro commands fail for test_runner.py / pytest_runner.py jobs #8462

Open

tony-davis requested review from archana-ramalingam, geomin12 and talumbau September 23, 2026 20:12

tony-davis marked this pull request as ready for review September 23, 2026 20:12

talumbau approved these changes Sep 23, 2026

View reviewed changes

geomin12 requested a review from endurthiabhilash September 23, 2026 21:45

archana-ramalingam approved these changes Sep 23, 2026

View reviewed changes

endurthiabhilash approved these changes Sep 23, 2026

View reviewed changes

geomin12 approved these changes Sep 24, 2026

View reviewed changes

tony-davis merged commit 703336a into main Sep 24, 2026
148 of 150 checks passed

tony-davis deleted the users/tony-davis/tensilelite-common-ci branch September 24, 2026 13:52

github-project-automation Bot moved this from TODO to Done in TheRock Triage Sep 24, 2026

Navigation Menu

Repository navigation

feat(ci): run TensileLite common GEMM tests in a tensilelite-common job - #8459

feat(ci): run TensileLite common GEMM tests in a tensilelite-common job#8459
tony-davis merged 3 commits into
mainfrom
users/tony-davis/tensilelite-common-ci

tony-davis commented Sep 23, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

endurthiabhilash left a comment

Uh oh!

geomin12 left a comment

Uh oh!

tony-davis commented Sep 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

tony-davis commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Device / architecture coverage

Risk level

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Checks Passed — Ready for Review

Uh oh!

therock-pr-bot Bot commented Sep 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

endurthiabhilash left a comment

Choose a reason for hiding this comment

Uh oh!

geomin12 left a comment

Choose a reason for hiding this comment

Uh oh!

tony-davis commented Sep 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tony-davis commented Sep 23, 2026 •

edited

Loading

therock-pr-bot Bot commented Sep 23, 2026 •

edited

Loading