(feat): [AILIKFD-40] add kfdtest to TheRock CI by andmar-amd · Pull Request #6334 · ROCm/TheRock

andmar-amd · 2026-07-03T18:12:09Z

Re-launching kfdtest integration into TheRock CI.

Add kfdtest subproject with correct dependencies (statically links libhsakmt and LLVM)
Add --kfdtest flag to install_rocm_from_artifacts.py
Add kfdtest to BUILD_TOPOLOGY.toml (artifact group, build stage, and artifact)
Add sysdeps to kfdtest artifact dependencies for bundled libraries
Use get_target_property to dynamically locate ROCR-Runtime binary directory
Set RPATH to lib/rocm_sysdeps/lib only (kfdtest statically links main deps)

Co-Authored-By: Claude Sonnet 4.5
Signed-off-by: Andrew Martin andrew.martin@amd.com

JIRA ID: AILIKFD-40

Co-Authored-By: Claude Sonnet 4.5

Motivation

To have a copy of the kfdtest binary as part of TheRock release archive.
This is useful for both QA and various customers who use it to test and verify KFD.

Technical Details

Original PR: Adding kfdtest to The Rock CI. #5377 (merged then reverted in Revert "Adding kfdtest to The Rock CI." #6142)
Revert reason: ASAN build failure - linker couldn't find -lhsakmt and -lnuma
fix(kfdtest): [AILIKFD-40] fix overwirte of CMAKE_EXE_LINKER_FLAGS rocm-systems#8042
refactor(kfdtest): [AILIKFD-40] modernize CMake dependency discovery rocm-systems#8221
fix(kfdtest): handle NUMA CONFIG mode target correctly rocm-systems#9480

Test Plan

Test Result

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

therock-pr-bot · 2026-07-03T18:12:29Z

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-07-03T18:12:31Z

🎉 All checks passed! This PR is ready for review.

## Summary TheRock side of test-filter standardization for **rocSHMEM**, paired with [ROCm/rocm-systems#6334](ROCm/rocm-systems#6334). rocm-systems#6334 adds native CTest integration for rocSHMEM: `projects/rocshmem/tests/test_categories.yaml` plus `apply_ctest_category_labels()` wiring that installs relocatable `bin/rocshmem/**/CTestTestfile.cmake` files (a root dispatcher that `subdirs()` into `tests/functional` and `tests/unit`) carrying the `quick`/`standard`/`comprehensive`/`full` tier labels. This PR wires TheRock so the generic `test_runner.py` can drive those tests from the installed artifact: - **`comm-libs/CMakeLists.txt`** — pass `-DBUILD_CTESTS=${THEROCK_BUILD_TESTING}` to the rocshmem subproject. The CTest registration and the install-tree `CTestTestfile.cmake` generation in #6334 are gated behind `BUILD_CTESTS` (default `OFF`), so without this the install tree would contain no `bin/rocshmem/**` and `test_runner.py` would fail with `Test directory does not exist`. - **`comm-libs/artifact-rocshmem.toml`** — populate the previously-empty `test` component: `bin/rocshmem/**` (the install-tree CTest files), `bin/rocshmem_unit_tests`, the `share/rocshmem/*` CTest wrapper scripts, and the `device_bitcode` HSACOs. The functional binary and `rocshmem_info` already live in the `run` component (fetched via `rocshmem_run`). - **`build_tools/install_rocm_from_artifacts.py`** — add a `--rocshmem` artifact selector that pulls `rocshmem_lib`/`_run`/`_test` plus OpenMPI (`mpirun` + libs) needed to launch the tests. - **`build_tools/github_actions/fetch_test_configurations.py`** — add a `rocshmem` test job using `test_runner.py` (`fetch_artifact_args = "--rocshmem --tests"`), on Linux multi-GPU runners (the functional/unit tests launch via `mpirun` with `RANKS` 2..64, same setup as `rccl`). - **`fetch_test_configurations_test.py`** — relax the assertion that assumed exactly one multi-GPU job repo-wide, now that rocshmem is a second multi-GPU job alongside rccl. `test_runner.py` needs no `COMPONENT_DIR_MAPPING` entry (job name `rocshmem` == install dir `bin/rocshmem`). ## Test plan - [x] `python3 -m pytest build_tools/github_actions/tests/fetch_test_configurations_test.py` (22 passed) - [x] `black --check` on the edited Python files - [x] `--rocshmem` registered in `install_rocm_from_artifacts.py --help` - [x] `artifact-rocshmem.toml` parses; test `include` resolves - [x] CI: build with `THEROCK_BUILD_TESTING=ON`, confirm `bin/rocshmem/CTestTestfile.cmake` + `tests/{functional,unit}/CTestTestfile.cmake` are installed and bundled in the test artifact - [x] CI: `TEST_COMPONENT=rocshmem TEST_TYPE=quick python test_runner.py` discovers labels and runs `ctest -L quick` from `THEROCK_BIN_DIR/rocshmem` ## Test Results Test Run: https://github.com/ROCm/TheRock/actions/runs/30112055659/job/89560360151?pr=5691 Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>

ScottTodd · 2026-07-29T14:41:44Z

Requested review from @nunnikri who originally reviewed #5377.

A few points:

The PR template is not populated. Please either fill in "Motivation", "Technical Details", "Test Plan", "Test Result" or delete them from the PR description. Write the description as you would a good commit message, as it will be used for that when merged.
This has unit test failures that must be fixed: https://github.com/ROCm/TheRock/actions/runs/30304359105/job/90104834154?pr=6334
This has build failures that must be fixed: https://github.com/ROCm/TheRock/actions/runs/30304359735/job/90105058884?pr=6334#step:13:196
If this failed ASan originally, you should test ASan with it now. You can apply the ci:asan label to the PR to get some ASan build coverage (though this CI feature is new and might not be fully representative)

andmar-amd · 2026-08-05T13:32:14Z

@nunnikri . kfdtest is not running in the CI, it was disabled on advised by Saad and Luis. The big problem with it is that it wants exclusive control of the GPU and tend to crash when it has to share. It is designed to be selfish in that way. So the idea is just to compile and then those who want to use it (QA, customers) can just run it.

I can merge here, then do an improvement PR for runtime (b/c this issue only affect stand alone build) with the above changes?

nunnikri · 2026-08-05T14:23:41Z

Though I waited so long for this merge, I hesitate merge it

b/c I was double checking ROCM-28573, wrt to building kfdtest locally. And I found this (below). It does not seem to be an issue on the rock, but is it valuable to have this fix b4 merge?

... Found the issue! Lines 255-262 show that HSAKMT_FIND_NUMA_DEPENDENCY is only set to ON when:

BUILD_SHARED_LIBS is ON

AND NUMA was found via find_package(NUMA)

AND at least one NUMA library is a CMake target

The /opt/rocm installation likely had BUILD_SHARED_LIBS=OFF, so the dependency wasn't propagated to the config file. The proper fix is to set HSAKMT_FIND_NUMA_DEPENDENCY=ON unconditionally when NUMA is found as a target, regardless of BUILD_SHARED_LIBS.

The workaround script is fine for now, but the proper upstream fix would be:

if(NUMA_FOUND) foreach(_numa_library IN LISTS NUMA_LIBRARIES) if(TARGET "${_numa_library}") set(HSAKMT_FIND_NUMA_DEPENDENCY ON) break() endif() endforeach() endif()

(Remove the if(BUILD_SHARED_LIBS) condition at line 255) ...

TheRock is giving a platform to build, test and distribute ROCm components. But we cant decide on the features each components wants. It should be discussed with BU and finalize.
cc : @saadrahim

nunnikri · 2026-08-05T14:27:50Z

@nunnikri . kfdtest is not running in the CI, it was disabled on advised by Saad and Luis. The big problem with it is that it wants exclusive control of the GPU and tend to crash when it has to share. It is designed to be selfish in that way. So the idea is just to compile and then those who want to use it (QA, customers) can just run it.

I can merge here, then do an improvement PR for runtime (b/c this issue only affect stand alone build) with the above changes?

Running kfdtest in TheRock ci we didnt want at this point, may be later can add. The idea was for users/QA to have an option to run these test.
I can see the kfdfiles are getting added to packages.

https://therock-ci-artifacts.s3.amazonaws.com/30953000930-linux/packages/deb/pool/main/index.html


 dpkg -c amdrocm-runtime-test10.1_10.1.0~dev20260804-30953000930_amd64.deb | grep kfd
-rwxr-xr-x root/root   2692448 2026-08-04 18:12 ./opt/rocm/core-10.1/bin/kfdtest
-rwxr-xr-x root/root     14131 2026-08-04 16:50 ./opt/rocm/core-10.1/bin/run_kfdtest.sh
drwxr-xr-x root/root         0 2026-08-04 16:50 ./opt/rocm/core-10.1/share/kfdtest/
-rw-r--r-- root/root     10483 2026-08-04 16:50 ./opt/rocm/core-10.1/share/kfdtest/kfdtest.exclude

But one thing we should add before merging the PR. Every component should have a LIcense file under which the component is distributed. Usually MIT for ROCm components. Please get that added before merging. Otherwise I am good.

andmar-amd · 2026-08-05T15:25:43Z

Thanks, License added, hoping that it will be green.

FYI ROCm/rocm-systems#9716 will take care of that other issue.

andmar-amd · 2026-08-05T19:41:54Z

@nunnikri the last build is now finish, do check if the license file you need is now present (how can I check this on my own)?

therock-pr-bot · 2026-08-06T17:11:41Z

Pre-commit check failed

⛔ pre-commit failed

Please run locally:

python -m pip install pre-commit
pre-commit install
pre-commit run --all-files --show-diff-on-failure

This repo uses .pre-commit-config.yaml.

nunnikri

Looks like again the PR got messed up. All the submodules got bumped.
Need to revert all the changes in compiler, rocgdb, math-libs, third-party, rocm-libraries, rocm-systems

andmar-amd · 2026-08-08T03:24:18Z

@nunnikri

Thanks for that, I did realize that the last rebase had those sub-module changes that was pulled into mine. I have, stashed, hard reset, pop, cherry-pick, and fixed things.

- Add kfdtest subproject to core/CMakeLists.txt with proper build dependencies - Install LICENSE.kfdtest to share/doc/kfdtest/ in stage directory - Create kfdtest_test_generic.tar.zst artifact - Add kfdtest to amdrocm-runtime-test package (DEB and RPM) - kfdtest requires amdrocm-sysdeps and amdrocm-llvm runtime dependencies Resolves: AILIKFD-40 Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

andmar-amd · 2026-08-09T18:17:41Z

@nunnikri

The license file is in the pkg now, please have a look and see if you like things.

nunnikri

Now the package contents looks good.

/opt/rocm/core-10.1/bin/kfdtest
/opt/rocm/core-10.1/bin/run_kfdtest.sh
/opt/rocm/core-10.1/share/doc/kfdtest
/opt/rocm/core-10.1/share/doc/kfdtest/LICENSE.kfdtest
/opt/rocm/core-10.1/share/kfdtest
/opt/rocm/core-10.1/share/kfdtest/kfdtest.exclude

ScottTodd · 2026-08-12T15:02:45Z

This was merged with test failures: https://github.com/ROCm/TheRock/actions/runs/31300075067/job/93211327008

See #7298

## Motivation kfdtest was added as a subproject in #6334, but the committed consumer graph was never regenerated. The drift check compares the whole file, so any PR touching a graph-affecting path either fails it or has to carry these lines in an unrelated diff. GitHub issue: #7298 ## Technical Details Regenerated `test_tools/therock_consumer_graph.json` with `check_consumer_graph_drift.py --write`. No other files change. The diff is the kfdtest node plus its seven reverse edges from `core/CMakeLists.txt`: `amd-llvm`, `rocr-runtime`, `therock-libdrm`, `therock-numactl`, `therock-yaml-cpp`, `therock-zlib`, `therock-zstd`. No behavior change. Selection can now emit `kfdtest`, but there is no `kfdtest` key in `fetch_test_configurations.py`, so no job is scheduled. ## Test Plan Drift check, policy validation, selection unit tests, pre-commit. ## Test Result - `--validate-policies`: OK, 18 component keys against 116 graph nodes - `determine_rocm_test_dependencies_test.py`: 49 passed - pre-commit: passed ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/CONTRIBUTING.md.

github-project-automation Bot added this to TheRock Triage Jul 3, 2026

github-project-automation Bot moved this to TODO in TheRock Triage Jul 3, 2026

therock-pr-bot Bot added the Not ready to Review PR has unresolved policy failures — reviews blocked label Jul 3, 2026