Skip to content

feat(rocshmem): add TheRock CI and Test Filter Standardization - #5691

Merged
dileepr1 merged 21 commits into
mainfrom
users/dravindr/tr_rocshmem
Jul 28, 2026
Merged

dileepr1 merged 21 commits into
mainfrom
users/dravindr/tr_rocshmem

Conversation

@dileepr1

@dileepr1 dileepr1 commented Jun 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

TheRock side of test-filter standardization for rocSHMEM, paired with ROCm/rocm-systems#6334.

rocm-systems#6334 adds native CTest integration for rocSHMEM: projects/rocshmem/tests/test_categories.yaml plus apply_ctest_category_labels() wiring that installs relocatable bin/rocshmem/**/CTestTestfile.cmake files (a root dispatcher that subdirs() into tests/functional and tests/unit) carrying the quick/standard/comprehensive/full tier labels. This PR wires TheRock so the generic test_runner.py can drive those tests from the installed artifact:

  • comm-libs/CMakeLists.txt — pass -DBUILD_CTESTS=${THEROCK_BUILD_TESTING} to the rocshmem subproject. The CTest registration and the install-tree CTestTestfile.cmake generation in (feat): [AILIKFD-40] add kfdtest to TheRock CI #6334 are gated behind BUILD_CTESTS (default OFF), so without this the install tree would contain no bin/rocshmem/** and test_runner.py would fail with Test directory does not exist.
  • comm-libs/artifact-rocshmem.toml — populate the previously-empty test component: bin/rocshmem/** (the install-tree CTest files), bin/rocshmem_unit_tests, the share/rocshmem/* CTest wrapper scripts, and the device_bitcode HSACOs. The functional binary and rocshmem_info already live in the run component (fetched via rocshmem_run).
  • build_tools/install_rocm_from_artifacts.py — add a --rocshmem artifact selector that pulls rocshmem_lib/_run/_test plus OpenMPI (mpirun + libs) needed to launch the tests.
  • build_tools/github_actions/fetch_test_configurations.py — add a rocshmem test job using test_runner.py (fetch_artifact_args = "--rocshmem --tests"), on Linux multi-GPU runners (the functional/unit tests launch via mpirun with RANKS 2..64, same setup as rccl).
  • fetch_test_configurations_test.py — relax the assertion that assumed exactly one multi-GPU job repo-wide, now that rocshmem is a second multi-GPU job alongside rccl.

test_runner.py needs no COMPONENT_DIR_MAPPING entry (job name rocshmem == install dir bin/rocshmem).

Test plan

  • python3 -m pytest build_tools/github_actions/tests/fetch_test_configurations_test.py (22 passed)
  • black --check on the edited Python files
  • --rocshmem registered in install_rocm_from_artifacts.py --help
  • artifact-rocshmem.toml parses; test include resolves
  • CI: build with THEROCK_BUILD_TESTING=ON, confirm bin/rocshmem/CTestTestfile.cmake + tests/{functional,unit}/CTestTestfile.cmake are installed and bundled in the test artifact
  • CI: TEST_COMPONENT=rocshmem TEST_TYPE=quick python test_runner.py discovers labels and runs ctest -L quick from THEROCK_BIN_DIR/rocshmem

Test Results

Test Run: https://github.com/ROCm/TheRock/actions/runs/30112055659/job/89560360151?pr=5691

Made with Cursor

Paired with rocm-systems#6334, which adds native CTest integration for
rocSHMEM (tests/test_categories.yaml + apply_ctest_category_labels) and
installs relocatable bin/rocshmem/**/CTestTestfile.cmake files carrying the
quick/standard/comprehensive/full tier labels.

TheRock side:
- comm-libs/CMakeLists.txt: pass -DBUILD_CTESTS=${THEROCK_BUILD_TESTING} to the
  rocshmem subproject so the CTest registration runs and the install-tree
  CTestTestfile.cmake files are produced (the registration is gated behind
  BUILD_CTESTS, which defaults OFF).
- comm-libs/artifact-rocshmem.toml: include the install-tree CTest files
  (bin/rocshmem/**), the unit test binary, the CTest wrapper scripts and the
  device_bitcode HSACOs in the test component so the artifact bundles
  everything ctest needs (the functional binary and rocshmem_info come from
  the run component).
- install_rocm_from_artifacts.py: add a --rocshmem artifact selector that
  fetches rocshmem_lib/_run/_test plus OpenMPI (mpirun + libs) for the tests.
- fetch_test_configurations.py: add a rocshmem test job driven by the generic
  test_runner.py (ctest -L <quick|standard|comprehensive|full> from
  THEROCK_BIN_DIR/rocshmem) on linux multi-GPU runners.
- fetch_test_configurations_test.py: relax the single-multi-GPU-job assertion
  now that rocshmem is a second multi-GPU job alongside rccl.

WIP: depends on rocm-systems#6334. Do not merge until #6334 lands and TheRock's
rocm-systems source includes it; paired validation runs from #6334 CI by
pointing its TheRock ref at users/dravindr/tr_rocshmem.

Co-authored-by: Cursor <cursoragent@cursor.com>
Point the rocm-systems submodule to the latest develop tip
(068718c) which now contains the merged rocshmem CTest test-filter
standardization work (rocm-systems#6334, merge c541756). This lets
TheRock build rocshmem with the native CTest infrastructure and
exercise the tsf test job end-to-end.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dileepr1
dileepr1 requested a review from a team as a code owner June 8, 2026 19:19
@dileepr1
dileepr1 marked this pull request as draft June 8, 2026 19:19
dileepr1 and others added 2 commits June 11, 2026 14:46
Point the rocm-systems submodule at develop tip 87fa69c, which now
includes rocshmem#6804 (Simple Local Runtime). #6804 removes the
configure-time MPIEXEC early-return in CTestFunctionalTests.cmake that
was force-disabling BUILD_CTESTS and preventing installation of
bin/rocshmem/CTestTestfile.cmake. With SLR as the MPI-less fallback,
the install-tree CTest file is generated and test_runner.py can find
build/bin/rocshmem.

Co-authored-by: Cursor <cursoragent@cursor.com>
Bump the rocm-systems submodule to rocshmem/therock-ci-fix (51ec214),
which fixes the install-tree CTest gating so ctest can discover tests
on a no-MPI test host. The install-root CTestTestfile template now
gates the mpiexec early-return behind USE_SLR_LAUNCHER, so in SLR mode
it falls through to the subdirs() calls and bin/rocshmem tests are
listed (previously "No tests found in build/bin/rocshmem").

Co-authored-by: Cursor <cursoragent@cursor.com>

@geomin12 geomin12 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests fail + merge conflict

but it looks good

…shmem

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	build_tools/github_actions/tests/fetch_test_configurations_test.py
#	rocm-systems
@therock-pr-bot

therock-pr-bot Bot commented Jun 23, 2026 •

Copy link
Copy Markdown

✅ All Policy Checks Passed

Check Status Details
🌿 Branch Name ✅ Pass —
📝 PR Title/Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot therock-pr-bot Bot added the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 23, 2026
@therock-pr-bot

therock-pr-bot Bot commented Jun 23, 2026 •

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

dileepr1 and others added 2 commits June 30, 2026 13:43
The rocshmem functional and unit test executables install to
share/rocshmem/ (CMAKE_INSTALL_DATADIR), but artifact-rocshmem.toml
referenced them as bin/rocshmem_functional_tests / bin/rocshmem_unit_tests,
so neither was packaged into any artifact. At test time the install-tree
CTestTestfile.cmake invokes them via ../../../../share/rocshmem/, which
was absent, causing 74/77 functional tests to fail with
"rocshmem_functional_tests: No such file or directory".

Include the binaries at their real share/rocshmem/ paths in the test
component and drop the dead bin/rocshmem_functional_tests glob from the
run component.

Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	rocm-systems
@therock-pr-bot therock-pr-bot Bot removed the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 30, 2026
dileepr1 and others added 7 commits July 7, 2026 09:12
…shmem

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	rocm-systems
The rocshmem/therock-ci-fix work was squash-merged into rocm-systems
develop, which now contains the install-tree SLR fixes (USE_SLR_LAUNCHER
guard in CTestInstallRoot.cmake.in and the ${CMAKE_COMMAND}->env
translation for relocatable install-time test commands).

Pointing at develop (e5f2ddb) instead of the feature branch also brings
the compiler-runtime components (amd-comgr/mirage/rocjitsu) up to date,
fixing the Linux build failures caused by the stale therock-ci-fix pin
that were skipping all downstream Linux stages and the rocshmem test
job. It also makes the submodule pin merge-ready (a develop SHA rather
than an unmerged branch).

Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	rocm-systems
The --rocshmem artifact flag is read via args.rocshmem in
retrieve_artifacts_by_run_id's any([...]) check, but the test helper
_make_run_id_args built its args Namespace without a rocshmem key,
causing test_debug_tools_includes_amd_llvm_dev to fail with
AttributeError: 'Namespace' object has no attribute 'rocshmem'.

Co-authored-by: Cursor <cursoragent@cursor.com>
The functional and unit test executables now install to bin/
(CMAKE_INSTALL_BINDIR) in rocm-systems so their RPATH ($ORIGIN/../lib)
resolves the ROCm runtime libs (e.g. libhiprtc.so.7). Update the test
artifact manifest to package them from bin/ instead of share/rocshmem/,
matching the new install-tree relative path (../../../rocshmem_*_tests).

Co-authored-by: Cursor <cursoragent@cursor.com>
Bump rocm-systems pin to users/egabriel/therock-ci-fix-for-real-this-time
(1f5830378), which fixes install-time ctest path resolution by baking in
absolute install paths (share/rocshmem/...) instead of moving the test
binaries to bin/. Revert the artifact manifest binary paths back to
share/rocshmem/ to match that branch's install layout.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dileepr1 dileepr1 changed the title Add test filter standardization for rocshmem feat(rocshmem): add test filter standardization Jul 15, 2026
dileepr1 and others added 6 commits July 15, 2026 12:28
…shmem

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	rocm-systems
… from bin/

Bump rocm-systems pin to users/egabriel/therock-ci-fix-for-real-this-time
(3f2eb96e5), which installs the functional/unit test executables to bin/
(CMAKE_INSTALL_BINDIR) so their RPATH ($ORIGIN/../lib) resolves the ROCm
runtime libs. Update the artifact manifest to package them from bin/.

Note: that branch still emits the executable's install-time ctest path as
../../../../share/rocshmem/rocshmem_*_tests while the binaries now install
to bin/, so the tests may not locate the executable until the generator's
relative path is corrected to ../../../rocshmem_*_tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
Bump rocm-systems pin to users/egabriel/therock-ci-fix-for-real-this-time
(83918cb6b), which corrects the install-time ctest executable path to
../../../rocshmem_*_tests to match the bin/ install location. Combined with
the bin/ RPATH ($ORIGIN/../lib), the functional/unit test binaries should
now be located and able to load the ROCm runtime libs (e.g. libhiprtc.so.7).

Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	rocm-systems
The rocshmem test wrappers run rocshmem_info to auto-detect the backend.
rocshmem_info is installed via install(PROGRAMS ...) rather than as a
target, so it keeps its build-tree RPATH and fails to load libamdhip64.so.7
in the relocated test artifact (non-fatal, but noisy: 85 warnings/run).
Add a rocshmem COMPONENT_OVERRIDES entry prepending <ROCM_PATH>/lib to
LD_LIBRARY_PATH so the probe resolves the ROCm runtime libs.

Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	rocm-systems
The rocshmem_info backend probe (installed via install(PROGRAMS ...), so it
keeps a non-relocatable build-tree RPATH) needs more than the ROCm runtime
libs: it also links librocm_sysdeps_numa.so.1, which lives under
lib/rocm_sysdeps/lib in the relocated artifact. Add that dir to the rocshmem
LD_LIBRARY_PATH override alongside lib/ so the probe resolves all its libs.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dileepr1
dileepr1 marked this pull request as ready for review July 27, 2026 13:46
@dileepr1
dileepr1 requested review from geomin12 and madkasul July 28, 2026 13:29

@geomin12 geomin12 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dileepr1
dileepr1 merged commit a1f5915 into main Jul 28, 2026
139 of 156 checks passed
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Jul 28, 2026
@dileepr1
dileepr1 deleted the users/dravindr/tr_rocshmem branch July 28, 2026 15:47
@dileepr1 dileepr1 changed the title feat(rocshmem): add test filter standardization feat(rocshmem): add TheRock CI and Test Filter Standardization Jul 28, 2026
Comment on lines +499 to +502
# rocSHMEM tests launch via mpirun and link against TheRock's
# vendored OpenMPI, so pull it (with its run component for mpiexec).
extra_artifacts.append("openmpi")
argv.append("openmpi_run")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't build openmpi by default and we don't distribute a vendored version.

You may be interested in how rocprofiler-sdk has recently started running mpi tests by using a test dockerfile with a system install:

Comment on lines +499 to +501
# rocSHMEM functional/unit tests launch via mpirun with RANKS 2..64, so
# they need a multi-GPU runner (same setup as rccl).
"multi_gpu": {"linux": ["gfx94X-dcgpu", "gfx950-dcgpu"]},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi @amd-shiraz new 8 GPU runner like linux-gfx942-8gpu-ossci-rocm-8txs9-runner-wknrq in use here, e.g. https://github.com/ROCm/TheRock/actions/runs/30112055659/job/89560360151

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will cause huge queues with current capacity for gfx942-8gpu. do these actually need multi gpu? cc @suchitravemuri1

"job_name": "rocshmem",
"fetch_artifact_args": "--rocshmem --tests",
"timeout_minutes": 30,
"test_script": f"python {_get_script_path('test_runner.py')}",

@ScottTodd ScottTodd Jul 28, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What tests do you specifically expect to run with this script? Looking at the logs on https://github.com/ROCm/TheRock/actions/runs/30112055659/job/89560360151, I see

28/28 Test #191: pingpong_n2_w32_z1_uuid ..................................   Passed    1.53 sec

100% tests passed, 0 tests failed out of 28

Label Time Summary:
AMO                       =  21.74 sec*proc (4 tests)
OTHER                     =  47.31 sec*proc (7 tests)
RMA                       = 128.30 sec*proc (17 tests)
backend_all               = 129.94 sec*proc (17 tests)
backend_gda               =  55.17 sec*proc (7 tests)
backend_ipc               =  67.41 sec*proc (11 tests)
comprehensive             = 197.34 sec*proc (28 tests)
full                      = 197.34 sec*proc (28 tests)
functional                = 197.34 sec*proc (28 tests)
gpu_all                   = 197.34 sec*proc (28 tests)
launcher_slr              = 197.34 sec*proc (28 tests)
nightly                   =  96.18 sec*proc (12 tests)
pr                        =   7.85 sec*proc (2 tests)
pre-commit                = 197.34 sec*proc (28 tests)
quick                     = 197.34 sec*proc (28 tests)
smoke                     = 197.34 sec*proc (28 tests)
standard                  = 197.34 sec*proc (28 tests)
variant_default_stream    =  15.98 sec*proc (2 tests)
variant_uuid              = 189.14 sec*proc (27 tests)

Total Test time (real) =  88.85 sec
# TEST_COMPONENT: rocshmem -> Test Directory: build/bin/rocshmem
# TEST_TYPE: quick -> Category: quick
# AMDGPU_FAMILIES: gfx94X-dcgpu -> GPU Architecture: gfx94X

# Discovering available test labels...
# Warning: No GPU specific test suites available

# No GPU suite found for gfx94X, excluding all ex_gpu tests
# Running: ctest -L ^quick$ -LE ex_gpu --output-on-failure --parallel 1 --timeout 7200 --test-dir build/bin/rocshmem -V --tests-information 1,,1

That seems pretty light on tests, even for the "quick" filter? The "No GPU suite found" logs are also concerning given that this claims it should be running multi-GPU tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants