Repository navigation
feat(rocshmem): add TheRock CI and Test Filter Standardization - #5691
Conversation
Paired with rocm-systems#6334, which adds native CTest integration for
rocSHMEM (tests/test_categories.yaml + apply_ctest_category_labels) and
installs relocatable bin/rocshmem/**/CTestTestfile.cmake files carrying the
quick/standard/comprehensive/full tier labels.
TheRock side:
- comm-libs/CMakeLists.txt: pass -DBUILD_CTESTS=${THEROCK_BUILD_TESTING} to the
rocshmem subproject so the CTest registration runs and the install-tree
CTestTestfile.cmake files are produced (the registration is gated behind
BUILD_CTESTS, which defaults OFF).
- comm-libs/artifact-rocshmem.toml: include the install-tree CTest files
(bin/rocshmem/**), the unit test binary, the CTest wrapper scripts and the
device_bitcode HSACOs in the test component so the artifact bundles
everything ctest needs (the functional binary and rocshmem_info come from
the run component).
- install_rocm_from_artifacts.py: add a --rocshmem artifact selector that
fetches rocshmem_lib/_run/_test plus OpenMPI (mpirun + libs) for the tests.
- fetch_test_configurations.py: add a rocshmem test job driven by the generic
test_runner.py (ctest -L <quick|standard|comprehensive|full> from
THEROCK_BIN_DIR/rocshmem) on linux multi-GPU runners.
- fetch_test_configurations_test.py: relax the single-multi-GPU-job assertion
now that rocshmem is a second multi-GPU job alongside rccl.
WIP: depends on rocm-systems#6334. Do not merge until #6334 lands and TheRock's
rocm-systems source includes it; paired validation runs from #6334 CI by
pointing its TheRock ref at users/dravindr/tr_rocshmem.
Co-authored-by: Cursor <cursoragent@cursor.com>
Point the rocm-systems submodule to the latest develop tip (068718c) which now contains the merged rocshmem CTest test-filter standardization work (rocm-systems#6334, merge c541756). This lets TheRock build rocshmem with the native CTest infrastructure and exercise the tsf test job end-to-end. Co-authored-by: Cursor <cursoragent@cursor.com>
Point the rocm-systems submodule at develop tip 87fa69c, which now includes rocshmem#6804 (Simple Local Runtime). #6804 removes the configure-time MPIEXEC early-return in CTestFunctionalTests.cmake that was force-disabling BUILD_CTESTS and preventing installation of bin/rocshmem/CTestTestfile.cmake. With SLR as the MPI-less fallback, the install-tree CTest file is generated and test_runner.py can find build/bin/rocshmem. Co-authored-by: Cursor <cursoragent@cursor.com>
Bump the rocm-systems submodule to rocshmem/therock-ci-fix (51ec214), which fixes the install-tree CTest gating so ctest can discover tests on a no-MPI test host. The install-root CTestTestfile template now gates the mpiexec early-return behind USE_SLR_LAUNCHER, so in SLR mode it falls through to the subdirs() calls and bin/rocshmem tests are listed (previously "No tests found in build/bin/rocshmem"). Co-authored-by: Cursor <cursoragent@cursor.com>
geomin12
left a comment
There was a problem hiding this comment.
tests fail + merge conflict
but it looks good
…shmem Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # build_tools/github_actions/tests/fetch_test_configurations_test.py # rocm-systems
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
The rocshmem functional and unit test executables install to share/rocshmem/ (CMAKE_INSTALL_DATADIR), but artifact-rocshmem.toml referenced them as bin/rocshmem_functional_tests / bin/rocshmem_unit_tests, so neither was packaged into any artifact. At test time the install-tree CTestTestfile.cmake invokes them via ../../../../share/rocshmem/, which was absent, causing 74/77 functional tests to fail with "rocshmem_functional_tests: No such file or directory". Include the binaries at their real share/rocshmem/ paths in the test component and drop the dead bin/rocshmem_functional_tests glob from the run component. Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # rocm-systems
…shmem Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # rocm-systems
The rocshmem/therock-ci-fix work was squash-merged into rocm-systems
develop, which now contains the install-tree SLR fixes (USE_SLR_LAUNCHER
guard in CTestInstallRoot.cmake.in and the ${CMAKE_COMMAND}->env
translation for relocatable install-time test commands).
Pointing at develop (e5f2ddb) instead of the feature branch also brings
the compiler-runtime components (amd-comgr/mirage/rocjitsu) up to date,
fixing the Linux build failures caused by the stale therock-ci-fix pin
that were skipping all downstream Linux stages and the rocshmem test
job. It also makes the submodule pin merge-ready (a develop SHA rather
than an unmerged branch).
Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # rocm-systems
The --rocshmem artifact flag is read via args.rocshmem in retrieve_artifacts_by_run_id's any([...]) check, but the test helper _make_run_id_args built its args Namespace without a rocshmem key, causing test_debug_tools_includes_amd_llvm_dev to fail with AttributeError: 'Namespace' object has no attribute 'rocshmem'. Co-authored-by: Cursor <cursoragent@cursor.com>
The functional and unit test executables now install to bin/ (CMAKE_INSTALL_BINDIR) in rocm-systems so their RPATH ($ORIGIN/../lib) resolves the ROCm runtime libs (e.g. libhiprtc.so.7). Update the test artifact manifest to package them from bin/ instead of share/rocshmem/, matching the new install-tree relative path (../../../rocshmem_*_tests). Co-authored-by: Cursor <cursoragent@cursor.com>
Bump rocm-systems pin to users/egabriel/therock-ci-fix-for-real-this-time (1f5830378), which fixes install-time ctest path resolution by baking in absolute install paths (share/rocshmem/...) instead of moving the test binaries to bin/. Revert the artifact manifest binary paths back to share/rocshmem/ to match that branch's install layout. Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # rocm-systems
… from bin/ Bump rocm-systems pin to users/egabriel/therock-ci-fix-for-real-this-time (3f2eb96e5), which installs the functional/unit test executables to bin/ (CMAKE_INSTALL_BINDIR) so their RPATH ($ORIGIN/../lib) resolves the ROCm runtime libs. Update the artifact manifest to package them from bin/. Note: that branch still emits the executable's install-time ctest path as ../../../../share/rocshmem/rocshmem_*_tests while the binaries now install to bin/, so the tests may not locate the executable until the generator's relative path is corrected to ../../../rocshmem_*_tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Bump rocm-systems pin to users/egabriel/therock-ci-fix-for-real-this-time (83918cb6b), which corrects the install-time ctest executable path to ../../../rocshmem_*_tests to match the bin/ install location. Combined with the bin/ RPATH ($ORIGIN/../lib), the functional/unit test binaries should now be located and able to load the ROCm runtime libs (e.g. libhiprtc.so.7). Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # rocm-systems
The rocshmem test wrappers run rocshmem_info to auto-detect the backend. rocshmem_info is installed via install(PROGRAMS ...) rather than as a target, so it keeps its build-tree RPATH and fails to load libamdhip64.so.7 in the relocated test artifact (non-fatal, but noisy: 85 warnings/run). Add a rocshmem COMPONENT_OVERRIDES entry prepending <ROCM_PATH>/lib to LD_LIBRARY_PATH so the probe resolves the ROCm runtime libs. Co-authored-by: Cursor <cursoragent@cursor.com>
…shmem Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # rocm-systems
The rocshmem_info backend probe (installed via install(PROGRAMS ...), so it keeps a non-relocatable build-tree RPATH) needs more than the ROCm runtime libs: it also links librocm_sysdeps_numa.so.1, which lives under lib/rocm_sysdeps/lib in the relocated artifact. Add that dir to the rocshmem LD_LIBRARY_PATH override alongside lib/ so the probe resolves all its libs. Co-authored-by: Cursor <cursoragent@cursor.com>
| # rocSHMEM tests launch via mpirun and link against TheRock's | ||
| # vendored OpenMPI, so pull it (with its run component for mpiexec). | ||
| extra_artifacts.append("openmpi") | ||
| argv.append("openmpi_run") |
There was a problem hiding this comment.
We don't build openmpi by default and we don't distribute a vendored version.
You may be interested in how rocprofiler-sdk has recently started running mpi tests by using a test dockerfile with a system install:
| # rocSHMEM functional/unit tests launch via mpirun with RANKS 2..64, so | ||
| # they need a multi-GPU runner (same setup as rccl). | ||
| "multi_gpu": {"linux": ["gfx94X-dcgpu", "gfx950-dcgpu"]}, |
There was a problem hiding this comment.
fyi @amd-shiraz new 8 GPU runner like linux-gfx942-8gpu-ossci-rocm-8txs9-runner-wknrq in use here, e.g. https://github.com/ROCm/TheRock/actions/runs/30112055659/job/89560360151
There was a problem hiding this comment.
this will cause huge queues with current capacity for gfx942-8gpu. do these actually need multi gpu? cc @suchitravemuri1
| "job_name": "rocshmem", | ||
| "fetch_artifact_args": "--rocshmem --tests", | ||
| "timeout_minutes": 30, | ||
| "test_script": f"python {_get_script_path('test_runner.py')}", |
There was a problem hiding this comment.
What tests do you specifically expect to run with this script? Looking at the logs on https://github.com/ROCm/TheRock/actions/runs/30112055659/job/89560360151, I see
28/28 Test #191: pingpong_n2_w32_z1_uuid .................................. Passed 1.53 sec
100% tests passed, 0 tests failed out of 28
Label Time Summary:
AMO = 21.74 sec*proc (4 tests)
OTHER = 47.31 sec*proc (7 tests)
RMA = 128.30 sec*proc (17 tests)
backend_all = 129.94 sec*proc (17 tests)
backend_gda = 55.17 sec*proc (7 tests)
backend_ipc = 67.41 sec*proc (11 tests)
comprehensive = 197.34 sec*proc (28 tests)
full = 197.34 sec*proc (28 tests)
functional = 197.34 sec*proc (28 tests)
gpu_all = 197.34 sec*proc (28 tests)
launcher_slr = 197.34 sec*proc (28 tests)
nightly = 96.18 sec*proc (12 tests)
pr = 7.85 sec*proc (2 tests)
pre-commit = 197.34 sec*proc (28 tests)
quick = 197.34 sec*proc (28 tests)
smoke = 197.34 sec*proc (28 tests)
standard = 197.34 sec*proc (28 tests)
variant_default_stream = 15.98 sec*proc (2 tests)
variant_uuid = 189.14 sec*proc (27 tests)
Total Test time (real) = 88.85 sec
# TEST_COMPONENT: rocshmem -> Test Directory: build/bin/rocshmem
# TEST_TYPE: quick -> Category: quick
# AMDGPU_FAMILIES: gfx94X-dcgpu -> GPU Architecture: gfx94X
# Discovering available test labels...
# Warning: No GPU specific test suites available
# No GPU suite found for gfx94X, excluding all ex_gpu tests
# Running: ctest -L ^quick$ -LE ex_gpu --output-on-failure --parallel 1 --timeout 7200 --test-dir build/bin/rocshmem -V --tests-information 1,,1
That seems pretty light on tests, even for the "quick" filter? The "No GPU suite found" logs are also concerning given that this claims it should be running multi-GPU tests?
Summary
TheRock side of test-filter standardization for rocSHMEM, paired with ROCm/rocm-systems#6334.
rocm-systems#6334 adds native CTest integration for rocSHMEM:
projects/rocshmem/tests/test_categories.yamlplusapply_ctest_category_labels()wiring that installs relocatablebin/rocshmem/**/CTestTestfile.cmakefiles (a root dispatcher thatsubdirs()intotests/functionalandtests/unit) carrying thequick/standard/comprehensive/fulltier labels. This PR wires TheRock so the generictest_runner.pycan drive those tests from the installed artifact:comm-libs/CMakeLists.txt— pass-DBUILD_CTESTS=${THEROCK_BUILD_TESTING}to the rocshmem subproject. The CTest registration and the install-treeCTestTestfile.cmakegeneration in (feat): [AILIKFD-40] add kfdtest to TheRock CI #6334 are gated behindBUILD_CTESTS(defaultOFF), so without this the install tree would contain nobin/rocshmem/**andtest_runner.pywould fail withTest directory does not exist.comm-libs/artifact-rocshmem.toml— populate the previously-emptytestcomponent:bin/rocshmem/**(the install-tree CTest files),bin/rocshmem_unit_tests, theshare/rocshmem/*CTest wrapper scripts, and thedevice_bitcodeHSACOs. The functional binary androcshmem_infoalready live in theruncomponent (fetched viarocshmem_run).build_tools/install_rocm_from_artifacts.py— add a--rocshmemartifact selector that pullsrocshmem_lib/_run/_testplus OpenMPI (mpirun+ libs) needed to launch the tests.build_tools/github_actions/fetch_test_configurations.py— add arocshmemtest job usingtest_runner.py(fetch_artifact_args = "--rocshmem --tests"), on Linux multi-GPU runners (the functional/unit tests launch viampirunwithRANKS2..64, same setup asrccl).fetch_test_configurations_test.py— relax the assertion that assumed exactly one multi-GPU job repo-wide, now that rocshmem is a second multi-GPU job alongside rccl.test_runner.pyneeds noCOMPONENT_DIR_MAPPINGentry (job namerocshmem== install dirbin/rocshmem).Test plan
python3 -m pytest build_tools/github_actions/tests/fetch_test_configurations_test.py(22 passed)black --checkon the edited Python files--rocshmemregistered ininstall_rocm_from_artifacts.py --helpartifact-rocshmem.tomlparses; testincluderesolvesTHEROCK_BUILD_TESTING=ON, confirmbin/rocshmem/CTestTestfile.cmake+tests/{functional,unit}/CTestTestfile.cmakeare installed and bundled in the test artifactTEST_COMPONENT=rocshmem TEST_TYPE=quick python test_runner.pydiscovers labels and runsctest -L quickfromTHEROCK_BIN_DIR/rocshmemTest Results
Test Run: https://github.com/ROCm/TheRock/actions/runs/30112055659/job/89560360151?pr=5691
Made with Cursor