[edgargabriel]

Motivation

This commit adds native CTest integration for rocshmem functional and unit
tests, providing an alternative to the shell-based driver.sh scripts while
maintaining backward compatibility.

Changes

• Added CTestFunctionalTests.cmake with functional tests
• Added CTestUnitTests.cmake with unit test configuration
• Created test_wrapper.sh for runtime backend detection and conditional skipping
• Created ctest_with_retry.sh for automatic retry of failed tests (matches driver.sh)
• Updated CMakeLists.txt in tests/functional_tests and tests/unit_tests
• Tests organized with labels: RMA, AMO, COLLECTIVE, SIGOPS, STREAM, HEATMAP, etc.
• Individual log files created per test (matching driver.sh behavior)

Compilation Flags

To enable CTest-based tests during build:

cmake -DGPU_TARGETS=gfx950 \
      -DROCSHMEM_USE_CTEST=ON \
      -DBUILD_FUNCTIONAL_TESTS=ON \
      -DBUILD_UNIT_TESTS=ON \
      ..
cmake --build . -j8

Running Functional Tests

cd build_all_backends/tests/functional_tests

# Run all functional tests
ctest -L functional

# Run with automatic retry (matches driver.sh behavior)
./ctest_with_retry.sh -L functional

# Run specific test suite
ctest -L RMA          # RMA tests
ctest -L AMO          # Atomic operations
ctest -L COLLECTIVE   # Collective operations
ctest -L HEATMAP      # Performance tests

# Run for specific backend (auto-skips incompatible tests)
ROCSHMEM_BACKEND=ipc ctest -L functional
ROCSHMEM_BACKEND=ro ctest -L functional
ROCSHMEM_BACKEND=gda ctest -L functional

# Run specific test
ctest -R put_n2_w1_z1_1048576B -V

Running Unit Tests

cd build_all_backends/tests/unit_tests

# Run all unit tests
ctest

# Run with verbose output
ctest -V

# Run with labels
ctest -L unit
ctest -L IPC
ctest -L SDMA

Log Files

Test logs are created in:

• Functional tests: build_all_backends/tests/functional_tests/test_logs/
• Unit tests: CTest captures output (use -V to see details)

Log file naming: <test_name>_n<ranks>_w<workgroups>_z<threads>_<size>B.log

Example:

test_logs/
├── put_n2_w1_z1_1048576B.log
├── get_n2_w1_z1_1048576B.log
├── amo_add_n2_w1_z1.log
└── init_n2_w1_z1.log

Custom log directory:

ROCSHMEM_TEST_LOG_DIR=/path/to/logs ctest -L functional

Running theRock CI categories

The generation of the theRock CI categories requires an additional entry when doing a sparse checkout. Please execute

git sparse-checkout add shared/ctest

in addition to setting the cone to point to the project/rocshmem directly.

cd build
ctest --test-dir=./tests -L quick
ctest --test-dir=./tests -L standard
ctest --test-dir=./tests -L comprehensive
ctest --test-dir=./tests -L full

Backward Compatibility

The original driver.sh scripts remain available:

• rocshmem_functional_driver.sh <executable> <suite> <log_dir> [hostfile]
• rocshmem_unit_driver.sh <binary> all

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

JIRA ID

AIROCSHMEM- 399

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[Copilot]

Pull request overview

Adds native CTest integration for rocSHMEM’s functional and unit tests, aiming to provide a CMake/CTest-driven alternative to the existing driver.sh workflows while keeping the legacy scripts available.

Changes:

• Introduces CTestFunctionalTests.cmake to register the functional test matrix (including labels and per-test logging via a wrapper).
• Introduces CTestUnitTests.cmake to register unit tests via MPI + gtest filters.
• Adds helper scripts: test_wrapper.sh (conditional skips + logging) and ctest_with_retry.sh (rerun-failed behavior).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
projects/rocshmem/tests/unit_tests/CTestUnitTests.cmake	New CTest registration for MPI-based unit tests.
projects/rocshmem/tests/unit_tests/CMakeLists.txt	Enables optional CTest registration via ROCSHMEM_USE_CTEST.
projects/rocshmem/tests/functional_tests/test_wrapper.sh	New runtime wrapper for backend-based skipping and per-test logging.
projects/rocshmem/tests/functional_tests/CTestFunctionalTests.cmake	New CTest registration for functional tests, labels, env, and mpirun invocation.
projects/rocshmem/tests/functional_tests/ctest_with_retry.sh	Wrapper to retry failed CTest tests via --rerun-failed.
projects/rocshmem/tests/functional_tests/CMakeLists.txt	Enables optional CTest registration and installs helper scripts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Yiltan]

We should probably add the groupings "quick, standard, comprehensive, full" for the tests to make The Rock integration easier.

Also filters for different arches, I think we currently do that in rocJenkins

https://github.com/ROCm/TheRock/blob/608407f8409b0e33b54361226ab09673927e0f0a/docs/development/test_filtering.md

[edgargabriel]

We should probably add the groupings "quick, standard, comprehensive, full" for the tests to make The Rock integration easier.

Also filters for different arches, I think we currently do that in rocJenkins

https://github.com/ROCm/TheRock/blob/608407f8409b0e33b54361226ab09673927e0f0a/docs/development/test_filtering.md

Done, added the quick, standard, comprehensive, and full category with the latest updates

[abouteiller]

Looks good in general! Maye we also need to update the documentation though

[edgargabriel]

High level workflow description, generated with the help of 'claude'

Here's the high-level logical flow of the CTest-based testing system:

1. CMake Configuration Time (when you run cmake)

cmake is invoked
↓
MaindCMakeLists.txt processes configuration options
↓
find_package(MPI) → populates MPIEXEC_EXECUTABLE, MPIEXEC_NUMPROC_FLAG
↓
ROCSHMEM_USE_CTEST=ON by default
↓
tests/CMakeLists.txt is processed
↓
Functional and Unit test subdirectories are added

In Functional Tests (CTestFunctionalTests.cmake):

Check MPIEXEC_EXECUTABLE exists (bail if not)
↓
Read MCA_PML and MCA_OSC environment variables
→ If set: use those values
→ If not set: default to "ucx"
↓
Store as MCA_PML_VALUE and MCA_OSC_VALUE
↓
For each test definition:
→ Build test command with:
- test_wrapper.sh
- MPIEXEC_EXECUTABLE (e.g., mpirun)
- MPIEXEC_NUMPROC_FLAG -n ${RANKS}
- -mca pml ${MCA_PML_VALUE}
- -mca osc ${MCA_OSC_VALUE}
- Environment variables (-x flags)
- Test executable + arguments
→ Register with add_test()
→ Assign labels (tier, category, backend, variant)
↓
~445 functional tests registered

In Unit Tests (CTestUnitTests.cmake):

Same MCA parameter detection
↓
Build unit test command with MCA parameters
↓
Register unit tests (currently 1 test: unit_tests_n4)

2. Test Execution Time (when you run ctest)

User runs: ctest -L quick
↓
CTest filters tests by label "quick"
↓
For each matching test:
CTest executes the TEST_COMMAND

Test Command Execution Flow:

test_wrapper.sh <test_name> -n -mca pml ucx -mca osc ucx [env vars] [executable] [args]
↓
Wrapper checks GPU count
→ If insufficient GPUs: exit 125 (skip test)
↓
Wrapper detects backend (smart detection):
→ If ROCSHMEM_BACKEND env var set: use it
→ Else: run rocshmem_info to detect compiled backends
- If 1 backend compiled: use it (apply skip logic)
- If multiple backends: set BACKEND="multi" (no skips)
↓
Wrapper applies skip logic based on backend:
→ If test requires specific backend and doesn't match: exit 125 (skip)
↓
Wrapper constructs final mpirun command:
mpirun -n -mca pml ucx -mca osc ucx -x ENV1 -x ENV2 ... --timeout X --map-by numa
↓
Execute the test
↓
Return exit code to CTest:
→ 0 = pass
→ 125 = skip
→ other = fail

3. Key Components

MCA Parameters:

• Set once at CMake configure time (not runtime)
• Read from MCA_PML and MCA_OSC environment variables
• Default to "ucx" if not set
• Baked into every test command as -mca pml ${MCA_PML_VALUE} -mca osc ${MCA_OSC_VALUE}

Backend Detection (in test_wrapper.sh):

1. Check if ROCSHMEM_BACKEND env var is set → use it
2. Otherwise, query rocshmem_info to see what's compiled
3. If single backend (e.g., only IPC): use it and apply skip logic
4. If multiple backends (e.g., RO+IPC+GDA): set to "multi" and don't skip tests

Test Organization:

• Labels: Tests tagged with tier (quick/standard/comprehensive/full), category (RMA/AMO/COLLECTIVE), backend (RO/IPC/GDA), variant
  (uuid/default_stream)
• Filtering: ctest -L quick runs only quick tier, ctest -L RMA runs only RMA tests, etc.
• Parallel execution: CTest can run tests in parallel (respects PROCESSORS property)

Backward Compatibility:

• rocshmem_functional_driver.sh still built and installed
• Works independently from CTest
• Users can choose: ctest (modern) or ./driver.sh (traditional)

4. Environment Variable Flow

Configure time:
MCA_PML, MCA_OSC → Baked into test commands

Runtime:
ROCSHMEM_BACKEND → Passed to wrapper, controls backend selection
ROCSHMEM_BACKEND_TYPE → Alternative to ROCSHMEM_BACKEND (wrapper checks both)

Test execution:
ROCSHMEM_MAX_NUM_CONTEXTS → Set per-test via -x flag
UCX_ROCM_IPC_SIGPOOL_MAX_ELEMS → Set via -x flag
ROCSHMEM_HEAP_SIZE → Set via -x flag
ROCSHMEM_TEST_UUID → Set for uuid variant tests
ROCSHMEM_TEST_USE_DEFAULT_STREAM → Set for default_stream variant

5. Summary

The flow is: Configure → Register → Filter → Execute → Report

• Configure: Detect MPI, read MCA parameters, build test commands
• Register: Each test registered with labels and command
• Filter: User selects tests via labels
• Execute: Wrapper checks resources, detects backend, runs mpirun
• Report: CTest collects results (pass/fail/skip)

All MCA parameters and mpirun flags are determined at CMake configure time, ensuring consistent test execution and matching the old driver.sh
behavior.

[edgargabriel]

@abouteiller I think I was able to recover what was accidentally lost, if you have a few minutes, can you please re-review?

The major changes:

• configuration of theRock test categories are now done based on the test_configuration.yaml file
• included some more tests into the 'quick' category to cover more functional areas, but still staying below the 5 min limit
• recognizing the MCA_PML and MCA_OSC environment variables at rocshmem configure time that influence the arguments of the mpirun command line that is being constructed
• use the variables set by find_package(MPI) instead of trying to define new ones.