[atulkulk]

Motivation

RCCL’s existing LLVM source-based coverage workflow reports host coverage but does not reliably instrument or collect coverage from GPU device code. Enabling device coverage manually also depends on compiler capabilities and the availability of the AMDGPU profile runtime, making the workflow fragile across ROCm installations.

This PR provides a single test-runner workflow for host and device coverage. Device coverage is automatically enabled when the selected ROCm compiler supports it and gracefully falls back to host-only coverage on older or incomplete installations.

Technical Details

The build now supports ENABLE_DEVICE_COVERAGE=AUTO|ON|OFF:

• AUTO enables device coverage when ROCm 7.15 or newer is selected and the compiler provides both device coverage instrumentation and the AMDGPU libclang_rt.profile.a runtime. Host+device coverage for this PR was validated on ROCm 10.1 (TheRock), not ROCm 7.15.
• AUTO falls back to host-only coverage when those requirements are not met.
• ON is strict and fails configuration when device coverage is unavailable.
• OFF explicitly disables device instrumentation.

A new DeviceCoverage.cmake module probes the selected compiler instead of relying only on the ROCm version. The probe verifies support for the required LLVM coverage options and locates the AMDGPU profile runtime through the compiler resource directory.

The RCCL device-link flow now:

• Uses clang-offload-bundler from the same LLVM installation as the selected compiler.
• Instruments RCCL device code with LLVM source-based coverage.
• Links the AMDGPU profile runtime into the device image.
• Retains __llvm_profile_hip_collect_device_data so device profile data is collected when the process terminates.
• Rejects missing or mixed LLVM toolchain components with actionable configuration errors.

The test runner’s --coverage-report option now drives the complete workflow:

• Enables host coverage and requests automatic device coverage.
• Uses workspace-local raw profile storage.
• Names profiles with hostname, process, and module identifiers to prevent collisions between ranks and nodes.
• Removes stale profiles before a new test run.
• Preserves existing profiles in report-only mode.
• Resolves llvm-profdata and llvm-cov from the selected ROCm installation.
• Propagates missing profiles, merge failures, and report-generation failures to the runner exit status.
• Keeps rccl-tests outside device instrumentation while using them to exercise the instrumented RCCL library.

Configuration expansion also supports shell-style defaults such as ${VARIABLE:-default}.

Issue Tracking

JIRA ID: AICOMRCCL-2318
JIRA ID: ROCM-29951

AICOMRCCL-2318 tracks enabling compiler-gated device coverage in RCCL. ROCM-29951 tracks the HIP runtime fix for missing host-only LLVM profile descriptors. Host+device coverage was validated on ROCm 10.1 (TheRock) on MI300X, not on ROCm 7.15.

Test Plan

Automated regression tests

cd projects/rccl/tools/scripts/test_runner
python3 -m unittest discover -s tests -v

These tests cover automatic device-coverage selection, host-only fallback, compiler capability probing, profile-runtime detection, workspace profile isolation, stale-profile cleanup, report-only profile preservation, ROCm-relative LLVM tool selection, report failure propagation, and configuration variable expansion.

Host and device coverage

Run host+device coverage on MI300X using ROCm 10.1 (TheRock). Do not use ROCm 7.15 for this validation. AUTO still gates device instrumentation at ROCm 7.15 or newer; the numbers in Test Result come from the 10.1 run.

Device coverage introduces substantial execution overhead. Use a 30-minute timeout for the 1 GiB AllReduce, Reduce, and ReduceScatter tests while retaining the standard timeout for other tests.

export ROCM_PATH=/scratch/users/atkulkar/therock-10.1
export ROCM_SYSTEMS_ROOT="$(git rev-parse --show-toplevel)"
export WORKDIR="${ROCM_SYSTEMS_ROOT}/projects/rccl"
export RCCL_TESTS_DIR="${ROCM_SYSTEMS_ROOT}/projects/rccl-tests"
export MPI_PATH=/home/AMD/atkulkar/softwares/ompi
export UCX_PATH=/scratch/users/atkulkar/softwares/ucx
export XPMEM_PATH=/home/AMD/atkulkar/softwares/xpmem

export PATH="${ROCM_PATH}/bin:${ROCM_PATH}/lib/llvm/bin:${MPI_PATH}/bin:${PATH}"
export LD_LIBRARY_PATH="${ROCM_PATH}/lib:${ROCM_PATH}/lib64:${MPI_PATH}/lib:${UCX_PATH}/lib:${XPMEM_PATH}/lib:${LD_LIBRARY_PATH:-}"
export HSA_NO_SCRATCH_RECLAIM=1
export COVERAGE_OUTPUT="${WORKDIR}/rccl_test_artifacts_$(date +%Y%m%d_%H%M%S)"

cd "${WORKDIR}/tools/scripts/test_runner"

python3 test_runner.py \
  --config configs/mi300x_mellanox_ib.json \
  --coverage-report \
  --output "${COVERAGE_OUTPUT}" \
  --verbose

Verify that device coverage is selected, the collective runs complete, workspace-local profiles are generated, host and device profiles merge, and HTML/text reports contain nonzero device-function coverage.

Test Result

Environment:

• ROCm: TheRock 10.1
• Hardware: AMD Instinct MI300X (gfx942)
• Commit: 6787a9c4
• Suite: 20 collective runs (host+device coverage)
• Profiles merged: 4340
• Report generated: 2026-09-09 16:24 EDT

Results:

• Test-runner regression suite: 13/13 passed.
• Collective coverage suite: 20/20 passed.
• Host and device profiles merged successfully.
• HTML and text reports generated successfully.
• Line coverage: 23.76%.
• Function coverage: 33.68%.
• Region coverage: 10.13%.
• Branch coverage: 8.35%.
• Host-only fallback validation on an older ROCm installation.
• PR CI.

Coverage artifact:

• File: rccl-host-device-coverage-mi300x-6787a9c4.tar.gz
• SHA-256: f7a4b188c8db7387ab6be092ddf8d6a638bcc9e7d663b61e7167694a2860e489
• Contents: HTML report, text reports, coverage summary, and merged LLVM profile data.
• Ticket: AICOMRCCL-2318

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[atulkulk]

Uploading rccl-host-device-coverage-mi300x-27-tests-4fde916.tar.gz…

[Copilot]

Pull request overview

This PR adds a compiler-gated, unified host+device LLVM source-based coverage workflow for RCCL, with automatic fallback to host-only coverage when device instrumentation/runtime support is unavailable.

Changes:

• Introduces ENABLE_DEVICE_COVERAGE=AUTO|ON|OFF with compiler capability probing and device profile-runtime detection (DeviceCoverage.cmake), plus CMake glue to enable/disable device coverage safely.
• Extends the RCCL device-link pipeline to instrument device code, link the AMDGPU profile runtime, and ensure device profile collection at process exit.
• Updates the Python test runner to drive coverage end-to-end (workspace-isolated profraws, stale cleanup, ROCm-relative llvm tool resolution, failure propagation) and adds regression tests.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
projects/rccl/tools/scripts/test_runner/tests/test_device_coverage_cmake.py	Adds unit tests for the CMake probe that locates the device profile runtime and validates compiler coverage flag support.
projects/rccl/tools/scripts/test_runner/tests/test_coverage_reporting.py	Adds unit tests for workspace profraw isolation/cleanup and coverage report failure propagation/tool resolution.
projects/rccl/tools/scripts/test_runner/tests/test_config_expansion.py	Adds unit test for ${VAR:-default} expansion behavior when env var is empty.
projects/rccl/tools/scripts/test_runner/tests/test_build_coverage_flags.py	Adds unit tests for coverage-driven build flag/CMake option selection.
projects/rccl/tools/scripts/test_runner/test_runner.py	Makes coverage report generation failures affect the runner exit status.
projects/rccl/tools/scripts/test_runner/README.md	Updates CLI/docs for coverage behavior, workspace layout, and report-only workflow via --output.
projects/rccl/tools/scripts/test_runner/pytest.ini	Configures pytest discovery/import mode for the runner’s tests.
projects/rccl/tools/scripts/test_runner/lib/test_parser.py	Updates --coverage-report help text to reflect device-coverage behavior.
projects/rccl/tools/scripts/test_runner/lib/test_executor.py	Implements coverage build configuration, workspace-local profraw handling, llvm tool resolution, and device ELF inclusion in reports.
projects/rccl/tools/scripts/test_runner/lib/test_config.py	Adjusts ${VAR:-default} expansion to treat empty env vars as “use default”.
projects/rccl/tools/scripts/test_runner/conftest.py	Ensures the test_runner package root is importable for pytest.
projects/rccl/tools/scripts/test_runner/configs/mi455x_perf.json	Enables rccl-tests build configuration and uses defaulting env-var expansion for RCCL_TESTS_DIR.
projects/rccl/tools/scripts/test_runner/configs/mi455_ainic_roce_func.json	Enables rccl-tests build configuration and uses defaulting env-var expansion for RCCL_TESTS_DIR.
projects/rccl/tools/scripts/test_runner/configs/mi355x_thor2_roce_func.json	Enables rccl-tests build configuration and uses defaulting env-var expansion for RCCL_TESTS_DIR.
projects/rccl/tools/scripts/test_runner/configs/mi300_mellanox_ib_func.json	Enables rccl-tests build configuration and uses defaulting env-var expansion for RCCL_TESTS_DIR.
projects/rccl/tools/rccl-device-compile	Adds coverage-related assembly filtering, output-dir creation, and profile runtime linking support for device ELF builds.
projects/rccl/src/CMakeLists.txt	Enables device-side compilation flags when device coverage is active and force-links the HIP device-data drain symbol.
projects/rccl/README.md	Documents host vs host+device coverage install flags.
projects/rccl/install.sh	Adds --enable-device-coverage flag and enforces --debug requirement for coverage modes.
projects/rccl/CMakeLists.txt	Adds ENABLE_DEVICE_COVERAGE cache option, AUTO/ON/OFF logic, compiler/runtime probing, and implied host coverage.
projects/rccl/cmake/DeviceLinker.cmake	Pins toolchain components for device coverage, threads device coverage flags/runtime into the device-link pipeline, and emits per-arch device ELF symlinks.
projects/rccl/cmake/DeviceCoverage.cmake	New module to probe compiler support and locate AMDGPU profile runtime via the compiler resource directory.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[amd-argus]

ARGUS

Findings

2 finding(s) that fall outside the diff and could not be anchored to a line

• projects/rccl/cmake/DeviceLinker.cmake:881 (Medium) The two GIN SDMA fat objects are the only device TUs left uninstrumented under ENABLE_DEVICE_COVERAGE
• projects/rccl/tools/scripts/test_runner/README.md:622 (Low) The report-only example at README.md:622 still omits --output and now always fails

This PR changes 1,055 lines across 22 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: the new sys.exit(1) at test_runner.py:129 fires before emit_results(), so a run whose tests all passed but whose coverage report failed loses its dashboard artifacts and exits an indistinguishable 1.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[amd-argus]

ARGUS

Findings

3 finding(s) that fall outside the diff and could not be anchored to a line

• projects/rccl/cmake/DeviceLinker.cmake:875 (Medium) The two GIN-SDMA fat objects are instrumented through DL_INHERITED_FLAGS but never receive the profile-runtime link flags their 21 siblings get
• projects/rccl/tools/scripts/test_runner/lib/test_executor.py:2556 (Low) The dashboard build_type derivation did not get the coverage-forces-debug term the build_dir derivation gained
• projects/rccl/.github/PULL_REQUEST_TEMPLATE.md:2 (Low) The PR description embeds internal paths, a username and an unreleased toolchain, which the PR template asks contributors to keep out

This PR changes 1,327 lines across 25 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: with ENABLE_ROCSHMEM_GIN on, the two GIN-SDMA objects now get instrumented through DL_INHERITED_FLAGS but never get the profile-runtime link flags their 21 siblings do.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[sdarisala]

1. What code in the PR gates the device coverage support only for rccl-tests?

2. What is the plan to leverage the device coverage support in the CI?

[sdarisala]

I am not able to access "Uploading rccl-host-device-coverage-mi300x-27-tests-4fde916.tar.gz…". I get "not found" error. Can you please check?

[atulkulk]

@amd-argus review it again

[atulkulk]

I am not able to access "Uploading rccl-host-device-coverage-mi300x-27-tests-4fde916.tar.gz…". I get "not found" error. Can you please check?

Uploaded to the Jira ticket: AICOMRCCL-2318

[atulkulk]

@amd-argus The open review comments are addressed on HEAD 6787a9c4 (after your last pass on b79b721). Resolving them.

First-round items were already confirmed in your “Resolved by b79b721” replies.

Re-review items landed in 6787a9c4:

• ENABLE_DEVICE_LINKER=OFF is a full-coverage blocker (AUTO falls back, ON fails configure)
• AUTO/ON policy is one _full_coverage_blocker decision; cache is FORCEd to the effective value
• CHANGELOG documents --enable-full-coverage as strict; the runner uses ENABLE_FULL_COVERAGE=AUTO
• GIN-SDMA fat objects now pass ${DL_DEVICE_PROFILE_RT_LINK_FLAGS} with the inherited coverage flags
• Dashboard rccl_build_type treats --coverage-report as debug, matching build_dir
• Runner tests cover empty install_flags vs coverage, fold the seed-flag cases with subTest, assert if not device_elfs, and load rccl-device-compile via importlib
• Report-only README example includes --output
• python3 -m unittest discover -s tests -t . is wired in .github/workflows/rccl-host-unit-tests.yml

GIN-SDMA still uses DL_INHERITED_FLAGS + DL_DEVICE_PROFILE_RT_LINK_FLAGS rather than the folded DL_DEVICE_COVERAGE_FLAGS variable. Same flags reach the compile; not changing that in this PR.

Please re-review on 6787a9c4.

[atulkulk]

1. What code in the PR gates the device coverage support only for rccl-tests?
2. What is the plan to leverage the device coverage support in the CI?

1. Device coverage is not gated to rccl-tests. ENABLE_FULL_COVERAGE instruments RCCL’s host and device code regardless of the consuming test executable. rccl-tests is currently the workload used by the test runner to exercise that instrumentation and generate the report.
2. This will be a separate task and a follow-up PR.

[amd-argus]

ARGUS

This PR changes 1,414 lines across 23 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: the AUTO host-only fallback still builds device-*.elf, so a coverage run on pre-7.15 ROCm feeds uninstrumented objects to llvm-cov (test_executor.py:2238).

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[atulkulk]

@argus re-review

[amd-argus]

ARGUS

This PR changes 1,859 lines across 23 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: Worth settling before approval: rccl_resolve_full_coverage's capability_error and rocm_version_ok blockers are still unpinned, so AUTO can stop falling back with the suite green.

_{Comment @argus review to re-run, or @argus re-review after pushing.}

[amd-argus]

ARGUS

This PR changes 1,874 lines across 23 files. A change this size is hard to review
well, for a person and for this bot alike: reviewers lose the thread, defects hide in the
volume, and the feedback arrives too late to be cheap to act on. Splitting it into a
stacked pull request
gives each piece a reviewable diff and keeps the history. This review covers the whole change
regardless.

Bottom line: LGTM.

_{Comment @argus review to re-run, or @argus re-review after pushing.}