TheRock : Code Coverage - enable coverage test runs and report generation by jainprad · Pull Request #8602 · ROCm/TheRock

jainprad · 2026-09-29T16:29:07Z

Motivation

GitHub issue: #6572

This is the test part of TheRock's code coverage support, split out of the code coverage PR (7783) so it can be reviewed on its own. It covers installing an instrumented build for a test job, running the tests with profile collection, and turning the collected profiles into coverage reports. It does not depend on the build part (#8598); the coverage CI workflows that call it follow in a separate PR.

Technical Details

build_tools/install_rocm_code_coverage_build.py: adds --code-coverage-run-github-repo and --code-coverage-release-type, so the instrumented replacement artifacts can come from a different repository and release channel than the baseline install (--run-id/--run-github-repo). The defaults are $GITHUB_REPOSITORY and ci. create_backend_from_env in _therock_utils/artifact_backend.py takes the matching release_type.
build_tools/github_actions/merge_coverage_report.py (new): merges the .profraw files from a test run into .profdata, then writes an lcov report, a per-file text summary and an optional annotated HTML report. It uses llvm-profdata and llvm-cov from the installed ROCm tree, so the tools match the compiler that built the objects.
.github/workflows/test_code_coverage_component.yml:
- adds a baseline_repository input, forwarded as --run-github-repo;
- falls back to this run's own artifacts when generic_run_id is empty;
- sets up Python 3.12;
- skips additional requirements cleanly when a component has none;
- moves three ${{ inputs.* }} expansions out of its run: block into environment variables, and marks the container image expression as reviewed for zizmor. These six findings are already on main and fail the security scan for any PR that touches this file. Every image the expression can resolve to is digest-pinned.
.github/actions/setup_test_environment/action.yml: optional RUN_GITHUB_REPO input, forwarded to install_rocm_from_artifacts.py as --run-github-repo. Empty by default.
build_tools/artifact_manager.py: fetch --artifact-names narrows a stage fetch to a named set of artifacts. Empty by default.

Producing a report locally after running an instrumented project's tests:

export LLVM_PROFILE_FILE="$PWD/coverage-report/profraw/%p-%m.profraw"
# ... run the tests ...
python build_tools/github_actions/merge_coverage_report.py \
  --profraw-dir coverage-report/profraw \
  --rocm-dir <installed ROCm tree> \
  --object-globs "lib/libhiprand.so*" \
  --summary-output coverage-report/coverage_summary.txt \
  --html-output coverage-report/html

Test Plan

Unit tests: build_tools and test_tools (Python 3.12), including the new install_rocm_code_coverage_build_test.py and merge_coverage_report_test.py and the new artifact_manager_tool_test.py cases.
pre-commit run --all-files, and zizmor 1.30.0 on the changed workflow and action with the repository's config.
End to end on an MI325X (gfx942): install a baseline plus instrumented hipRAND with install_rocm_code_coverage_build.py, run hipRAND's tests with LLVM_PROFILE_FILE set, and generate the lcov, text and HTML reports with merge_coverage_report.py.
CI on this PR: the full Multi-Arch CI, which uses artifact_manager.py in every build job and setup_test_environment in every test job.

Test Result

build_tools: 2467 passed, 8 skipped. test_tools: 74 passed. Tests for the changed modules: 104 passed.
pre-commit: all hooks pass. zizmor: 0 high findings (6 before the workflow change).
End to end on an MI325X (gfx942), with the tests run in the CI test container:
- install_rocm_code_coverage_build.py installed the baseline from a ROCm/TheRock CI run (36550488477) and replaced hipDNN from the instrumented ROCm/rocm-libraries coverage run 36168158239, which --code-coverage-run-github-repo resolved to therock-ci-artifacts-external. Only hipDNN's paths were replaced: libhipdnn_backend.so carries the profile sections and libamdhip64.so does not. Looking the same run up against ROCm/TheRock finds 0 artifacts.
- hipDNN's tests: 11/11 passed with LLVM_PROFILE_FILE set, writing 39 .profraw files.
- merge_coverage_report.py: lcov for 183 source files, the text summary, and a 181-page HTML report using --path-equivalence. Totals are 81.09% of lines, 75.42% of functions, 83.92% of regions and 80.07% of branches, identical to that run's own Coverage Report job.
- The install step of test_code_coverage_component.yml, run with a stub installer, assembles the right arguments both with and without a cross-repo baseline.
CI on this PR: pre-commit, Unit Tests (Linux and Windows), Security scan including zizmor, and the PR bot pass. Multi-Arch CI: in progress.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/CONTRIBUTING.md.

Adds the test side of TheRock's code coverage support, split out of the code coverage PR (7783) so it can be reviewed on its own. It does not depend on the build part (8598). - install_rocm_code_coverage_build.py: --code-coverage-run-github-repo and --code-coverage-release-type, so the instrumented replacement artifacts can come from a different repository and release channel than the baseline install. create_backend_from_env takes the matching release_type. - merge_coverage_report.py: merges the .profraw files from a test run into .profdata and writes an lcov report, a per-file text summary and an annotated HTML report, using the llvm-profdata and llvm-cov from the installed ROCm tree. Unit tested. - test_code_coverage_component.yml: baseline_repository input, falls back to this run's own artifacts when generic_run_id is empty, sets up Python 3.12, and skips additional requirements cleanly when there are none. - setup_test_environment: optional RUN_GITHUB_REPO input, forwarded to install_rocm_from_artifacts.py as --run-github-repo. - artifact_manager.py fetch: optional --artifact-names include list. test_code_coverage_component.yml also moves three ${{ inputs.* }} expansions out of its run: block into environment variables and marks its container image expression as reviewed for zizmor. Those findings are already on main and fail the security scan for any PR that touches the file; every image the expression can resolve to is digest-pinned. Co-authored-by: Cursor <cursoragent@cursor.com>

therock-pr-bot · 2026-09-29T16:29:38Z

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-29T16:30:12Z

🎉 All checks passed! This PR is ready for review.

ScottTodd

I'll wait to review this until someone else makes a first pass review.

…mespace Fixes the failing Linux CodeCoverage jobs in the rocm-libraries nightly (run 36913152797): - Instrumented subprojects no longer compile through ccache. ccache drops every -Xarch_<arch> <arg> pair, which removed the device-side negation (kernels were instrumented, and their colon-named profiles broke the profraw upload) and rocRAND's and hipRAND's -Xarch_host coverage flags (those libraries were not instrumented at all). - The instrumented stack publishes under <run_id>-coverage. The nightly's regular build shares the workflow run, and the two overwrote each other's math-libs artifacts. WorkflowOutputRoot looks a suffixed run id up by its numeric part. - external_repo_config is forwarded to the coverage builds, which otherwise measured TheRock's pinned submodules rather than the commit under test. - rocWMMA's coverage option is ROCWMMA_CODE_COVERAGE. rocPRIM, hipCUB and rocThrust report on their own test binaries; a "!" glob excludes a sibling project's. - Coverage test timeouts are 4x the release ones, capped at 180 minutes. - Device-side profiles are dropped, with a warning, before upload. - merge_coverage_report.py fails a report whose profiles hold no counters, whose objects carry no coverage mapping, or in which no line ran, instead of publishing 0%. Ported from #7783 onto the split #8598/#8602/#8605 stack: the registry entries keep #8598's derived stage/test_component/coverage_config, the report's run id goes through env like the rest of the step, the external_repo_config input this PR had already added is merged with this one, and copy_compiler_runtime writes into the same <run_id>-coverage namespace. (cherry picked from commit 8523c5a) Co-authored-by: Cursor <cursoragent@cursor.com>

endurthiabhilash · 2026-10-23T18:46:34Z

@jainprad
Please resolve the merge conflicts before we do another round of reviews. #8680 fixed some security issues in overlapping files.

jainprad · 2026-10-26T16:07:18Z

@jainprad Please resolve the merge conflicts before we do another round of reviews. #8680 fixed some security issues in overlapping files.

@endurthiabhilash , Done

endurthiabhilash · 2026-10-29T04:16:58Z

          python3 build_tools/install_rocm_code_coverage_build.py \
            --code-coverage-run-id="${CODE_COVERAGE_RUN_ID}" \
            --run-id="${GENERIC_RUN_ID}" \
            --artifact-group="${AMDGPU_FAMILIES}" \


The baseline_repository input is never used. The step sets BASELINE_REPOSITORY in env, but the command line passes no --run-github-repo so, the baseline fetch therefore still queries generic_run_id against the current repo. This seems like a regression from mainline rebase. Please check.

This regressed in the main merge: resolving the conflict with #8680's security changes kept main's install command, which dropped the --run-github-repo forwarding and also the fallback for an empty generic_run_id (that case was passing --run-id=). 3b6c4de restores both, using environment variables the way main now does, plus the additional_requirements_files[0] skip that was lost the same way. Through a stub installer, a cross-repo baseline now gets --run-github-repo=<repo>, a same-repo baseline gets no extra argument, and an empty generic_run_id falls back to this run's ID.

… merge Resolving the main-merge conflict with #8680's security changes kept main's install command. baseline_repository was then set in the step env but never passed as --run-github-repo, so a cross-repo baseline was looked up in the current repository, and an empty generic_run_id went through as --run-id= instead of falling back to this run's own artifacts. Restore both in the env-var form main now uses, along with the first-element check that skips Setup Additional Requirements when a component has none, and drop the step-level GENERIC_RUN_ID that the job env already sets. Co-authored-by: Cursor <cursoragent@cursor.com>

github-project-automation Bot added this to TheRock Triage Sep 29, 2026

github-project-automation Bot moved this to TODO in TheRock Triage Sep 29, 2026

github-advanced-security Bot found potential problems Sep 29, 2026

View reviewed changes

Comment thread .github/workflows/test_code_coverage_component.yml Fixed

jainprad changed the title ~~feat(coverage): enable coverage test runs and report generation~~ TheRock : Code Coverage - enable coverage test runs and report generation Sep 29, 2026

jainprad mentioned this pull request Sep 29, 2026

TheRock : Code Coverage - add code coverage CI workflows #8605

Open

1 task done

endurthiabhilash requested a review from ScottTodd October 5, 2026 15:44

jainprad marked this pull request as ready for review October 5, 2026 15:46

jainprad requested a review from endurthiabhilash October 5, 2026 15:48

endurthiabhilash mentioned this pull request Oct 5, 2026

TheRock : Code Coverage #7783

Open

1 task done

endurthiabhilash requested a review from reboss October 6, 2026 19:00

ScottTodd reviewed Oct 7, 2026

View reviewed changes

jainprad mentioned this pull request Oct 16, 2026

TheRock : Code Coverage - run code coverage on pull requests #9320

Draft

1 task done

endurthiabhilash requested a review from madkasul October 23, 2026 18:15

Merge branch 'main' into users/jainprad/coverage-test-enablement

3afbe64

jainprad added 3 commits October 27, 2026 04:35

Merge branch 'main' into users/jainprad/coverage-test-enablement

926f498

Merge branch 'main' into users/jainprad/coverage-test-enablement

097ac0a

Merge branch 'main' into users/jainprad/coverage-test-enablement

70a8818

endurthiabhilash reviewed Oct 29, 2026

View reviewed changes

jainprad and others added 2 commits October 29, 2026 11:06

Merge branch 'main' into users/jainprad/coverage-test-enablement

8161f09

Navigation Menu

Repository navigation

TheRock : Code Coverage - enable coverage test runs and report generation - #8602

TheRock : Code Coverage - enable coverage test runs and report generation#8602
jainprad wants to merge 7 commits into
mainfrom
users/jainprad/coverage-test-enablement

jainprad commented Sep 29, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 29, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 29, 2026

Uh oh!

Uh oh!

ScottTodd left a comment

Uh oh!

endurthiabhilash commented Oct 23, 2026

Uh oh!

jainprad commented Oct 26, 2026

Uh oh!

endurthiabhilash Oct 29, 2026 •

edited

Loading

Uh oh!

jainprad Oct 29, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jainprad commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Checks Passed — Ready for Review

Uh oh!

therock-pr-bot Bot commented Sep 29, 2026

Uh oh!

Uh oh!

ScottTodd left a comment

Choose a reason for hiding this comment

Uh oh!

endurthiabhilash commented Oct 23, 2026

Uh oh!

jainprad commented Oct 26, 2026

Uh oh!

endurthiabhilash Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jainprad Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jainprad commented Sep 29, 2026 •

edited

Loading

therock-pr-bot Bot commented Sep 29, 2026 •

edited

Loading

endurthiabhilash Oct 29, 2026 •

edited

Loading

jainprad Oct 29, 2026 •

edited

Loading