Repository navigation
Conversation
…verage-workflows-therock # Conflicts: # .github/workflows/multi_arch_build_portable_linux.yml
…verage-workflows-therock # Conflicts: # .github/workflows/multi_arch_build_portable_linux.yml
## Motivation
Nightly code-coverage runs need to test a normal (non-instrumented)
TheRock build in which only specific components (e.g. rocBLAS, rocSOLVER)
are swapped for their instrumented counterparts, then collect llvm
profraw files from the test run. This lets coverage be measured for a
targeted component without paying the cost of a fully instrumented build.
## Technical Details
- Add `.github/workflows/test_code_coverage_component.yml`, a reusable
(`workflow_call`) job mirroring the standard component test workflow with
coverage-specific additions:
- Installs the coverage build via `install_rocm_code_coverage_build.py`
instead of `install_rocm_from_artifacts.py`.
- `coverage_enabled` input sets `LLVM_PROFILE_FILE` (with `%p`/`%m` so
forked/multi-lib shards don't overwrite profraw files) and uploads the
profraw as a workflow artifact, keyed by `coverage_artifact_prefix`.
- Uploads coverage even on test failure (partial shard data is still useful).
- Add `build_tools/install_rocm_code_coverage_build.py`:
- Installs the generic build via `install_rocm_from_artifacts.main`, then
downloads instrumented replacement artifacts from the code-coverage run.
- `--replace-rocblas` / `--replace-rocsolver` map to the `blas` / `solver`
artifacts; only library paths matching the keyword are swapped, leaving
co-packaged libs (e.g. hipBLAS) untouched.
- Flattens replacement archive members (file/dir/symlink/hardlink) into the
install tree, preserving exec bits and link structure.
Review feedback asked for coverage to be an add-on that reuses the existing build process, rather than a feature the shared pipeline has to know about. multi_arch_build_portable_linux.yml is restored to its state on main, dropping the build_stages, stage_projects, and extra_cmake_options inputs along with the per-stage `if` rework that left the file with duplicate `if` keys and made it invalid YAML. The coverage workflows now call multi_arch_build_portable_linux_artifacts.yml once per stage they actually need, so no stage filtering is required anywhere. The nightly builds compiler-runtime and then math-libs as explicit jobs, which makes configure_coverage_ci.py's build_stages output unnecessary. The math-libs stage is deliberately not narrowed with stage_projects: its test jobs fetch artifacts by this run's id and so need the stage's full dependency set, and coverage_cmake_options already decides which projects get instrumented. COMPILER_RT_BUILD_PROFILE_ROCM is dropped from compiler/CMakeLists.txt, from the coverage workflows, and from the docs, since the ROCm profiling runtime is expected to come from the compiler build rather than from a TheRock level option. Co-authored-by: Cursor <cursoragent@cursor.com>
Implements the nightly half of RFC0014: option C for scheduling and option B for artifacts, on the phase 1 scope of every onboarded project on the single default architecture (gfx942 / gfx94X-dcgpu). The regular nightly now dispatches multi_arch_ci_coverage_nightly.yml once its build stage finishes, passing its own run id as baseline_run_id. The cron on the coverage workflow is gone: it would have produced runs with no baseline. Coverage keeps its own run id, so a coverage failure leaves the nightly's status alone and the nightly's unstable test jobs do not gate it, while the baseline it tests against is still fresh from the same commit. The dispatch is gated on the COVERAGE_NIGHTLY_ENABLED repository variable and on the nightly channel. Instrumenting the whole stack in one build and then reporting per project means a project's dependencies would otherwise be instrumented too, which both writes stray profiles and moves the coverage denominator whenever a dependency changes. Test jobs therefore install the baseline run in full and overlay only the project under test from the coverage run. overlay_coverage_artifacts.py does the overlay; it works at the subproject stage directory level rather than the artifact level because TheRock's artifacts are grouped (rand carries both rocRAND and hipRAND), and fails rather than silently reporting coverage for a non-instrumented build. The baseline is read from its own release channel, since artifacts are bucketed per channel. configure_coverage_ci.py grows the artifact_names and artifact_relpaths the overlay needs, and now collapses a whole-group selection to THEROCK_COVERAGE_ROCM_LIBRARIES_ALL / _ROCM_SYSTEMS_ALL / THEROCK_COVERAGE_ALL, which is how the RFC describes the nightly full-stack build. quartz_tracking_id is threaded through the coverage workflows, which the new dispatch edge pulls into the release graph.
Coverage now has a single entry point, the nightly workflow dispatched downstream from the regular nightly. The label-triggered PR workflow and the per-project instrumented rebuild it needed are removed, which also takes the stage_projects hook back out of the shared per-stage build workflow. multi_arch_ci_coverage_linux.yml keeps only the test, merge, and report half; the instrumented stack is always built by its caller. The registry fields that existed solely to drive the per-project rebuild (stage_project, and the coverage_flag / build_stage matrix keys) go with it. Co-authored-by: Cursor <cursoragent@cursor.com>
code_coverage.md covers the CMake options and onboarding a project, but not how a nightly report is actually produced. This adds the CI walkthrough: the dispatch and its gates, the instrumented build, how the baseline run id reaches the test jobs, the artifact overlay, and report generation. Linked from code_coverage.md, ci_overview.md, and the docs index. Co-authored-by: Cursor <cursoragent@cursor.com>
The RFC's option C, where the regular nightly dispatches coverage and hands over its run id, lands separately. This drops the trigger_coverage_nightly job, so multi_arch_release_linux.yml is untouched by this PR and coverage is started by hand with a baseline_run_id from a recent nightly (the RFC's option B1). Nothing downstream changes: baseline_run_id and baseline_release_type are already inputs, so adding the trigger later only means adding the dispatching job. Co-authored-by: Cursor <cursoragent@cursor.com>
The phase 1 work exists to show that an instrumented build and a coverage report work end to end, and it currently cannot show that. multi_arch_ci_coverage_nightly.yml is triggered by hand, but a workflow_dispatch is only offered once the file is on the default branch, so there is no way to run it before it merges and no way to see it in this PR's checks. Merging it unvalidated to make it validatable is the wrong order. A caller is not subject to that restriction. Paths under ./.github/workflows resolve against the caller's own commit, so a pull_request-triggered caller runs this branch's copy of the coverage workflow and reports as jobs of this PR. The coverage workflow therefore gains a workflow_call trigger, duplicating the dispatch inputs since Actions does not expand YAML anchors. The baseline is pinned rather than resolved, so that a change in the report means a change in this branch: Multi-Arch CI run 33990288042, whose Stage - Compiler Runtime and Stage - Math Libs (gfx94X-dcgpu) both succeeded. That run's own test jobs failed, which does not matter here, as only its build stages are consumed. It published to therock-ci-artifacts rather than the nightly bucket, so baseline_release_type is ci and not the default; the bucket for a run follows the release type it published under. A ROCm/TheRock run rather than a rockrel release nightly. This workflow only installs the baseline at test time and never configures against it, so the embedded artifact workspace paths that rule out a cross-repository baseline elsewhere do not apply, but keeping the baseline in this repository is one fewer variable while the instrumented build is still unproven. The caller also has to grant id-token and actions:read itself, because a called workflow cannot hold permissions its caller does not. The paths filter is deliberately narrow and concurrency cancels in progress: each run is a multi-hour instrumented build plus a full test suite, and it is not worth spending one on code that has already been superseded. This file is scaffolding and is not part of the design. Coverage on presubmit is precisely what phase 1 rejected, and leaving this in place would charge that cost to every PR touching coverage. Delete it, and the workflow_call trigger, once the run has gone green. Co-authored-by: Cursor <cursoragent@cursor.com>
The nightly coverage run built an instrumented stack and then, at test time, threw most of it away: each job installed a regular nightly's artifacts in full and overlaid only the project under test, so the project's dependencies stayed non-instrumented. That is unnecessary. A coverage run builds every stage its selection needs and instruments only the projects the selection names, so everything those projects depend on is already built normally in the same run. Its own artifacts are the stack to test against. The isolation the overlay was protecting comes from somewhere else anyway. merge_coverage_report.py hands llvm-cov an explicit -object per binary, and llvm-cov only reports functions found in those objects' coverage mappings, so counters written by a sibling project are never looked up. A selection covering several projects still yields one report per project. Removing it also removes the failure mode it introduced. Run 34098007786 died in the overlay step, where shutil.copytree could not replace an existing libhiprand.so -> .so.1 -> .so.1.1 chain that the baseline install had already put in place. Drops the script and its tests, the baseline_run_id and baseline_release_type inputs threaded through four workflows, and the --artifact-names flag added to artifact_manager.py for the overlay's sake alone. Co-authored-by: Cursor <cursoragent@cursor.com>
THEROCK_COVERAGE_* set <PROJECT>_ENABLE_COVERAGE and therock_subproject.cmake passed that name straight through to the subproject. Almost no project implements it. Of the projects registered here only hipBLASLt, hipSPARSE and hipDNN use that spelling; the rest use BUILD_CODE_COVERAGE, CODE_COVERAGE or ENABLE_CODE_COVERAGE. hipRAND, the only project the registry had until now, gates its instrumentation on BUILD_CODE_COVERAGE. So -DHIPRAND_ENABLE_COVERAGE=ON was forwarded, ignored, and the build produced uninstrumented binaries. The tests would have written no profraw files and the report job would have failed on finding none. Run 34098007786 never got that far. Registry entries now name the option their project actually implements, and therock_subproject.cmake translates TheRock's flag into it. The translation is per subproject on purpose: BUILD_CODE_COVERAGE and CODE_COVERAGE are generic names, so defining either globally would instrument everything in the build that understands it, which is what per-project coverage exists to avoid. Also expands the registry to the 27 projects across rocm-libraries and rocm-systems, so a run can measure a chosen subset, and makes the nightly build only the stages that subset needs. Eight of those cannot be measured by this pipeline. hipSPARSE, rocALUTION and rocprofiler-sdk instrument with gcov, which writes .gcda files rather than the .profraw llvm-profdata merges; MIOpen, aqlprofile, amdsmi, rocprofiler-compute and rocprofiler-systems have no build-time coverage option at all. They stay registered so the gap is recorded, but carry an unsupported_reason instead of an option name, stay out of the group aliases, and are rejected with that reason when named rather than failing hours later with an empty report. Option names verified against the pinned rocm-libraries and rocm-systems commits. Co-authored-by: Cursor <cursoragent@cursor.com>
Phase 1 instruments the whole stack in one build but has to report per project, so a test job installs a non-instrumented baseline run and overlays only the project under test. That path was removed in 0585dc3 on the argument that object_globs already scope the report. They scope the export, but not what gets measured: inline and template code from headers is emitted linkonce_odr into every binary that uses it, so an instrumented sibling writes counters that llvm-profdata merges into this project's and no -object choice separates again. It is worst for the header-only projects (rocPRIM, hipCUB, rocThrust, rocWMMA), where nearly all the measured code lives in headers. This reverts that commit and restores baseline_run_id as the RFC specifies. The overlay crashed in run 34098007786 because shutil.copytree recreates symlinks with os.symlink(), which raises once the baseline install has already placed the libfoo.so -> libfoo.so.1 chain -- dirs_exist_ok forgives directory collisions, not symlink ones. Replaced with a merge_tree() that unlinks what it is about to replace, covered by two regression tests. The PR check now pins ROCm/rockrel 34068476531, a real Multi-Arch Release (nightly), and selects every project rather than one. Unlike PR #7944, which copies baseline stages into its own run and builds on top of them, nothing here configures CMake against the baseline, so the embedded /__w/rockrel/rockrel paths that rule out release nightlies there do not matter: the baseline is only installed on the test runner, where prebuilt binaries run and no package config is read. Selecting everything exposed two gaps a single-project run could not: - comm-libs takes hipify and rocjitsu from emulation, which the coverage workflow never built, so selecting rccl or rocshmem would have failed on a missing inbound artifact. build_stage_plan now derives the required stages transitively from the build topology and orders them into waves, and the workflow builds one wave per job. - The artifacts bucket lookup resolves the run through the GitHub API against this repository, which 404s for a rockrel id. Plumbed a RUN_GITHUB_REPO passthrough down to install_rocm_from_artifacts.py. Also added a test asserting every registry entry names the stage the topology says builds its artifacts. The registry duplicates that fact with nothing keeping the two in step, and a project moving upstream would otherwise surface as a stage never built and an overlay that finds nothing, hours into a run. Co-authored-by: Cursor <cursoragent@cursor.com>
Run 34119302776 selected every onboarded project and found the stack is
not ready to be instrumented all at once:
- libclang_rt.profile_rocm.a is linked without -ldl or -lpthread, so
an instrumented executable cannot resolve dlsym, dladdr or
pthread_once. Shared libraries tolerate the undefined symbols, which
is why this stayed hidden until projects that build their own
binaries were selected: rocRAND's tools, hipCUB's and rocPRIM's
tests.
- rccl's CMake refuses coverage outside a Debug build, and TheRock
builds Release.
- rocshmem does not survive coverage together with HIP relocatable
device code: "unknown special variable with appending linkage:
llvm.global_ctors".
Narrow to hipRAND, which built cleanly instrumented in 34098007786, so
the baseline install and the per-project overlay can be proven on their
own. The rocm-systems projects wait on the second and third problem, the
rest of math-libs on the first; the selection widens once they are
fixed. Nothing about the workflow changes -- it still resolves the whole
registry when asked, and the emulation/comm-libs wave jobs stay in place
for when rccl and rocshmem come back.
Co-authored-by: Cursor <cursoragent@cursor.com>
-fprofile-instr-generate links libclang_rt.profile_rocm.a, whose InstrProfilingPlatformROCm.cpp calls dlsym, dladdr and pthread_once. The driver puts neither -ldl nor -lpthread on the link line, and the manylinux base still keeps those symbols in libdl and libpthread rather than in libc, so an instrumented shared library ends up with undefined references. The library itself links, since shared objects tolerate that, and the failure surfaces one step later when an executable links against it: ld.lld: error: undefined reference: dlsym >>> referenced by library/libhiprand.so.1.1 >>> (disallowed by --no-allow-shlib-undefined) This blocked every instrumented project, not just the ones building their own executables, and is why run 34126836356 failed on hipRAND alone. It stayed hidden until 1fa0945 made the coverage flag actually reach the subproject; before that hipRAND was building uninstrumented. Add the two libraries to the subproject's linker flags alongside the coverage option. The flags go through the project init file, which appends to CMAKE_<TYPE>_LINKER_FLAGS and so lands ahead of the objects that need the symbols; push-state/--no-as-needed keeps the DT_NEEDED entries from being dropped on that account, and pop-state keeps the relaxation from applying to anything else on the line. Verified the mechanism outside ROCm, reproducing the shape of the failure: a static archive with an unresolved reference linked into a shared library, an executable linking against it under --no-allow-shlib-undefined, and the same init-file flags fixing it and recording the expected DT_NEEDED. Co-authored-by: Cursor <cursoragent@cursor.com>
The staged build jobs existed for one dependency edge: comm-libs takes hipify and rocjitsu from emulation, so selecting rccl or rocshmem had to build emulation first. Those two projects are the only consumers, and both are blocked upstream -- rccl's CMake refuses coverage outside a Debug build, and rocshmem fails on coverage plus HIP relocatable device code. The other 17 measurable projects are all in math-libs, so the wave jobs were carrying no traffic. Drop them, along with the topology walk and wave ordering in configure_coverage_ci.py. build_stage_plan() becomes resolve_build_stages(), which checks the selection against BUILDABLE_STAGES and reports whether math-libs is needed. Naming an unbuildable project is now an error at matrix time. Without it nothing would be built for that project and the failure would not surface until the test job found no instrumented files to overlay, hours into the run. An empty selection meant every measurable project, which would now reject itself, so it narrows to the buildable set. That applies to the default only; an explicit request is refused rather than reduced. The ordering constraint is recorded under "Adding a stage" in docs/development/code_coverage.md for whoever restores comm-libs. Co-authored-by: Cursor <cursoragent@cursor.com>
Run 34135220913 proved the path end to end on hipRAND once the profile runtime's own dependencies were added to the link line: instrumented build, baseline install, overlay, tests, and a 77% line report. The narrowing to a single project has served its purpose. Select the 13 rocm-libraries projects that build a shared library. The four header-only ones are left out: they have no library of their own to instrument, so they report against instrumented test binaries, and their inline code is the linkonce_odr case the overlay cannot fully isolate. That is worth doing, but worth doing on its own. rccl and rocshmem are not selectable at all now that comm-libs has no build job, and both are blocked upstream regardless. hipBLASLt is included deliberately despite failing to link hipblaslt-test with "unable to find library -lrocroller" in run 34119302776. Whether that is coverage-related was never established, and this is how we find out. Co-authored-by: Cursor <cursoragent@cursor.com>
Run 34159520920 answered the question the last commit set out to ask: the hipBLASLt link failure is caused by coverage, and the cause is upstream. The coverage-only branch of hipblaslt's clients/CMakeLists.txt links hipblaslt-test against a bare `rocroller` while the lines around it use imported targets like roc::tensilelite-host. rocRoller exports under roc::, and in TheRock it is a separate subproject consumed through its package config, so `rocroller` is not a target in hipBLASLt's build and survives to the linker as -lrocroller with no -L to resolve it. hipBLASLt's standalone build has rocRoller in-tree and so never notices. Nothing in TheRock can fix this: rocm-libraries is a pinned submodule and patches/ only covers third-party. Confirmed against the non-coverage build of the same commit, which succeeds and never links rocroller into hipblaslt-test at all, because the whole block is behind HIPBLASLT_ENABLE_COVERAGE. Rather than only dropping it from the PR check, which would leave the nightly's own default selection broken, record it as a blocked_reason. That is a new registry field for a project that is measurable but whose instrumented build fails, as distinct from unsupported_reason, which means it cannot be measured at all and is not expected to change. A blocked project leaves the default selection and every group alias, including the group lists --emit-cmake writes, so a group build cannot set a flag the Python side just declined to. It stays selectable by name, with the reason printed as a warning: rerunning it is the only way to find out the block has been lifted. Co-authored-by: Cursor <cursoragent@cursor.com>
Run 34164420766 built all twelve projects and reported six of them. The six failures were three separate problems. Keep the instrumentation off the GPU. The upstream coverage options pass -fprofile-instr-generate and -fcoverage-mapping unqualified and the HIP driver forwards both to the device compilation, so kernels were instrumented as well. rocSPARSE showed what that costs: all 22683 tests pass, then the HIP runtime aborts with "Cannot create GlobalVar Obj for symbol: __profd_..." on a rocPRIM trampoline kernel, because libclang_rt.profile_rocm looks the device profile symbols up with hipModuleGetGlobal at exit. rocBLAS and rocSOLVER are the same story without the diagnostic: 101618 and 20514 tests pass, then a segfault where the profile is written. All four also wrote a device profile named gfx942:sramecc+:xnack-.0.<host name>, which upload-artifact rejects for the colons. -Xarch_device confines the negation to the device compilation. Checked against the ROCm clang in this tree: with it, the device image loses __profd_, __profc_ and the per-TU __llvm_profile_sections_ symbol the runtime enumerates, while the host keeps -fprofile-instrument=clang and its coverage mapping. Order is what makes this work, so it goes on the compile rule after <FLAGS> rather than in CMAKE_<LANG>_FLAGS, which the project's own COMPILE_OPTIONS would otherwise override. The rule is not set yet in CMAKE_PROJECT_TOP_LEVEL_INCLUDES, so it needs its own file passed as CMAKE_PROJECT_INCLUDE. hipDNN's object glob named a library that does not exist. Its shared library is libhipdnn_backend.so, as TheRock's own packaging metadata already recorded. Profiles were being collected the whole time and thrown away for want of anything to attribute them to. hipTensor's option is HIPTENSOR_CODE_COVERAGE, not the bare CODE_COVERAGE its neighbour rocALUTION uses. Passing the wrong name is silent, which is exactly how this got missed: the library built uninstrumented and the tests passed having written no profile at all. Finally, give coverage runs four times the test budget. Instrumented builds are -O0 -g and count every branch, so rocRAND tests that normally take milliseconds took 25 to 60 seconds each and the shard hit the 15 minute wall a long way from finishing. Co-authored-by: Cursor <cursoragent@cursor.com>
timeout-minutes takes a number and the env context hands back strings, so the step needs fromJSON around it. actionlint catches this; the value is still computed once in the job env rather than twice at the two use sites. mdformat cannot wrap inside a code span, so the flag list was left as one overlong line. It reads better as a block anyway. Also records in the PR check what run 34164420766 established, since the project list there is otherwise hard to account for. Co-authored-by: Cursor <cursoragent@cursor.com>
…model Correcting hipTensor's option name to HIPTENSOR_CODE_COVERAGE instrumented it for the first time, and it does not link: libhiptensor.so grows past the 2GB the small code model allows and ld.lld fails on out-of-range R_X86_64_PC32 relocations from .rodata into .text. Upstream enables coverage as a blanket add_compile_options over a library built on composable_kernel, so the coverage mapping and name data scale with an already very large amount of instantiated template code. The option name stays correct so that retesting the block is one field away. Taking it out of the PR check list matters as much as the blocked_reason, since a named project is still honoured. A stage is built as a unit, so this one link failure failed math-libs in run 34201973090 and skipped the report for all twelve. Co-authored-by: Cursor <cursoragent@cursor.com>
The split put the same explanations in two places. Both documents described the option-name translation, the host-only scoping, the baseline-and-overlay arrangement and the object_globs export scoping, and code_coverage.md had grown two separate accounts of host-only instrumentation against each other. Nothing kept them in step. code_coverage.md now runs local build, local report, then CI in one order, with each mechanism explained where it first appears and referred back to afterwards. The CI half keeps everything the flow document carried, including the job graph. Readers who arrived via the Infrastructure index or ci_overview.md now land on the Coverage CI section rather than a separate file. Co-authored-by: Cursor <cursoragent@cursor.com>
code_coverage.md answers "what are the options and how do I onboard a project", which is the wrong shape for someone meeting this pipeline for the first time. This is the narrative version: why two builds are involved, what the overlay is for, and what each of the eight files in the critical path contributes, followed by an appendix covering every file in the change. The two link to each other rather than repeating; the reference keeps the mechanisms and this keeps the story. Co-authored-by: Cursor <cursoragent@cursor.com>
The report job produced coverage.info and nothing else. That is the format Codecov consumes, not one anybody reads: answering "what is the number" meant opening Codecov, and there was no line-by-line view attached to the run at all. merge_coverage_report.py now renders the one profdata index three ways. llvm-cov report gives coverage_summary.txt, the per-file table with the totals on the last line, and llvm-cov show gives html/index.html, source annotated line by line and C++ names demangled. All three go into the existing coverage-report-<project>-<family> artifact, and the totals are also written to the job's summary page so the number is visible without downloading anything. Only the HTML opens source files; lcov and the table are produced from the coverage mappings alone. The report job checks out TheRock, not the projects it measures, so it now runs fetch_sources.py for the stage that built the project, which the registry already records and which is threaded through the matrix as build_stage. Paths are recorded as the build container saw them, so --path-equivalence maps that prefix onto the workspace. That fetch is continue-on-error and runs after the artifacts are already down. It is the only step here that can fail for reasons of its own -- a large shallow checkout on a runner that has just downloaded an instrumented stack -- and losing it should cost the annotated view rather than the whole report. Files generated into the build tree, version headers and the like, exist in no checkout and show as uncovered in the HTML; the text table still counts them. Co-authored-by: Cursor <cursoragent@cursor.com>
The baseline nightly is optional, and passing one is precisely the case where the coverage run does reuse the regular build, so describing the run as not reusing it was backwards. The report job also emits a text summary and an HTML report now, not just the lcov file. Also list the coverage docs in Further Reading, which had been the only body reference missing from those lists. Co-authored-by: Cursor <cursoragent@cursor.com>
THEROCK_FLAG_KPACK_SPLIT_ARTIFACTS is being deprecated, so the coverage build jobs no longer opt out of split packaging. The compiler-runtime job carried nothing else in extra_cmake_options, so that input goes away there entirely. Drops the note telling local builds to pass the same flag. It already recorded that host-only instrumentation likely made the opt-out unnecessary; pointing readers at a deprecated option is worse than leaving the caveat unstated. Co-authored-by: Cursor <cursoragent@cursor.com>
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
…rkflow The merge of #7783 and #7828 left two implementations of the same job. multi_arch_ci_coverage_linux.yml called test_component.yml, whose coverage support installs a baseline run and overlays the project under test with overlay_coverage_artifacts.py, while test_code_coverage_component.yml and install_rocm_code_coverage_build.py did the same thing by swapping the component's files into a generic install -- and nothing called them. Point the test job at test_code_coverage_component.yml, so the installer path is the one that runs. baseline_run_id, baseline_release_type and baseline_repository map onto generic_run_id, generic_release_type and generic_repository; the component to swap is passed explicitly as the registry key rather than left to the test matrix's job_name, because that is what COMPONENT_MAP is keyed on. The installer resolved both runs from one RELEASE_TYPE and one --run-github-repo, which cannot describe the nightly: coverage publishes to ci while the baseline is a nightly, and a release nightly comes from ROCm/rockrel rather than from here. Whichever of the two lookups did not match would have read the wrong bucket. It now takes the coverage run's channel and repository separately, and create_backend_from_env grows a release_type override to make that expressible. Also carries over two things the copy was missing: the coverage test timeout floor, without which an instrumented shard hits the 15 minute wall long before finishing, and timeout-minutes on its two checkout steps, which workflow_step_timeouts_test.py requires. A test now asserts every selectable coverage project has a COMPONENT_MAP entry naming an artifact the registry agrees on. The two lists have to stay in step for --replace-<project> to resolve, and nothing else keeps them that way. Co-authored-by: Cursor <cursoragent@cursor.com>
The filter still named test_component.yml and overlay_coverage_artifacts.py after the test job moved to test_code_coverage_component.yml and the installer. A PR touching only the installer or the coverage component workflow would not have triggered the one check that exercises them, while an unrelated edit to test_component.yml would still have spent a multi-hour instrumented build. The header-only exclusion note names the component swap rather than the overlay for the same reason. The run descriptions above it are left alone: those runs did use the overlay. Co-authored-by: Cursor <cursoragent@cursor.com>
The last assertion in test_html_renders_into_a_directory compared against a literal "/llvm/llvm-cxxfilt" while write_html interpolates a Path, which renders "\llvm\llvm-cxxfilt" on Windows. The Unit Tests :: windows-2022 job has been failing on it since the HTML report was added; 45d0240 fixed the other assertions in this file the same way, before this one existed. Interpolate the same Path the call is given, as the -output-dir assertion two lines up already does. Co-authored-by: Cursor <cursoragent@cursor.com>
…ponent Every test shard in run 35081996415 failed on the first line of the install step with "pip3: command not found", so nothing downstream of it ran. The coverage component workflow opened with a bare `pip3 install -r requirements-test.txt`, which assumes pip3 is on PATH in the test container. It is not: test_component.yml gets Python, uv, the venv and the test requirements from the setup_test_environment action, and that half was dropped when this workflow was written. Nothing called it until now, so the gap had no way to surface. Use the action here too. It also installs the generic non-instrumented build, which is what install_rocm_code_coverage_build.py was doing by delegating to install_rocm_from_artifacts, and it already takes the run id, the channel and the owning repository the baseline needs. The installer grows --skip-generic-install and is left with the component swap alone. Reusing it rather than repeating its four setup steps keeps the Python version, the uv install and the venv activation in one place, and is what the review on #7783 asked for: coverage as an add-on to the existing process rather than a parallel copy of it. Co-authored-by: Cursor <cursoragent@cursor.com>
…outs Six of the eleven projects in run 35092904145 ran no tests at all: rocBLAS, rocSPARSE, hipSPARSELt, hipBLAS, rocFFT and hipFFT each died with "libomp.so: cannot open shared object file" before the first case. They are the projects whose test clients link OpenMP. The baseline was the cause, not the swap. Comparing rocsparse-test from both runs, the baseline's copy carries $ORIGIN/../lib/llvm/lib/x86_64-unknown-linux-gnu in its RUNPATH and the branch's copy does not, because the host LLVM install moved off LLVM_ENABLE_PER_TARGET_RUNTIME_DIR between the two. libomp.so accordingly sits in lib/llvm/lib/<triple>/ in rockrel 34068476531 (2026-09-07) and in lib/llvm/lib/ in this branch's build. Each binary is correct against its own tree; the swap puts one in the other. Re-pin to rockrel 35038162281, "Multi-Arch Release (nightly) | Linux: all" from 2026-09-16, whose libomp.so is in lib/llvm/lib and which publishes every artifact the selection needs. Being days rather than weeks from this branch should also settle the other three skew failures: hipRAND's version assertion, rocSOLVER's missing rocPRIM trampoline_kernel symbol, and rocRAND's abort inside the baseline's older clr. Then stop the layout from mattering. The test job now puts lib/llvm/lib and any triple subdirectory holding a libomp.so on LD_LIBRARY_PATH, so a client resolves it under either layout. pytest_runner.py already does this, for the same reason. test_runner.py inherits the environment and preserves LD_LIBRARY_PATH, so it reaches ctest. Re-pinning alone would leave the next layout change to be discovered hours into a run; the library path alone would fix nothing but libomp. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The nightly code coverage work is split across two branches that are reviewed
separately but have to run as one pipeline. This branch carries both so the
combined flow can be built, run and reviewed in one place.
users/jainprad/codecoverage-workflows-therock: the nightly coverageentry point, the instrumented build, the CMake coverage options and the
merge/report tooling.
users/madkasul/code-coverage-nightly: the coverage component testworkflow and the installer that swaps instrumented components into an
otherwise normal TheRock install.
ISSUE ID : #7783
Technical Details
users/jainprad/codecoverage-mergeis branched from #7783's head(
00edba1) with #7828's head (1436b8c) merged into it. The two branchestouch disjoint files, so the merge is clean and needed no conflict resolution:
#7828 only adds
.github/workflows/test_code_coverage_component.ymlandbuild_tools/install_rocm_code_coverage_build.py. No submodule pointers move.The diff against
mainis exactly the union of the two branches' diffs.From #7783:
.github/workflows/multi_arch_ci_coverage_nightly.yml,multi_arch_ci_coverage_linux.yml— the nightly coverage entry point and theper-project test/merge/report half.
.github/workflows/coverage_nightly_pr_check.yml— temporary scaffolding torun the coverage workflow from a branch; to be deleted before landing on
main, along with theworkflow_calltrigger it relies on.CMakeLists.txt,cmake/therock_subproject.cmake— coverage group options,per-subproject option translation, host-only instrumentation and the profile
runtime's link dependencies.
build_tools/github_actions/configure_coverage_ci.py,merge_coverage_report.py,overlay_coverage_artifacts.pyand their tests —the coverage registry and matrix, lcov/HTML/text report generation, and the
baseline overlay.
docs/development/code_coverage.mdand related doc updates.From #7828:
.github/workflows/test_code_coverage_component.yml— a reusable componenttest job that installs the coverage build, sets
LLVM_PROFILE_FILEanduploads the profraw files even when tests fail.
build_tools/install_rocm_code_coverage_build.py— installs a generic build,then replaces only the named components with their instrumented counterparts.
Test Plan
Nothing new is being written here, so the value is in the combined pipeline
running: the coverage nightly workflow off this branch, exercising the
component test workflow and installer from #7828 against the instrumented
build from #7783.
Test Result
Pending. This is a draft opened so the combined branch can be run and reviewed.
Submission Checklist