[danielbolsson]

Motivation

HRR CI checks the HIP Record and Replay code generator, its host-only unit tests on Linux and Windows, and its capture/replay integration suite. The suite policy stays in projects/hrr. By default the integration suite runs under rocJitsu (emulation/rocjitsu) on a CPU runner. Real GPU runners are a manual choice.

Technical Details

• .github/workflows/hrr-ci.yml is the entry point. Triggers: pull requests, pushes to develop, a nightly schedule, and workflow_dispatch (families, backend, ignore_test_config). backend is simulator or gpu and defaults to simulator.
  • Pull requests and pushes go through a paths filter: the union of the ci.yaml paths.cpu, paths.capture and paths.suite lists, less paths.excluded. A pull request that touches no HRR path starts no run. test_hrr_ci_configure.py fails when the filter and ci.yaml disagree. So HRR CI Summary must not become a required check while the filter is in place.
  • configure runs hrr_ci_configure.py, which applies projects/hrr/tests/config/ci.yaml to the changed files (renames included) and the labels:
    • the CPU checks when HRR paths change;
    • the capture/replay suite when a capture path or paths.suite changes, the PR has the ci:hrr-capture label (and touches an HRR path), nightly, and on dispatch. Fork PRs, and PRs whose head repository is gone, get the CPU checks and the rocJitsu jobs, not the real-GPU backend (the build job runs for them too). Adding any other label cancels nothing and runs no HRR job;
    • workflow_dispatch uses its backend input. Any other event uses backend.default (simulator), unless the event is listed in backend.gpu.triggers (empty on develop);
    • the simulator jobs are simulator.default_families: seven rocJitsu families (gfx90a, gfx94x, gfx950, gfx110x, gfx1151, gfx120x, gfx125x), each with its emulation/rocjitsu/configs/*.json, on ubuntu-24.04 inside TheRock's no_rocm test image, in local mode (rocjitsu --launcher). gfx103x, gfx1150 and gfx1153 have no rocJitsu config. The dispatch families input replaces the list. ci:gfx... labels are TheRock's and do not select an HRR family. A simulator family with no config fails configure. gfx125x stays enabled: false for real GPUs; the simulator list does not consult that flag;
    • the GPU backend runs the enabled families on TheRock runner labels. Only gfx94x is enabled. gfx950 takes the 8-GPU runner only on schedule. A draft that should hit hardware on every push sets backend.gpu.triggers: [pull_request] and families.<key>.enabled: true in that branch's ci.yaml;
    • configure fails unless targets.arch of integration.yaml covers every GPU target of every family in ci.yaml.
  • api-checks ("API IDs and workloads") runs hrr_check_sources.py: API-ID compatibility with the base commit, and the API matrix.
  • scripts lints the suite configs, which must agree on targets.arch, and runs the HRR helper-script tests.
  • unit builds and runs the host-only unit suite on Ubuntu and Windows against the ROCm SDK version pinned in ci.yaml, with this tree's HIP headers. Those binaries do not run under rocJitsu.
  • gpu calls hrr-gpu.yml.
  • HRR CI Summary writes one table of every platform and every ci.yaml family, with the reason for each family that did not run (hrr_ci_report.py). The verdict comes from toJSON(needs) and the JUnit files, not from the jobs API. It fails when configure fails, when a needed job is failure or cancelled, when a job configure selected was skipped, when a listed job is still running, or when a JUnit file has a failed case. In a label run that runs no HRR job it reports the newest other run on the PR head that has a verdict, and fails unless that run passed, so a label cannot turn a failing HRR CI green.
• .github/workflows/hrr-gpu.yml (workflow_call):
  • build runs in TheRock's build image. It installs the artifacts of a recent rocm-systems TheRock CI run, then builds CLR with HRR capture compiled in from this tree, plus hrr-playback and the integration suite for targets.arch of integration.yaml. There is no capture-build cache: one was dropped on 2026-10-21 because a cache hit failed Verify with librocm_kpack.so.0: cannot open shared object file (cause unknown). The same-run handoff to the test jobs is hrr-capture-<run_id> and hrr-base-<run_id>.
  • Build rocJitsu builds the rocJitsu CLI from emulation/rocjitsu once and shares it with the family jobs, which takes a run from about 142 runner minutes to about 23.
  • rocJitsu runs one ubuntu-24.04 job per simulator family. It downloads that rocJitsu build and runs run_catch2.py with --launcher (local mode) so case listing still uses hrr-integration-tests directly. --daemon is not used: it is broken (issues [rocjitsu] Daemon mode: hipMemCreate returns hipErrorOutOfMemory, the same call succeeds in local mode #13721, [rocjitsu] Daemon mode on gfx90a: HRR replay spins at 100% CPU and never finishes (SET_EVENT_MISS) #13722). --arch is the family target, not auto. Cases that hit rocJitsu issues [rocjitsu] Multi-GPU configs: replay at fixed addresses hits "DRM files currently share one simulated GPUVM" and reads wrong data #13723, [rocjitsu] RDNA3 and RDNA4 configs: hipMemcpyToArray and hipMemcpyFromArray succeed but copy no data #13724, [rocjitsu] Local mode: the application's RLIMIT_FSIZE kills the simulated GPU with SIGXFSZ #13726 and [rocjitsu] hipIpcGetMemHandle fails with HSA_STATUS_ERROR since #13035 (shared runtime API headers) #13803 (draft fix PR fix(rocjitsu): handle legacy libc and small-runner test environments #13337) are skipped with emulator: true in integration.yaml. Unit_HRR_PinnedHostFork is also skipped on gfx950 under the emulator (#14033): the capture subprocess exited 42 on the two-GPU configuration and passed on the other families.
  • test is the real-GPU job. It runs only when gpu_families is non-empty, on TheRock's test image and container options, with --arch auto. Like TheRock's test_component.yml, it runs TheRock's cleanup_processes.sh twice: from its own checkout before the workspace checkout, and after the suite. The second run cannot fail the job.
• TheRock refs in the HRR workflows match the pin in therock-multi-arch-ci.yml.
• .github/skip-ci-config.toml lists the HRR workflows and projects/hrr/tests, tools and skills, which TheRock never reads, so a pull request that changes only those skips TheRock CI without ci:skip. Any other projects/hrr change builds every TheRock stage until the TheRock pin includes TheRock #8965, which builds hrr-playback in compiler-runtime on Linux. This PR itself still needs ci:skip: it changes skip-ci-config.toml, which TheRock does not classify.
• The local Docker runner (hrr_dev.py and projects/hrr/tests/docker/Dockerfile) is not in this pull request. It is on users/danielbolsson/hrr-local-dev-20c1, which changes only projects/hrr.

The build job and hrr_capture_build.py are still interim. ROCm/TheRock#8965 adds a HIP_HRR_CAPTURE flag and an hrr-capture build variant. The test matrix stays in projects/hrr.

Issue Tracking

JIRA ID: AIRUNTIME-2871

Test Plan

• python3 -m unittest for test_hrr_ci_configure.py, test_hrr_capture_build.py, and test_run_catch2.py. These do not start Docker or use a GPU.
• pytest projects/hrr/tests/scripts projects/hrr/tools/tests at every commit of this branch.
• The new skip-ci-config.toml patterns, checked with TheRock's load_skip_ci_patterns_for_external_repo at the pinned ref.
• HRR CI on this pull request: configure selects the seven rocJitsu families, the capture build compiles, and the simulator jobs run in the test image with no symlink step. The summary fails if any of those jobs or cases fail.
• A workflow_dispatch with backend=gpu and a chosen families list, for a real GPU run.

Test Result

The HRR CI run on the head of this branch is the test. Local unit tests: python3 -m unittest for test_hrr_ci_configure, test_hrr_ci_report, test_hrr_ci_schedule and test_hrr_capture_build pass. The Docker helper tests live on users/danielbolsson/hrr-local-dev-20c1.

Other local results:

• pytest: 336 passed at the head.
• Skip patterns: the HRR workflows and tests, tools and skills files are skippable. playback/, include/, CMakeLists.txt, CLR's HRR sources and therock-multi-arch-ci.yml are not.

Not run here: a real GPU job. An earlier revision of this PR ran the integration suite on real GPUs (gfx90a, gfx94x, gfx950, gfx110x, gfx1151, gfx120x). That run predates the simulator default.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[andraghetti]

Thanks Daniel. Splitting this out of #12778 makes sense.

I went through the diff against the version I reviewed there (447a92e). The ROCr build looks right to me. CLR picks up the in-tree install, rocr-install/lib is in the artifact, and it sits before the SDK on LD_LIBRARY_PATH. I checked the artifact: amdhip64 and the test binary both resolve one libhsa-runtime64.so.1, so we never end up with two copies. The step costs about 10 s cold.

The run on this head is green: 6 families ran the suite, including both containers, and 3 timed out in the queue as designed. The fixes for Copilot's points look good.

Could you update the description? It says the workflow is unchanged from #12778 and that CI cannot run before #12778 merges. CI already ran here (run 38233722764), and the ROCr build plus the review fixes are new since my review.

One question inline, not blocking for me. Approving. My approval does not cover .github/ as code owner, so this still needs someone from rocm-systems-reviewers.

[marbre]

Let's discuss offline if we should rather integrate into TheRock CI. Overall I feel, even if it is not integrated into TheRock, this needs to be split into separate workflows.

As you asked for DevOps feedback, allow me to point you to https://github.com/ROCm/TheRock/blob/main/docs/development/style_guides/github_actions_style_guide.md. The current PR does not follow best practices, especially as it does inline several scripts and has way to much logic embedded into the workflows.

[ScottTodd]

Added a few high level comments. This generally looks difficult to maintain, overlaps with what some other workflows provide, and is would benefit from following github actions best practices.

I do think component-specific CI (in addition to TheRock CI) could be appropriate for HRR, but it would be easier to start from a thinner workflow (200 LOC) and add on features incrementally instead of doing this all as a single 1100+ LOC PR.

[danielbolsson]

@andraghetti, thanks. 2e9cd5d moves the hipArray block (block 19) out of Unit_HRR_AllApis_Direct into Unit_HRR_AllApisArrays_Direct, with its own Unit_HRR_AllApisArraysRoundtrip (skipped where the device has no image support). The rocJitsu skip for #14612 now applies only to that case, so Unit_HRR_AllApisRoundtrip runs again on gfx90a, gfx110x, gfx1151 and gfx120x. I could not build or run the GPU suite in this environment; the HRR script tests (config validation) pass, so the new case needs CI to confirm.

[andraghetti]

Gated on MI350 at 2e9cd5d: integration and hidden cases green; the CPU stage fails only on the stale header below.

[andraghetti]

Gated on MI350 at bf6d8ce: GREEN.

[amd-shiraz]

@danielbolsson how much time is the new checks add to a ci run ?

[danielbolsson]

@danielbolsson how much time is the new checks add to a ci run ?

From last run; the continuous time it took to run HRR CI was 29 minutes, but the total accumulated run time on CPU runners was 1h 3 min. We are not using any GPU runners.

The most expensive part is compiling of the rocjitsu emulator itself that takes 20 min. We currently need to do this because TheRock build does not provide it with the launcher we need. It is coming, and when it does we can remove this step from our CI. We currently also need to build CLR with HRR enabled which takes 7 minutes. When HRR is enabled again then we can use TheRock build of CLR.