Repository navigation
fix(ci): run rocprofiler-register TSan and LSan tests without ASLR - #13169
andraghetti wants to merge 6 commits into
Conversation
therock-ci-config dropped Config.get_gpu_families() on 30 Sep (aad3ef6985), so every resolve fell back to the hardcoded labels. For gfx90a that is linux-gfx90a-gpu-rocm (smci250) instead of linux-gfx90a-1gpu-ossci-rocm. Read get_gpu_runner_labels() instead.
The mock still served get_gpu_families(trigger_types), which therock-ci-config dropped on 30 Sep, so the suite kept passing against an API the resolver no longer calls. The mock now mirrors v2: runner-config-v2.json keyed by family, and get_gpu_runner_labels(). TRIGGER_TYPES had no user left and goes with it. With the resolver switched back to get_gpu_families, 5 of the 10 tests fail.
The resolve-ci-config fix moves the rocprofiler-register sanitizer jobs onto linux-gfx942-8gpu-asan-sandbox-rocm. Those hosts run with vm.mmap_rnd_bits=32, which the gcc-12 TSan and LSan runtimes cannot handle: TSan aborts with "unexpected memory mapping" and LSan segfaults. Wrap the TSan and LSan runs in setarch -R, as profiler-hub-sanitizers.yml does. This disables ASLR for the test processes only and leaves the host setting untouched. ASan is not affected and keeps running as before.
github-actions
Bot
added
the
github actions
The wrapper was a matrix key set only on the TSan and LSan entries. Every job runs in the same privileged container with seccomp=unconfined, so setarch -R works on all runners. Call it on the run line instead.
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
…raghetti/hrr-ci/register-sanitizer-aslr
There was a problem hiding this comment.
🟢 Approval recommended
The focused workflow change matches the documented failure mode and has successful CI validation.
0 open findings
What changed in this PR
Disables ASLR for rocprofiler-register CI processes, preventing GCC 12 TSan and LSan failures on high-randomization runners.
Changes:
- Runs the CI script through
setarch -R. - Documents the affected sanitizer/runtime configuration.
| File | Description |
|---|---|
.github/workflows/rocprofiler-register-continuous-integration.yml |
Disables ASLR during configure, build, and test execution. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| apt-get install -y clang-tidy-15 | ||
| update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-15 100 | ||
|
|
||
| - name: Configure, Build, and Test |
There was a problem hiding this comment.
build needs to be separated into a separate github job that uses aws-linux-scale-rocm-prod
we shouldn't use 115 minutes of GPU time doing CPU-only builds
| # ci-tag: '-undefined-behavior-sanitizer' | ||
| # runner_type: 'sandbox' | ||
|
|
||
| runs-on: ${{ matrix.runner_type == 'sandbox' && needs.ci_config.outputs.gfx94x_sandbox_runner || needs.ci_config.outputs.gfx94x_runner }} |
There was a problem hiding this comment.
unfortunately, i will need to block this PR. we cannot use GPU runners to do builds, it must only do tests
it's a waste of resources, time and money to run builds on GPU runners. I know that ASAN, TSAN and LSAN are costly in time, and it takes up GPU runner resources
we will need to re-architect this before moving forward.
There was a problem hiding this comment.
feel free to chat with me on teams about this re-architect, but it will need to be done. i don't want to support CIs that waste resources
There was a problem hiding this comment.
Fair point, you're totally right Geo. Unfortunately the build/test split is more than we can take on right now, so I'm closing this.
For whoever does it: TSan and LSan need setarch -R on hosts with vm.mmap_rnd_bits=32.
Motivation
Since #12960, the rocprofiler-register sanitizer jobs resolve to
linux-gfx942-8gpu-asan-sandbox-rocm. On those hosts the TSan and LSan jobs fail (run 37532779386). ASan passes.The cause is the host setting
vm.mmap_rnd_bits=32. The gcc-12 TSan and LSan runtimes cannot handle it: TSan aborts with "unexpected memory mapping" and LSan segfaults.ROCm/therock-ci-config#54 has since disabled the gfx94x ASAN sandbox. Until it is back, the sanitizer jobs wait on the fallback label and do not run. This change keeps TSan and LSan green once the sandbox returns.
Technical Details
setarch $(uname -m) -R python3 ./scripts/run-ci.py ...for every job in the matrix.setarch -Rdisables ASLR for the CI processes only. The host setting stays as it is.profiler-hub-sanitizers.ymldoes the same.seccomp=unconfined, sosetarch -Rworks on every runner, not only on the sandbox.Issue Tracking
None.
Test Plan
vm.mmap_rnd_bits=32, with and withoutsetarch -R, and withvm.mmap_rnd_bits=28.workflow_dispatchon the sandbox runners, from a copy of this branch.Test Result
vm.mmap_rnd_bits=32, 131 of 132 TSan tests fail with "unexpected memory mapping". 56 of 132 LSan tests fail, 24 of them with a segfault.setarch -R, or withvm.mmap_rnd_bits=28, all 132 tests pass for TSan, LSan and ASan.setarch -Ron every job, the 7 build and test jobs onlinux-gfx942-1gpu-ccs-csp-ossci-rocmpass, 135 of 135 tests each: run 38219897178. The sanitizer jobs stay queued, because feat(ci): disable ASAN sandbox runners for gfx94x and gfx950 therock-ci-config#54 disabled the gfx94x ASAN sandbox.Submission Checklist