Repository navigation
Fix ASan runtime discovery and component loader paths - #8721
Merged
Merged
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
sa-faizal
marked this pull request as ready for review
October 3, 2026 19:53
sa-faizal
added
the
ci:host-asan
sa-faizal
marked this pull request as draft
October 4, 2026 06:53
This was referenced Oct 4, 2026
sa-faizal
marked this pull request as ready for review
October 8, 2026 04:37
sa-faizal
force-pushed
the
users/safaizal/asan-runtime-discovery-20261003
branch
from
October 8, 2026 18:56
28bc05d to
4c3d143
Compare
nunnikri
approved these changes
Oct 8, 2026
| # Importing is_asan from amdgpu_family_matrix.py | ||
| sys.path.append(str(THEROCK_DIR / "build_tools" / "github_actions")) | ||
| from amdgpu_family_matrix import is_asan | ||
| from configure_asan_env import get_asan_runtime_path |
Contributor
There was a problem hiding this comment.
Not an issue, just an observation. asan env is getting imported in release job also
| THEROCK_DIR = SCRIPT_DIR.parent.parent.parent | ||
| sys.path.append(str(THEROCK_DIR / "build_tools" / "github_actions")) | ||
| from amdgpu_family_matrix import is_asan | ||
| from configure_asan_env import get_asan_runtime_path |
Contributor
There was a problem hiding this comment.
Not an issue, just an observation. asan env is getting imported in release job also
sa-faizal
added a commit
that referenced
this pull request
Oct 8, 2026
TensileLite pytest jobs can fail during startup when an imported extension loads ASan too late. Preload the discovered runtime when starting the pytest child, preserving existing preloads, sanitizer settings, and the parent environment. Reuse the existing environment-prepend test to check preload order and parent-environment preservation; no test methods are added. This PR is stacked on #8721 and contains only the pytest follow-up. Retarget to `main` after that dependency lands. Validation: - All 45 existing pytest-runner tests passed on Linux and Windows. Formatting and diff checks passed. [Current Linux/Windows unit CI passed](https://github.com/ROCm/TheRock/actions/runs/37729317109). - On **MI325 (gfx942)**, the current runner with matching [host-ASan artifacts](https://github.com/ROCm/TheRock/actions/runs/37149900303) collected and passed **25/25 existing timing tests**. This validates host-side startup and timing assertions, not GPU kernel execution. - The 25 assertions pass, but teardown exits 1 with 679,993 bytes in 587 allocations reported by LeakSanitizer. Controlled Python 3.12.3 baselines reproduce leaks from importing pytest alone (69,553 bytes) and NumPy alone (941,456 bytes). The same 25 tests without shared conftest/plugin loading retain only the pytest baseline. These dependency shutdown failures remain unresolved; no sanitizer suppression was added. [rocm-libraries #13055](ROCm/rocm-libraries#13055) addresses the separate isolated rocisa import leak. ISSUE ID : #8735 *AI-assisted change.* Fixes #8735 --------- Co-authored-by: Sahil Faizal <safaizal@amd.com>
dileepr1
added a commit
that referenced
this pull request
Oct 13, 2026
Takes main's versions of the ASAN env helpers, which #8721 refactored (get_asan_runtime_path, and _resolve_library_path now appends the clang resource directory), and restores main's test_components gate: #8123 fixed the sanity LeakSanitizer failure, so the TEMPORARY !failure() removal is no longer needed and the run reproduces production behaviour.
dileepr1
added a commit
that referenced
this pull request
Oct 13, 2026
#8721 landed the same ASAN runtime work on main (both runtime library names, and the clang resource directory on LD_LIBRARY_PATH), so the older #8341 copy carried here was stale and failed 14 of its own tests against the refactor. The branch is now exactly #8455 on current main, which is what would run if #8455 merged.
dileepr1
added a commit
that referenced
this pull request
Oct 13, 2026
#8721 changed ASAN runtime discovery to anchor on the artifact tree, which the wheel tests have no equivalent of. Keep this branch's PATH-based lookup as the generalisation of both, and re-anchor #8721's get_asan_runtime_path on it so test_hiptests.py keeps working. test_hiptests.py itself takes main's version. Co-authored-by: Cursor <cursoragent@cursor.com>
dileepr1
added a commit
that referenced
this pull request
Oct 13, 2026
…iscovery #8721 added get_asan_runtime_path for standalone launchers but no test. This branch changes how it finds a compiler, so pin the contract its callers rely on: an explicit ASAN_RUNTIME_PATH wins, a missing one raises, discovery works without the toolchain on PATH, and failures surface as FileNotFoundError. Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Oct 14, 2026
dileepr1
added a commit
that referenced
this pull request
Nov 3, 2026
## Summary Fixes #6331. `rocm-sdk test` aborts on every Linux ASAN run with `ASan runtime does not come first in initial library list`, so the wheel test jobs fail without executing a single test. A process's library load order is fixed at `exec`, so nothing inside the test can insert the ASAN runtime ahead of the interpreter; the only lever is the environment before Python starts. [#8123](#8123) added that preloading but wired it into `test_component.yml` only, and `test_rocm_wheels.yml` had no ASAN handling at all. ## What changed - **`test_rocm_wheels.yml`** — new optional `artifact_group` input, gated on `contains(inputs.artifact_group, 'asan')` the same way `test_component.yml` is. Resolves the ASAN environment after each install, because the runtime ships inside the wheels and `rocm[profiler]` installs into its own venv, then sets `LD_PRELOAD` on the `rocm-sdk test` steps. - **`multi_arch_ci_linux.yml`** — passes `artifact_group` through. - **`configure_asan_env.py`** — locates the compiler on `PATH` as well as in an artifact tree, since these jobs `pip install` ROCm and have no artifact tree. Both runtime spellings are handled (#8077). Exports `LSAN_OPTIONS` pointing at the new suppressions file. `get_asan_runtime_path()` from #8721 keeps working for standalone launchers, now with unit tests. - **`test_tools/lsan_suppressions.txt`** — new. Preloading also puts the uninstrumented CPython and `/usr/bin/perl` under LeakSanitizer, which reports their interpreter internals as leaks. These are scoped suppressions rather than `detect_leaks=0`, so leaks in the ROCm libraries under test still fail the job. - **`test_component.yml`** — puts the extracted `llvm/bin` on `PATH` for the above. ## Validation **CI:** [run 40887799669](https://github.com/ROCm/TheRock/actions/runs/40887799669) — `test_rocm_wheels.yml` on `gfx94X-dcgpu` against ASAN wheels, green. All three `rocm-sdk test` steps execute and pass: | Step | Result | | --- | --- | | `Test rocm[libraries]` | `Ran 19 tests` — OK | | `Test rocm[libraries,devel]` | `Ran 27 tests` — OK | | `Test rocm[profiler]` | `Ran 21 tests` — OK | (Validated with workflow run since [#8978](#8978) removes the ASAN GPU runner fields unless the event is a pull_request in ROCm/rocm-systems. An empty runner list skips Test Python rather than failing.) Leak detection stays on rather than being switched off: LeakSanitizer prints its suppression accounting at each exit and finds nothing outside the suppressed templates. ``` Suppressions used: count bytes template 74 83083 libpython ``` **Counterfactual:** [run 40888153220](https://github.com/ROCm/TheRock/actions/runs/40888153220) is the same wheels with `artifact_group` left empty, so the preload is off. All three steps abort with `ASan runtime does not come first in initial library list` and execute zero tests. With the input empty every gate in this PR is off, which is `main`'s behaviour today — so that run is #6331 reproduced against current wheels. **Unit:** `build_tools/github_actions/tests/` — 17 new tests, all passing, and no pre-existing test changed state. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Host-ASan tests fail before execution because the runtime directory is missing from
LD_LIBRARY_PATH, while HIP and rocprofiler-sdk launchers query only the older architecture-suffixed filename.Export the runtime directory and reuse the validated configured runtime or shared discovery fallback, supporting current and legacy filenames. This addresses loader failures in rocPRIM/hipCUB/rocThrust resource helpers and TensileLite's rocisa import. Existing tests are reused; no test methods are added.
Validation:
lddoutput. The SDK launcher generated the unsuffixed preload path and correct loader directory. No sanitizer suppressions were added.ISSUE ID : #7202