Repository navigation
Conversation
github-actions
Bot
added
github actions
project: none
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
newling
marked this pull request as ready for review
September 30, 2026 23:03
newling
force-pushed
the
users/newling/hipblaslt-asan-test-runner
branch
from
October 5, 2026 17:49
55deb15 to
d820382
Compare
This was referenced Oct 5, 2026
newling
force-pushed
the
users/newling/hipblaslt-asan-test-runner
branch
from
October 6, 2026 22:01
c65f79d to
337340b
Compare
sa-faizal
added a commit
that referenced
this pull request
Oct 8, 2026
ISSUE ID: #12727 ## Motivation The hipBLASLt per-PR ASAN lane has been hanging under full ASAN with `HSA_XNACK=1`. This switches it to host-only ASAN with XNACK off, matching the coverage already described in `TESTING.md`. GPU correctness tests still run, and ASAN still checks CPU memory accesses. The trade-off is that no gfx90a device-ASAN runtime tests remain in CI. Before restoring that coverage, we need to review runner capacity and full-ASAN stability with DevOps, potentially for a dedicated nightly pool. The cause of the hangs remains under investigation. [#12728](#12728) added shorter timeouts and diagnostics. [#12813](#12813) offers an alternative runner-pool workaround. The investigation is tracked in [TheRock-Infra #978](ROCm/TheRock-Infra#978). ## Technical Details - Use `HOST_ASAN` artifacts and `HSA_XNACK=0` in the focused lane. - Stop requiring `xnack+` targets for host-only ASAN; preserve existing full-ASAN behavior. - Update `TESTING.md` and remove the stale gfx942 `ci:asan` row. ## Test Plan Run the existing gfx90a build and smoke tests, plus whitespace and applicable pre-commit checks. ## Test Result [Build and all 950 smoke tests passed](https://github.com/ROCm/rocm-libraries/actions/runs/37402899925) on `c92deaa`. The documentation follow-up passes whitespace checks; pre-commit completed with no applicable hooks. CI for the latest commit must pass before merge. ## Submission Checklist - [x] Reviewed the [contributing guidelines](https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests). ## Risk level Medium (3/5): retains host-ASAN and GPU correctness checks, but removes device-ASAN coverage from the gfx90a lane. A passing run does not prove the hangs are fixed. --------- Co-authored-by: Sahil Faizal <197828758+sa-faizal@users.noreply.github.com> Co-authored-by: Sahil Faizal <safaizal@amd.com>
Contributor
Author
|
The ASAN CI job is now semi-fixed (failing component is removed from it) so this PR is less relevant. |
newling
marked this pull request as draft
October 9, 2026 23:55
newling
force-pushed
the
users/newling/hipblaslt-asan-test-runner
branch
2 times, most recently
from
October 13, 2026 03:46
cc0a8c2 to
ce9db58
Compare
newling
force-pushed
the
users/newling/hipblaslt-asan-test-runner
branch
from
October 26, 2026 15:25
ce9db58 to
49a339f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hipBLASLt's gfx90a AddressSanitizer (ASAN) smoke job can hang during library initialization, before any tests start. The symptom is tracked in #12727, and ROCm/TheRock-Infra#978 tracks the investigation of the
linux-gfx90a-gpu-rocmrunner pool.This PR changes the test runner to
linux-gfx90a-1gpu-ossci-rocm. It changes one runner label and can land independently of #12716 and #12720. The 20-minute test timeout and runner diagnostics from the already-merged #12728 remain in place.In the controlled runner comparison, the same artifact, container and ASAN settings hung before the version banner on the original pool and completed the smoke suite on the selected pool. Two debugger captures showed the stalled process waiting for GPU code-cache invalidation during the first
hipMemsetin handle creation. The specific host/runtime cause remains unresolved. The original pool also has intermittent passes, but another startup timeout on October 5 shows that the symptom persists.The full ASAN workflow passed both before the rebase and after the rebase. The latter completed all 948 smoke tests. Reusing that artifact, a diagnostic pilot and the first qualification trial also passed the complete smoke suite and normally linked HIP and hipBLASLt startup probes on two distinct physical GPU sets. Their original-pool counterparts remain queued, so the planned ten-pair comparison has no completed pairs yet.
The latest commit,
c65f79d4d37, adds only the inline workaround explanation and issue link. Root pre-commit and actionlint pass; its fresh CI run is queued.This is a runner-pool workaround. Repeated runs are still needed to establish reliability, and #12727 and ROCm/TheRock-Infra#978 remain open for investigation of the underlying cause.