Repository navigation
ci(hipblaslt): shorten ASAN CI test timeout and log the runner on hang - #12728
Merged
Merged
Conversation
The hipBLASLt ASAN CI test step hangs to its full 180-minute timeout on a meaningful share of runs instead of failing on an actual test or sanitizer result. Shorten the test step to a 20-minute cap (30-minute step limit, 90-minute job limit) so a hang fails fast with a clear error, and log the runner name so a future hang can be attributed to a specific host. ISSUE ID : #12727 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
github-actions
Bot
added
github actions
project: none
⏳ Waiting on Required Checks⏳ Required checks have not all concluded yet. This bot run is exiting without failing so those checks can report independently.
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
⏳ Waiting for required checks to start or finish:
|
Contributor
Author
Boo! |
talumbau
approved these changes
Sep 30, 2026
Contributor
TheRock Submodule Bump ActivityNewest first
|
ScottTodd
reviewed
Oct 5, 2026
Comment on lines
+262
to
+265
| env: | ||
| RUNNER_NAME: ${{ runner.name }} | ||
| run: | | ||
| echo "Runner: ${RUNNER_NAME}" |
Member
There was a problem hiding this comment.
- Prints the runner name (
${{ runner.name }}) at the start of the sanity-check step, so a
future hang can be attributed to a specific host — GitHub's API does not otherwise expose
which runner ran a job.
FYI the second line of log output on every github actions workflow run is the runner name, e.g. https://github.com/ROCm/rocm-libraries/actions/runs/36607831817/job/109615347152#step:1:2
Current runner version: '2.336.0'
Runner name: 'smci250-ccs-aus-c05-13-dev-1gpu-gfx90a-gpu1-1790722069'
Runner group name: 'default'
Machine name: 'smci250-ccs-aus-c05-13-dev-1gpu-gfx90a-gpu1'
GITHUB_TOKEN Permissions
Contents: read
Metadata: read
Secret source: Actions
Cache mode: write
This was referenced Oct 5, 2026
1 task done
sa-faizal
added a commit
that referenced
this pull request
Oct 8, 2026
ISSUE ID: #12727 ## Motivation The hipBLASLt per-PR ASAN lane has been hanging under full ASAN with `HSA_XNACK=1`. This switches it to host-only ASAN with XNACK off, matching the coverage already described in `TESTING.md`. GPU correctness tests still run, and ASAN still checks CPU memory accesses. The trade-off is that no gfx90a device-ASAN runtime tests remain in CI. Before restoring that coverage, we need to review runner capacity and full-ASAN stability with DevOps, potentially for a dedicated nightly pool. The cause of the hangs remains under investigation. [#12728](#12728) added shorter timeouts and diagnostics. [#12813](#12813) offers an alternative runner-pool workaround. The investigation is tracked in [TheRock-Infra #978](ROCm/TheRock-Infra#978). ## Technical Details - Use `HOST_ASAN` artifacts and `HSA_XNACK=0` in the focused lane. - Stop requiring `xnack+` targets for host-only ASAN; preserve existing full-ASAN behavior. - Update `TESTING.md` and remove the stale gfx942 `ci:asan` row. ## Test Plan Run the existing gfx90a build and smoke tests, plus whitespace and applicable pre-commit checks. ## Test Result [Build and all 950 smoke tests passed](https://github.com/ROCm/rocm-libraries/actions/runs/37402899925) on `c92deaa`. The documentation follow-up passes whitespace checks; pre-commit completed with no applicable hooks. CI for the latest commit must pass before merge. ## Submission Checklist - [x] Reviewed the [contributing guidelines](https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests). ## Risk level Medium (3/5): retains host-ASAN and GPU correctness checks, but removes device-ASAN coverage from the gfx90a lane. A passing run does not prove the hangs are fixed. --------- Co-authored-by: Sahil Faizal <197828758+sa-faizal@users.noreply.github.com> Co-authored-by: Sahil Faizal <safaizal@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ISSUE ID : #12727
Motivation
The
hipBLASLt ASAN CIworkflow runs the hipBLASLt AddressSanitizer smoke tests on everytouching PR, and its test step is not currently a reliable read on the change under review.
When the test job hangs — which recent runs show happening far more often than an actual
test or sanitizer failure — the step keeps running for its full 180-minute timeout before
being killed, and nothing in the log says which runner it ran on. A silent three-hour wait
that looks identical to a real hang leaves the lane effectively unusable as a review signal.
Technical Details
testjob's overall timeout from 210 to 90 minutes, and theTest hipBLASLt (quick)step's timeout from 180 to 30 minutes.hipblaslt-testinvocation intimeout -k 60 20m. When that fires (exit code124 or 137), the step prints an
::error::line naming the runner and the 20-minutecutoff, then exits with that code; any other failure keeps its own original exit code.
${{ runner.name }}) at the start of the sanity-check step, so afuture hang can be attributed to a specific host — GitHub's API does not otherwise expose
which runner ran a job.
Test Plan
script, and
actionlinton the modified workflow.is this change's own test.
Test Result
actionlint: no findings.timeout -k 60 20mwas killed at thecap (exit 124), printed the expected
::error::line, and left no child process runningafterward. A stand-in non-zero exit passed its original code through unchanged. A stand-in
pass exited 0.
hipBLASLt ASAN CIrun, are pending.Submission Checklist
Risk level
Low (2/5): narrow change to one non-required, informational CI lane, with no product code
or test content changed. The only effect on a passing run is the lower time cap — passing
smoke runs measured at 2-3 minutes, so the new 20-minute step cap and 90-minute job cap
leave a wide margin — and this lane isn't a required check, so an unexpected early cutoff
would not block a merge.
Related
the underlying runner-hang cause stays open there)
🤖 Generated with Claude Code