[rkayaith]

JIRA ID : AIHPBLAS-5081

Motivation

The hipBLASLt ASAN workflow runs only hipBLASLt *smoke* tests on gfx90a, so host-side leaks in hipSPARSELt, on gfx942/gfx950, and in tests outside *smoke* go unseen. This PR moves hipBLASLt host-ASAN CI from gfx90a to gfx942/gfx950, adds hipSPARSELt, and adds a nightly run of the broader standard tests.

Dropping gfx90a slightly reduces architecture-specific sanitizer coverage, but avoids the persistent gfx90a runner hangs that make this CI flaky (TheRock-Infra #978). Regular gfx90a Math CI coverage remains.

The leaks this coverage found are fixed in the PRs below this one in the stack, and in #14800 above it.

Technical Details

• Add hipBLASLt and hipSPARSELt on gfx942/gfx950 to the per-PR host ASAN CI, with extended test coverage in a nightly run (hipblaslt-asan-ci.yml, test_hipblaslt.py, test_hipsparselt.py, TESTING.md):
  • Per-PR (quick, *smoke*): 2 build jobs (gfx942 ~1h43m, gfx950 ~1h14m) and 4 test jobs (hipBLASLt gfx942 24m, hipBLASLt gfx950 19m, hipSPARSELt gfx942 16m, hipSPARSELt gfx950 13m), replacing the existing gfx90a build and hipBLASLt gfx90a test.
  • Nightly at 07:00 UTC (standard, *smoke*:*quick*:*pre_checkin*-*known_bug*): 2 build jobs (gfx942 ~1h24m, gfx950 ~1h01m) and 4 test jobs (hipBLASLt gfx942 1h34m, gfx950 2h00m; hipSPARSELt gfx942 1h05m, gfx950 59m).
  • Manual dispatch requires a test_type input.
  • Pull requests touching either library, either test driver, the suppression file or the workflow trigger the per-PR run.
  • hipSPARSELt quick excludes *known_bug* and the M=N=128 spmm_strided_batched smoke cases, which are much slower under ASAN (43-67 minutes per GPU); standard still runs them.
  • Timeouts are set per architecture, library and test type.
• Remove unused suppressions: drop the rocr::AMD::, hsaKmt and libhsa-runtime64.so rules, which matched nothing in these runs.
• Suppress ROCM-30419: add leak:^rocsparselt_matmul_descr_set_attribute$, and stop setting print_suppressions=0, so each run logs which suppressions matched.
• Update the shared build image to ccache 4.12.1 (.github/actions/ci-env/action.yml): use the image published by TheRock #9506, whose ccache keeps the -Xarch_host sanitizer flags. The image also updates CMake to 3.31.12 and sets USER root. This image is shared with component CI.

Test Plan

Run hipBLASLt ASAN CI on this PR (quick), and dispatch it with test_type: standard.

Test Result

The gfx942/gfx950 jobs passed in quick and standard runs. Times are whole-job times. Builds are the same for both test types. Test jobs start only after both builds complete.

The nightly standard run is new. The per-PR quick run replaces the gfx90a jobs with gfx942/gfx950 jobs for both libraries.

Job	quick (per-PR)	standard (nightly, new)
(removed) Build gfx90a	~25m (~3m hipBLASLt / <1m hipSPARSELt)	—
(new) Build gfx942	~1h43m (~42m hipBLASLt / ~18m hipSPARSELt)	~1h24m (~40m hipBLASLt / ~18m hipSPARSELt)
(new) Build gfx950	~1h14m (~35m hipBLASLt / ~1m hipSPARSELt)	~1h01m (~36m hipBLASLt / ~1m hipSPARSELt)
(removed) Test hipBLASLt gfx90a	4m (2m test time)	—
(new) Test hipBLASLt gfx942	24m (14m test time)	1h34m (1h25m test time)
(new) Test hipBLASLt gfx950	19m (5m test time)	2h00m (1h39m test time)
(new) Test hipSPARSELt gfx942	16m (8m test time)	1h05m (56m test time)
(new) Test hipSPARSELt gfx950	13m (4m test time)	59m (39m test time)

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

Risk level

Low (2/5): CI, test-driver and suppression changes, plus the shared build image update. The workflow is not a required check.

[therock-pr-bot]

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[therock-pr-bot]

🎉 All checks passed! This PR is ready for review.

[Copilot]

🟡 Changes recommended

The known-failing half-alpha reproducers terminate the shared sanitizer test process, preventing later selected tests from running.

1 open issue

• Separate HalfAlpha ASAN reproducers to preserve suite coverage · New

What changed in this PR

Expands the dedicated host-ASAN workflow to cover hipBLASLt and hipSPARSELt on gfx942/gfx950 while retaining hipBLASLt smoke coverage on gfx90a.

Changes:

• Adds architecture-specific builds and separate library test jobs, with timeouts and failed-test log uploads.
• Adds standard-tier filters and permits regular hipSPARSELt GPU targets for host-ASAN builds.
• Adds standalone half-alpha regression tests and updates sanitizer coverage documentation.

File	Description
test/​therock/​test_hipsparselt.py	Adds the standard-tier test filter.
test/​therock/​test_hipblaslt.py	Replaces a duplicate quick branch with standard-tier selection.
projects/​hipsparselt/​cmake/​hipsparselt_supported_architectures.cmake	Distinguishes host-ASAN from device-ASAN target requirements.
projects/​hipblaslt/​TESTING.md	Documents expanded sanitizer coverage and test tiers.
projects/​hipblaslt/​clients/​tests/​src/​half_alpha_gtest.cpp	Adds two standalone half-alpha regression cases.
projects/​hipblaslt/​clients/​tests/​src/​CMakeLists.txt	Registers the new regression source.
.github/​workflows/​hipblaslt-asan-ci.yml	Expands triggers, builds, GPU jobs, artifacts, and timeout handling.

_{Review effort: Balanced}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #14240      +/-   ##
===========================================
- Coverage    77.75%   77.75%   -0.00%     
===========================================
  Files         2904     2904              
  Lines       491207   491309     +102     
  Branches     74462    74495      +33     
===========================================
+ Hits        381923   381993      +70     
- Misses       84288    84309      +21     
- Partials     24996    25007      +11     

Flag	Coverage Δ		*Carryforward flag
TensileLite-CPP	53.85% <ø> (ø)		Carriedforward from 70c1d99
TensileLite-Unit	85.03% <ø> (ø)		Carriedforward from 70c1d99
hipBLAS	90.62% <ø> (ø)		Carriedforward from 70c1d99
hipBLASLt	70.07% <ø> (-0.01%)	⬇️
hipCUB	82.68% <ø> (ø)		Carriedforward from 70c1d99
hipDNN	87.25% <ø> (ø)		Carriedforward from 70c1d99
hipFFT	42.65% <ø> (ø)		Carriedforward from 70c1d99
hipRAND	76.12% <ø> (ø)		Carriedforward from 70c1d99
hipSOLVER	69.67% <ø> (ø)		Carriedforward from 70c1d99
hipSPARSE	87.03% <ø> (ø)		Carriedforward from 70c1d99
rocBLAS	69.20% <ø> (ø)		Carriedforward from 70c1d99
rocFFT	46.49% <ø> (ø)		Carriedforward from 70c1d99
rocRAND	57.32% <ø> (ø)		Carriedforward from 70c1d99
rocSOLVER	76.65% <ø> (ø)		Carriedforward from 70c1d99
rocSPARSE	83.54% <ø> (ø)		Carriedforward from 70c1d99
rocThrust	93.15% <ø> (ø)		Carriedforward from 70c1d99

*This pull request uses carry forward flags. Click here to find out more.
see 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[tony-davis]

Hi Rahul! Approving. This is a nice expansion, and I like that it already shook loose real leaks.

On the manylinux image bump: I think it's fine to keep here. It matches what TheRock already pins (#9589), our pinned therock-ref already uses it, and component CI passed on it. The thing is, bump automation only handles therock-ref and baseline-run-id, so nothing automated owns this line. I'd like infra to see it specifically. @ScottTodd @kailash-khalasi, could one of you take a quick look at the ci-env image change? It also moves CMake to 3.31.12 (plus USER root), not just ccache, so it'd be good for the description to say that too. I think the hipccl3 copy of ci-env still points at the old digest as well.

A few things I'd still like to see, none of them blocking from my side:

1. Who owns the nightly when it goes red? GitHub only emails whoever last edited the cron line, so I bet it'd be just you. Probably worth naming an owner in TESTING.md.
2. The standard run is from Oct 29, before the gfx90a removal and the suppression cleanup. One fresh standard dispatch on the final head before the nightly goes live would be great.
3. Has infra acked the added gfx942/gfx950 load? By my math it's roughly 1.2 GPU-hours per PR and ~9 nightly. Also, are those build times warm-cache?

Smaller: I'd guard the hipSPARSELt M=N=128 exclusion with is_asan(), since these drivers become canonical under RFC0010. A one-line comment on where the timeout numbers came from would help the next person too.

[rkayaith]

Updates since the last review:

• Added CMake 3.31.12 and USER root to the image description.
• Ran a fresh standard dispatch: both builds and all four GPU jobs passed, with no sanitizer reports. Updated the description’s timings and result link.
• Guarded the hipSPARSELt M=N=128 exclusion with is_asan(). The quick ASAN run passed too.
• Checked the build-cache stats: the quoted quick builds had about 98% ccache hits, so those are warm-cache timings.
• Added a workflow comment explaining that timeouts are set to ~1.5× observed runtimes.