Repository navigation
fix(hipblaslt): release matmul descriptors on early exits - #14714
Merged
rkayaith merged 1 commit intoNov 3, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the descriptor-lifetime cleanup.
No open issues 🎉
What changed in this PR
Fixes descriptor leaks on early exits from hipBLASLt’s testing_matmul_with_bias helper, including the no-solution path identified during expanded ASAN testing.
Changes:
- Adds
std::unique_ptrownership with matching destroy functions immediately after descriptor creation. - Removes end-of-function manual cleanup while preserving raw handle vectors required by
GroupedGemm::setProblem.
| File | Description |
|---|---|
projects/hipblaslt/clients/common/include/testing_matmul.hpp |
Automatically releases matrix-layout and matmul descriptors on normal completion and early returns. |
Review effort: Balanced
rkayaith
force-pushed
the
users/rkayaith/hipblaslt-matmul-descriptor-cleanup
branch
from
October 29, 2026 23:30
e4edad7 to
4685a67
Compare
rkayaith
force-pushed
the
users/rkayaith/hipblaslt-asan-memory-reclamation-skip
branch
from
October 29, 2026 23:30
5884e6c to
6bfd13f
Compare
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Codecov Report✅ All modified and coverable lines are covered by tests. ❌ Your project status has failed because the head coverage (68.77%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #14714 +/- ##
========================================
Coverage 77.51% 77.51%
========================================
Files 2903 2903
Lines 490383 490383
Branches 74218 74218
========================================
+ Hits 380081 380082 +1
Misses 85389 85389
+ Partials 24913 24912 -1
*This pull request uses carry forward flags. Click here to find out more. 🚀 New features to boost your workflow:
|
rkayaith
marked this pull request as ready for review
October 30, 2026 21:05
|
🎉 All checks passed! This PR is ready for review. |
rkayaith
force-pushed
the
users/rkayaith/hipblaslt-matmul-descriptor-cleanup
branch
from
November 2, 2026 18:12
4685a67 to
f6901f6
Compare
rkayaith
force-pushed
the
users/rkayaith/hipblaslt-asan-memory-reclamation-skip
branch
from
November 2, 2026 18:12
6bfd13f to
df93460
Compare
talumbau
approved these changes
Nov 2, 2026
1 task
rkayaith
force-pushed
the
users/rkayaith/hipblaslt-asan-memory-reclamation-skip
branch
from
November 2, 2026 23:26
df93460 to
5027937
Compare
rkayaith
force-pushed
the
users/rkayaith/hipblaslt-matmul-descriptor-cleanup
branch
from
November 2, 2026 23:26
f6901f6 to
543f01f
Compare
rkayaith
changed the base branch from
users/rkayaith/hipblaslt-asan-memory-reclamation-skip
to
develop
November 2, 2026 23:26
rkayaith
enabled auto-merge (squash)
November 2, 2026 23:33
testing_matmul_with_bias destroyed its matrix-layout and matmul descriptors only at the end of the function, so any early return leaked them. The expected "NO solution found!" failure in rocroller_predicate_test.unrollXYK leaks 5,098 bytes under host ASAN on gfx950. Own each descriptor with a unique_ptr deleter as soon as it is created and remove the manual destruction at the end. The raw handle vectors remain because GroupedGemm::setProblem takes vectors of raw handles.
rkayaith
force-pushed
the
users/rkayaith/hipblaslt-matmul-descriptor-cleanup
branch
from
November 3, 2026 15:43
543f01f to
91fb8ed
Compare
rkayaith
deleted the
users/rkayaith/hipblaslt-matmul-descriptor-cleanup
branch
November 3, 2026 20:24
assistant-librarian Bot
pushed a commit
to ROCm/hipBLASLt
that referenced
this pull request
Nov 3, 2026
fix(hipblaslt): release matmul descriptors on early exits (#14714) JIRA ID : N/A ## Motivation The test helper `testing_matmul_with_bias` destroys its matrix-layout and matmul descriptors only at the end of the function. Any early return leaks them, including the expected "NO solution found!" failure in `rocroller_predicate_test.unrollXYK`. Found while expanding ASAN testing in [#14240](ROCm/rocm-libraries#14240). Current ASAN CI does not run this test. The expanded-coverage PR [#14240](ROCm/rocm-libraries#14240) adds it to scheduled standard ASAN runs on gfx950. ## Technical Details Own each descriptor with a `std::unique_ptr` and its `hipblasLtMatrixLayoutDestroy` or `hipblasLtMatmulDescDestroy` deleter as soon as it is created, and remove the manual destruction at the end of the function. The raw handle vectors remain because `GroupedGemm::setProblem` takes vectors of raw handles. ## Test Plan With a host AddressSanitizer (`HOST_ASAN`) client build on gfx950, run from the repository root: ```sh ASAN_OPTIONS=quarantine_size_mb=600:detect_leaks=1 \ LSAN_OPTIONS=suppressions=test/therock/lsan.supp \ hipblaslt-test --gtest_filter='*rocroller_predicate_test.unrollXYK*' ``` ## Test Result Leaked bytes went from 5,098 to 0. <details> <summary>ASAN error output (before fix)</summary> ```text ================================================================= ==377==ERROR: LeakSanitizer: detected memory leaks Direct leak of 184 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d895ba74 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15ba74) #2 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #3 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #4 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #5 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #6 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #7 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #8 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d895ab5c in rocblaslt_matrix_layout_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15ab5c) #2 0x70f5d8bec5b2 in hipblasLtMatrixLayoutCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ec5b2) #3 0x63ceaafe626d in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda1'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #4 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #5 0x63ceaaf602e0 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e12e0) #6 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #7 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #8 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d895ab5c in rocblaslt_matrix_layout_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15ab5c) #2 0x70f5d8bec5b2 in hipblasLtMatrixLayoutCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ec5b2) #3 0x63ceaafe665d in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda2'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #4 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #5 0x63ceaaf60546 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e1546) #6 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #7 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #8 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d895ab5c in rocblaslt_matrix_layout_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15ab5c) #2 0x70f5d8bec5b2 in hipblasLtMatrixLayoutCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ec5b2) #3 0x63ceaafe5e7d in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda0'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #4 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #5 0x63ceaaf6007d in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e107d) #6 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #7 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #8 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d895ab5c in rocblaslt_matrix_layout_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15ab5c) #2 0x70f5d8bec5b2 in hipblasLtMatrixLayoutCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ec5b2) #3 0x63ceaafe5a8d in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #4 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #5 0x63ceaaf5fe0f in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e0e0f) #6 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #7 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #8 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 1976 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a91976 in std::vector<TensileLite::TensorDescriptor, std::allocator<TensileLite::TensorDescriptor>>::vector(std::vector<TensileLite::TensorDescriptor, std::allocator<TensileLite::TensorDescriptor>> const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x291976) #2 0x70f5d8a916b8 in TensileLite::ContractionProblem::ContractionProblem(TensileLite::ContractionProblem const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x2916b8) #3 0x70f5d8a2b2ec in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22b2ec) #4 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #5 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #6 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #7 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #8 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #9 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #10 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #11 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #12 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #13 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 1536 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d89c03d2 in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c03d2) #2 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #3 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #4 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #5 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #6 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #7 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #8 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #9 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 608 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a91739 in TensileLite::ContractionProblem::ContractionProblem(TensileLite::ContractionProblem const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x291739) #2 0x70f5d8a2b2ec in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22b2ec) #3 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #4 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #5 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #6 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #7 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #8 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #9 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #10 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #11 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #12 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 96 byte(s) in 4 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a91dd9 in TensileLite::TensorDescriptor::TensorDescriptor(TensileLite::TensorDescriptor const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x291dd9) #2 0x70f5d8a91a2a in std::vector<TensileLite::TensorDescriptor, std::allocator<TensileLite::TensorDescriptor>>::vector(std::vector<TensileLite::TensorDescriptor, std::allocator<TensileLite::TensorDescriptor>> const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x291a2a) #3 0x70f5d8a916b8 in TensileLite::ContractionProblem::ContractionProblem(TensileLite::ContractionProblem const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x2916b8) #4 0x70f5d8a2b2ec in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22b2ec) #5 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #6 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #7 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #8 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #9 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #10 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #11 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #12 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #13 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #14 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 96 byte(s) in 4 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a91f25 in TensileLite::TensorDescriptor::TensorDescriptor(TensileLite::TensorDescriptor const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x291f25) #2 0x70f5d8a91a2a in std::vector<TensileLite::TensorDescriptor, std::allocator<TensileLite::TensorDescriptor>>::vector(std::vector<TensileLite::TensorDescriptor, std::allocator<TensileLite::TensorDescriptor>> const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x291a2a) #3 0x70f5d8a916b8 in TensileLite::ContractionProblem::ContractionProblem(TensileLite::ContractionProblem const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x2916b8) #4 0x70f5d8a2b2ec in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22b2ec) #5 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #6 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #7 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #8 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #9 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #10 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #11 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #12 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #13 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #14 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 96 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2d783 in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22d783) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 64 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2cbd5 in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22cbd5) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 34 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2c56e in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22c56e) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 32 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2ca60 in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22ca60) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 32 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2d607 in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22d607) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 32 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2cd47 in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22cd47) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 32 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2c8e5 in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22c8e5) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 24 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2ceca in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22ceca) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 8 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2d1ae in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22d1ae) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 8 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2d31c in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22d31c) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 8 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2d040 in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22d040) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) Indirect leak of 8 byte(s) in 1 object(s) allocated from: #0 0x70f5d9d6d8cd in operator new(unsigned long) /__w/rocm-libraries/rocm-libraries/TheRock/compiler/amd-llvm/compiler-rt/lib/asan/asan_new_delete.cpp:109:35 #1 0x70f5d8a2d48d in TensileLite::ContractionProblemGemm::ContractionProblemGemm(TensileLite::ContractionProblemGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x22d48d) #2 0x70f5d8a93727 in TensileDataGemm::TensileDataGemm(TensileDataGemm const&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x293727) #3 0x70f5d89c045b in initTensileGemmData(_rocblaslt_handle*, rocblaslt::RocGemmType, hipblasOperation_t, hipblasOperation_t, hipDataType, hipDataType, hipDataType, hipDataType, rocblaslt_compute_type_, unsigned long, std::shared_ptr<void>&) (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x1c045b) #4 0x70f5d895bc78 in rocblaslt_matmul_desc_create (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x15bc78) #5 0x70f5d8becf3a in hipblasLtMatmulDescCreate (/__w/rocm-libraries/rocm-libraries/build/lib/libhipblaslt.so.1+0x3ecf3a) #6 0x63ceaafe903e in std::_Function_handler<void (), testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType)::'lambda17'()>::_M_invoke(std::_Any_data const&) matmul_gtest.cpp #7 0x63ceaae5e765 in catch_signals_and_exceptions_as_failures(std::function<void ()>, bool) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x4df765) #8 0x63ceaaf62403 in testing_matmul_with_bias(Arguments const&, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType, hipDataType) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5e3403) #9 0x63ceaaf501b8 in testing_matmul(Arguments const&) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x5d11b8) #10 0x63ceab034c8f in (anonymous namespace)::rocroller_predicate_test_unrollXYK_Test::TestBody() matmul_gtest.cpp #11 0x63ceaba7023c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/__w/rocm-libraries/rocm-libraries/build/bin/hipblaslt-test+0x10f123c) SUMMARY: AddressSanitizer: 5098 byte(s) leaked in 28 allocation(s). ``` </details> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests. ## Risk level Low (2/5): changes test-client descriptor ownership; matrix-multiplication library behavior is unchanged.
1 task done
Contributor
TheRock Submodule Bump ActivityNewest first
|
tony-davis
added a commit
that referenced
this pull request
Nov 6, 2026
… them (#14608) JIRA ID : AIHPBLAS-5352 ## Motivation `hipblaslt-test`, the hipBLASLt test client, keeps the host buffers that each test frees in a pool, so the next test can reuse them instead of allocating again. These buffers are pinned (allocated with `hipHostMalloc`, so the operating system can neither page them out nor reclaim them), and the pool kept every one until the process exited. On the gfx1250 hardware test host, which has about 499 GiB of memory, the 93 cases with 2^31- and 2^32-element sizes enabled in #13504 left about 123 GiB pinned in one process. With 4 CI runners each splitting the suite into 4 parallel test processes, the out-of-memory killer took the runners down on 10-23 and 10-27. Separately, a failed host allocation returns a null buffer, and the test then writes through it. I don't think this stack guarantees the gfx1250 host never runs short. What it does is make the client hold much less pinned memory, give it back sooner, and skip a case with the amounts instead of getting killed, which is the part of the problem the client can fix. If pressure comes back, the 110 GiB container caps from #14569 keep a kill inside one job, and putting the lock back (#14982 removes it) is a one-file revert. So I'm not planning a long soak test before merging. The pool comes from ROCm/hipBLASLt#1202, which reused allocations to speed up `hipblaslt-bench` across about 500,000 small matmuls. Reuse still matters for large buffers: on the test host below, `hipHostMalloc` takes about 0.18 s per GiB and `hipHostFree` about 0.06 s per GiB. ## Technical Details - A pooled host buffer of 1 GiB or more (`kHostPoolRetainLimit` in `d_vector.hpp`) now stays cached only while consecutive tests reuse it. The pool records the test count when a buffer comes back, and `HostPoolReleaseListener`, a new hook that Google Test calls after every test, frees the large buffers that the test just ended did not use. A run of same-shape cases, such as the four transpose combinations (NN, NT, TN, TT) of one size, still shares one allocation. - When a test asks for a large buffer that no cached buffer fits, the pool first frees the large buffers that earlier tests left, then pins new memory. This keeps the peak near one test's footprint when the shape changes. - A host buffer no longer gets the pool's 20% extra capacity when the extra would take it to 1 GiB or more (`host_pool_capacity()`), so a buffer under the limit stays under it. The existing cutoff for skipping the extra is a quarter of device memory, so on gfx1250 nearly every host buffer got it. - `HipHostBuffer::memcheck()` reports a failed pinned allocation, and `testing_matmul` checks every host buffer after allocating, including the four temporary buffers a swizzled A or B is built in. A failure now skips the test with the requested size ("could not allocate N GiB of pinned host memory"). - A pooled host buffer under 1 GiB is released once 100 consecutive tests have not reused it (`kHostPoolSmallIdleTests`). The pool reuses a buffer only for requests between half its size and its size, so without this a run of shrinking requests kept every earlier buffer pinned. - In `hipblaslt-bench`, a failed host allocation now exits with a failure, as a failed device allocation already did, instead of reporting success. - Device memory behaves as before. `hipblaslt-bench` does not install the listener, so it never frees large buffers early; it does see the 20% change for host buffers of 1 GiB or more. The hipBLASLt library itself is unchanged. This is the first of four stacked PRs for AIHPBLAS-5352. #14609 makes the client's host-memory query respect container memory limits, #14610 runs the memory preflight for every matmul case, and #14982 drops the host-wide lock in the gfx1250 workflow. ## Test Plan - `HostMemory_pre_checkin` (new, `clients/tests/src/host_memory_gtest.cpp`, 6 tests; the `_pre_checkin` suffix puts it in the standard tier the PR checks run). The tests check that a large buffer survives one test boundary and is freed after the next, that a same-shape test reuses it (what the first test wrote is still there), that a large miss frees two buffers earlier tests left, that a 1 GiB request gets exactly 1 GiB, that a small buffer is released after 100 tests that do not reuse it (and kept when one does), and that a failed allocation leaves a null buffer whose size is reported. A 0.9 GiB request gets exactly 0.9 GiB (the old rule gave it 1.08 GiB, over the limit). The tests skip, naming the buffer, on a host that cannot pin that much, and the reuse test skips on a Windows host where the pool frees its idle buffers before every request. A temporary variant without the small-buffer aging failed the new test. - To confirm the tests can fail, I built two temporary variants of the pool. One undid four behaviors at once (no release at the end of a test, no release on a miss, the 20% extra restored, `memcheck()` always succeeding), and it failed 4 of the 5 tests. The other freed large buffers whether or not a test used them, and it failed 3. Every test failed under at least one variant. - The 82 large-memory cases that apply to gfx90a (the five name patterns from #14545), and the `smoke` and `quick` tiers, on develop and on this branch, for wall time and peak resident memory. - Handle leaks: #14714 and #14700 on develop give the matrix layouts, matmul descriptors and HIP events owners, so an early return no longer leaks them. To check that with this stack's new early returns, a preloaded library counted creates and destroys of all three, on the full stack, over 1,892 matmul cases, once normally and once under a 1 MiB host limit so that most cases skip at the preflight. - Adjacent: the `FastCheck_pre_checkin` and `FastCheckDevice_pre_checkin` suites, including `memory_shortfall_reclaims_idle_buffers`, which uses the pool's existing `ReleaseCached()`. `hipblaslt-bench` was built, because it compiles the non-gtest form of the new check, but not run. ## Test Result One MI210 (gfx90a) on a shared host with 2.2 TB of memory. This branch's clients were built against the 10-25 TheRock nightly hipBLASLt gfx90a device library. Each pair of runs went back to back, but the host was busy (load average near 300), so the times carry some noise. | Measurement | develop | this PR | | --- | --- | --- | | 82 large-memory cases, wall time | 316.9 s | 313.8 s | | 82 large-memory cases, peak resident memory | 81.9 GiB | 49.2 GiB | | 82 large-memory cases, mean resident memory | 66.9 GiB | 16.6 GiB | | Resident memory after the last large case | 74 GiB | 2 GiB | | `smoke` + `quick` (9,171 tests), two runs | 66.6 s, 62.0 s | 63.6 s, 64.1 s | - Large-memory cases: 82 of 82 passed on both. `smoke` + `quick`: 9,169 passed and 2 skipped (`GroupedMatmul.smoke_Fp16NN` and `smoke_Bf16NN`, which skip on this GPU) on both. - `HostMemory_pre_checkin`, `FastCheck_pre_checkin` and `FastCheckDevice_pre_checkin`: 64 of 64 passed. - A first version that freed every large buffer at the end of each test took 449.6 s for the large cases against 318.2 s on develop, which is why the pool keeps a buffer while the next test reuses it. - Handle leaks: 0 layouts, descriptors or events left, both normally (13,580 created) and with 409 cases skipped at the preflight (11,235 created). - With the small-buffer aging, `smoke` + `quick` passed 9,225 tests in 59.3 s with peak resident memory 2.18 GiB. - gfx1250: this PR alone passed the hardware workflow's hipBLASLt tests in [40934683157](https://github.com/ROCm/rocm-libraries/actions/runs/40934683157), and Archana dispatched the full stack in [41582794047](https://github.com/ROCm/rocm-libraries/actions/runs/41582794047), which passed hipBLASLt twice with no kills and no memory skips. Her run on the rebased stack, [41717825380](https://github.com/ROCm/rocm-libraries/actions/runs/41717825380), was clean too. All three kept the host-wide lock; #14982 covers the run without it. - Automated GitHub checks are pending. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests. ## Risk level Medium (3/5). Every `hipblaslt-test` case passes through the new listener and the host pool, so a mistake here would affect the whole suite. The hipBLASLt library and device behavior are unchanged; only the test and bench clients change. The bench sees the 20% change near 1 GiB, and now exits with a failure when a host allocation fails. ## Related - Work tracking: [AIHPBLAS-5352](https://amd-hub.atlassian.net/browse/AIHPBLAS-5352), fixes 1 (pool retention) and 4 (null host buffer) of its list. - Cause: #13504. CI workaround this work is meant to replace: #14545, merged on 10-28. - Where the pool came from: ROCm/hipBLASLt#1202. Windows already clears the pool on smaller-memory hosts for the same reason: #4793, #6121. ## Device / Architecture Coverage Host-side test-client code only, with no kernel, library or kernel-selection change, so it does not depend on the GPU architecture and the standard PR checks are enough. Measured on gfx90a. ## Flags / Guardrails None. The 1 GiB threshold is the constant `kHostPoolRetainLimit`. [AIHPBLAS-5352]: https://amd-hub.atlassian.net/browse/AIHPBLAS-5352?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA ID : N/A
Motivation
The test helper
testing_matmul_with_biasdestroys its matrix-layout and matmul descriptors only at the end of the function. Any early return leaks them, including the expected "NO solution found!" failure inrocroller_predicate_test.unrollXYK. Found while expanding ASAN testing in #14240.Current ASAN CI does not run this test. The expanded-coverage PR #14240 adds it to scheduled standard ASAN runs on gfx950.
Technical Details
Own each descriptor with a
std::unique_ptrand itshipblasLtMatrixLayoutDestroyorhipblasLtMatmulDescDestroydeleter as soon as it is created, and remove the manual destruction at the end of the function. The raw handle vectors remain becauseGroupedGemm::setProblemtakes vectors of raw handles.Test Plan
With a host AddressSanitizer (
HOST_ASAN) client build on gfx950, run from the repository root:ASAN_OPTIONS=quarantine_size_mb=600:detect_leaks=1 \ LSAN_OPTIONS=suppressions=test/therock/lsan.supp \ hipblaslt-test --gtest_filter='*rocroller_predicate_test.unrollXYK*'Test Result
Leaked bytes went from 5,098 to 0.
ASAN error output (before fix)
Submission Checklist
Risk level
Low (2/5): changes test-client descriptor ownership; matrix-multiplication library behavior is unchanged.