Skip to content

test(miopen)[ALMIOPEN-3097]: destroy find-2.0 solutions in na_*find2 tests and re-enable them under ASAN - #15016

Open
aalabsi-amd wants to merge 2 commits into
developfrom
users/aalabsi-amd/miopen-find2-tests-destroy-solutions
Open

aalabsi-amd wants to merge 2 commits into
developfrom
users/aalabsi-amd/miopen-find2-tests-destroy-solutions

Conversation

@aalabsi-amd

@aalabsi-amd aalabsi-amd commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

JIRA ID : ALMIOPEN-3097

Motivation

Under ASAN, miopen_gtest exits 1 because LeakSanitizer reports leaks at exit from the find-2.0 batch-norm +
activation tests. Each gpu() verifier calls miopenFindSolutions, runs the returned solutions, and never calls
miopenDestroySolution. Every direct leak points at miopenFindSolutions (src/api/find2_0_commons.cpp:278,
new miopen::Solution{...}). The assertions pass; the leak is in the tests, not the library.

Because of this, #13107 excluded both files from ASAN builds (SKIP_TESTS in test/gtest/CMakeLists.txt), which
drops all 558 cases of */GPU_na_train_find2_* and */GPU_na_inference_find2_* from every ASAN build.

Technical Details

  • test/gtest/na_train_find2.cpp (4 verifiers) and test/gtest/na_inference_find2.cpp (1 verifier): a FindSolutions() helper wraps every solution returned by miopenFindSolutions in MIOPEN_MANAGE_PTR(miopenSolution_t, miopenDestroySolution), so each is destroyed when gpu() returns, including on an early fatal assertion. solutions_found is now initialized (it was read uninitialized if find failed).
  • test/gtest/CMakeLists.txt: remove the ALMIOPEN-3097 SKIP_TESTS entries, so ASAN builds compile and run both
    files again.

find_2_conv.cpp stays in the ASAN skip list; its leak has not been investigated and is out of scope here.

Test Plan

Local ASAN build of this branch against the rockrel Multi-Arch Release ASAN (nightly) #139 artifacts
(run 40425965688, TheRock e73fd19), -fsanitize=address -shared-libasan, THEROCK_SANITIZER=ASAN,
GPU_TARGETS=gfx942:xnack+; test environment as TheRock CI configure_asan_env.py
(ASAN_OPTIONS=detect_odr_violation=0:quarantine_size_mb=600:verify_asan_link_order=0:symbolize=1, HSA_XNACK=1,
leak detection on):

miopen_gtest --gtest_filter='*/GPU_na_train_find2_*:*/GPU_na_inference_find2_*'

Test Result

Before (develop, ASAN, PR #13107 canary run) After (this branch)
na_*find2 cases run under ASAN none in ASAN builds (excluded via SKIP_TESTS); the Smoke cases passed in the #13107 canary run, but the process failed at exit 558 passed, 0 failed, 0 skipped (12 suites, Smoke + Full)
LeakSanitizer at exit 47556 byte(s) leaked in 452 allocation(s): direct leaks from miopenFindSolutions <- na_train_find2.cpp:534/710, na_inference_find2.cpp:164 no report, process exit code 0
AddressSanitizer errors 0 0

Run: Slurm job 68262003 on banff-cyxtera-s83-2 (MI300X, gfx942:sramecc+:xnack+, amdgpu.noretry=-1), 73 min;
LSan suppressions file loaded but unused (Suppressions used never printed).

Second run:

  • Slurm job 68298368 on banff-cyxtera-s70-4 (gfx942:sramecc+:xnack+): 558 passed, 0 failed, exit code 0, and no
    LeakSanitizer or AddressSanitizer output

Submission Checklist

…tests and re-enable them under ASAN

The batch-norm + activation find-2.0 tests called miopenFindSolutions and
miopenRunSolution but never miopenDestroySolution, so LeakSanitizer failed
ASAN runs at exit. Destroy each solution after running it, and drop the
ALMIOPEN-3097 ASAN SKIP_TESTS entries for na_inference_find2.cpp and
na_train_find2.cpp.
@therock-pr-bot

therock-pr-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass PR does not contain code files — Unit Test auto-passed
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@JonathanLichtnerAMD JonathanLichtnerAMD left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Each of the 5 verifiers now destroys the solution returned by miopenFindSolutions, and nothing between find and the run loop can return early. Optional nit: an RAII deleter would keep this leak-proof if someone later adds an ASSERT_* before the loop. No suggested fixes.

Co-authored by Claude (AI).

…find2 tests

Review follow-up: wrap solutions in MIOPEN_MANAGE_PTR(miopenSolution_t,
miopenDestroySolution) via a FindSolutions() helper that takes ownership
right after miopenFindSolutions, so a later ASSERT_* or early return
cannot leak them. Also initializes solutions_found, which was read
uninitialized when the find call failed.
@aalabsi-amd

Copy link
Copy Markdown
Contributor Author

LGTM. Each of the 5 verifiers now destroys the solution returned by miopenFindSolutions, and nothing between find and the run loop can return early. Optional nit: an RAII deleter would keep this leak-proof if someone later adds an ASSERT_* before the loop. No suggested fixes.

Co-authored by Claude (AI).

Done in 0dc7dfc: solutions are now owned by MIOPEN_MANAGE_PTR(miopenSolution_t, miopenDestroySolution) via a
FindSolutions() helper that wraps them right after miopenFindSolutions, so an early ASSERT_*/return can't leak
them. Also initialized solutions_found. Re-ran under ASAN: 558/558 passed, no LSan report.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants