Skip to content

fix(ci): preload ASan before starting pytest - #8733

Merged
sa-faizal merged 2 commits into
users/safaizal/asan-runtime-discovery-20261003from
users/sa-faizal/pytest-asan-preload
Oct 8, 2026
Merged

sa-faizal merged 2 commits into
users/safaizal/asan-runtime-discovery-20261003from
users/sa-faizal/pytest-asan-preload

Conversation

@sa-faizal

@sa-faizal sa-faizal commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

TensileLite pytest jobs can fail during startup when an imported extension loads ASan too late. Preload the discovered runtime when starting the pytest child, preserving existing preloads, sanitizer settings, and the parent environment.

Reuse the existing environment-prepend test to check preload order and parent-environment preservation; no test methods are added. This PR is stacked on #8721 and contains only the pytest follow-up. Retarget to main after that dependency lands.

Validation:

  • All 45 existing pytest-runner tests passed on Linux and Windows. Formatting and diff checks passed. Current Linux/Windows unit CI passed.
  • On MI325 (gfx942), the current runner with matching host-ASan artifacts collected and passed 25/25 existing timing tests. This validates host-side startup and timing assertions, not GPU kernel execution.
  • The 25 assertions pass, but teardown exits 1 with 679,993 bytes in 587 allocations reported by LeakSanitizer. Controlled Python 3.12.3 baselines reproduce leaks from importing pytest alone (69,553 bytes) and NumPy alone (941,456 bytes). The same 25 tests without shared conftest/plugin loading retain only the pytest baseline. These dependency shutdown failures remain unresolved; no sanitizer suppression was added. rocm-libraries #13055 addresses the separate isolated rocisa import leak.

ISSUE ID : #8735

AI-assisted change.

Fixes #8735

@sa-faizal
sa-faizal added this pull request to stack #8734 October 4, 2026 18:53
@sa-faizal
sa-faizal requested review from dileepr1 and geomin12 October 4, 2026 18:54
@sa-faizal
sa-faizal marked this pull request as ready for review October 4, 2026 18:54
@therock-pr-bot

therock-pr-bot Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@dileepr1 dileepr1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to avoid startup ASan issues.
@sa-faizal Would you have the link to a test run with this change?

@geomin12 geomin12 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

going to let Dileep review this one

@sa-faizal

Copy link
Copy Markdown
Contributor Author

@sa-faizal

sa-faizal commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Pytest started, passed hundreds of tests, and reached 68%. That supports the startup fix working in this run. Full suite failed due to timeout. Increased to an hour and rerun - https://github.com/ROCm/TheRock/actions/runs/37402666539

@sa-faizal
sa-faizal requested a review from dileepr1 October 8, 2026 04:38

@dileepr1 dileepr1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the link.
The startup fix holds though the test is red.
Looks good to me.

@sa-faizal
sa-faizal requested a review from geomin12 October 8, 2026 18:19
@sa-faizal
sa-faizal force-pushed the users/sa-faizal/pytest-asan-preload branch from 2e01ffc to 6823d66 Compare October 8, 2026 18:56
@sa-faizal
sa-faizal removed the request for review from jayhawk-commits October 8, 2026 18:57
@sa-faizal
sa-faizal merged commit c6fcb63 into main Oct 8, 2026
163 checks passed
@sa-faizal
sa-faizal deleted the users/sa-faizal/pytest-asan-preload branch October 8, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[host-ASan] pytest starts Python before the sanitizer runtime is loaded

3 participants