Skip to content

ci: serialize ROCgdb GPU tests - #9688

Closed
ethanwee1 wants to merge 1 commit into
mainfrom
users/ethanwee/ci-stabilize-rocgdb
Closed

ethanwee1 wants to merge 1 commit into
mainfrom
users/ethanwee/ci-stabilize-rocgdb

Conversation

@ethanwee1

@ethanwee1 ethanwee1 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Scope clarification

This is not a Python 3.15 or shared-libpython failure. The failure came from TheRock #8394 CI, where the ROCgdb GPU job used Python 3.12.16 and the ROCgdb-specific no_rocm_image_ubuntu24_04_rocgdb test container.

The manylinux/Python 3.15 image is a separate, completed dependency: TheRock #9633 published digest sha256:f1f9dd9b..., verified with regular CPython 3.15.2 and shared CPython 3.15.0. #9688 does not modify that image or its shared libpython build.

Observed failure

The gfx94X ROCgdb test job ran:

python ./build/tests/rocgdb/test_rocgdb.py --parallel -f 0.25 --toolchain llvm --tests gdb.rocm

It produced widespread GPU-debug failures, including multi-inferior stress at 8/32 inferiors, stepping/fork/attach tests, and a 30-minute timeout. Cleanup found 33 lingering ROCgdb/GPU test processes.

Proposed change

At the ROCgdb revision pinned by TheRock (003062db3), --parallel sets FORCE_PARALLEL=1, selecting Make's check-parallel target. This PR tests serializing that GPU suite and extending its job timeout; it is a runner/test-stability fix, not Python enablement.

Lumachad correctly requested clarification because the behavior should be reconciled with the current test-runner implementation before merging. This PR remains draft while that review is resolved.

Validation

  • fetch_test_configurations_test: 107 tests passed.
  • Pre-commit and git diff --check passed.
  • Local linter diagnostics are clean.

Tracked in AIPYTORCH-1241.

@therock-pr-bot

therock-pr-bot Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ⚠️ Warning Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: build_tools/github_actions/fetch_test_configurations.py; no test file found
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Nov 2, 2026

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@ScottTodd
ScottTodd requested a review from lumachad November 2, 2026 16:50
@lumachad

lumachad commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

I don't think this is correct. The only thing the parallel invocation does at the moment is parallel compilation, which means we speed things up by pre-compiling everything. Tests are still run serially, one by one. Could you please clarify what problem you ran into?

@lumachad lumachad left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks incorrect. Let's discuss.

@lumachad lumachad added the ci:skip Skip all CI builds/tests for this PR label Nov 2, 2026
@ethanwee1 ethanwee1 closed this Nov 2, 2026
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:skip Skip all CI builds/tests for this PR

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants