Skip to content

[Testing] Clean up concurrent test groups when startup fails - #12625

Open
0z5a wants to merge 2 commits into
triton-lang:mainfrom
0z5a:0z5a/concurrent-startup-cleanup
Open

0z5a wants to merge 2 commits into
triton-lang:mainfrom
0z5a:0z5a/concurrent-startup-cleanup

Conversation

@0z5a

@0z5a 0z5a commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #12624.

When the second concurrent test process cannot start, _concurrent currently leaves the first running. Similarly, an exception while waiting can leave other pytest processes alive. Start each command in its own session, and on an exception kill its still-owned process group and reap each started process before propagating the error. This also reaches pytest workers in the group.

On the main thread, temporarily handle default SIGTERM and SIGHUP so those signals take the same cleanup path; restore the previous handlers afterward. Existing custom or ignored signal handlers are left alone. SIGKILL and processes that deliberately leave their group cannot be handled by this mechanism. This change is separate from #12360, which fixes aggregation of negative child exit statuses.

Testing

  • pytest -s --tb=short python/test/unit/runtime/test_driver.py -k 'runner or concurrent': 18 passed (CPU; CUDA_VISIBLE_DEVICES empty). The new cases cover partial startup with a real child and grandchild, a wait exception, the normal success path, and SIGTERM delivered to the runner.
  • On unmodified main, the two startup/wait cleanup regression tests failed, while the success control passed. Reproduction processes were bounded and cleaned up.
  • pre-commit run --from-ref origin/main --to-ref HEAD: passed.

Tests used the current Python source with an existing Triton 3.9 native extension built from an earlier commit; no compiler rebuild or GPU execution was needed for this Python-only change.

New contributor declaration

  • I am not making a trivial change, such as fixing a typo in a comment.
  • I have written a PR description following the contribution rules.
  • I have run pre-commit run --from-ref origin/main --to-ref HEAD.
  • I have added tests under python/test/.
  • I have not added any lit tests.

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
@0z5a
0z5a requested a review from ptillet as a code owner November 4, 2026 17:12
Wait for each group leader without reaping it until all owned groups have
been stopped, so a previously exited leader cannot leave its workers behind
or have its process-group ID reused during cleanup. Also defer default
termination signals while registering children and stopping and reaping
those groups.

On normal completion, stop any remaining workers before reaping leaders.
Require the default SIGCHLD disposition so children are not auto-reaped.

Test Plan: CPU-only pytest runner/concurrent selection (28 passed); all
applicable pre-commit hooks passed using the cached offline environments.

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Testing] Clean up already-started concurrent test processes when startup fails

1 participant