Skip to content

fix(scripts): isolate parallel-runner pytest children from console Ctrl+C on Windows - #57181

Open
lEWFkRAD wants to merge 1 commit into
NousResearch:mainfrom
lEWFkRAD:fix/parallel-runner-ctrlc-broadcast
Open

fix(scripts): isolate parallel-runner pytest children from console Ctrl+C on Windows#57181
lEWFkRAD wants to merge 1 commit into
NousResearch:mainfrom
lEWFkRAD:fix/parallel-runner-ctrlc-broadcast

Conversation

@lEWFkRAD

@lEWFkRAD lEWFkRAD commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Makes full-suite runs of scripts/run_tests_parallel.py survivable on native Windows with the project's Python 3.11 venv.

The runner relied on start_new_session=True to isolate its per-file pytest children — but on Windows that flag only maps to CREATE_NEW_PROCESS_GROUP starting with CPython 3.12 (the comment at the Popen call already admitted this). On 3.11 it is silently ignored, so every child shared the runner's console process group. Combined with os.kill(pid, 0) being a GenerateConsoleCtrlEvent broadcast on Windows (bpo-14484) rather than a liveness probe, a single probe anywhere in the run sprayed KeyboardInterrupt into every concurrent child and the runner itself — observed as the runner dying at 100% completion with ~200 collateral KeyboardInterrupt failures per full-suite run.

The fix passes explicit creationflags=CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW on win32 (each child becomes its own ctrl-event group root on its own invisible console — ctrl-event broadcasts can no longer fan out across children, and per-child conhost flashes disappear as a bonus), keeping start_new_session=True on POSIX where _kill_tree's killpg cleanup needs it.

It also fixes the one unguarded broadcaster the audit found: tests/tools/test_zombie_process_cleanup.py probed its three spawned sleep(60) children with bare os.kill(pid, 0) on all platforms (now psutil.pid_exists(), per the CONTRIBUTING rule), and its cleanup used os.kill(pid, signal.SIGKILL)signal.SIGKILL doesn't exist on Windows (now portable Popen.kill()). The other os.kill(pid, 0) call sites in tests are already win32-skipped; production call sites were migrated previously.

Related Issue

Fixes #57145

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • scripts/run_tests_parallel.py — platform-split isolation kwargs for the per-file pytest Popen: explicit CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW creationflags on win32, start_new_session=True on POSIX.
  • tests/tools/test_zombie_process_cleanup.py — probe via psutil.pid_exists() instead of os.kill(pid, 0); clean up via Popen.kill() instead of the Windows-nonexistent signal.SIGKILL.
  • tests/test_run_tests_parallel.py — new regression test test_children_spawn_in_their_own_process_group asserting the per-platform Popen isolation kwargs.

How to Test

  1. On native Windows with a Python 3.11 venv, run python scripts/run_tests_parallel.py -q — before this change the runner intermittently died near completion with mass collateral KeyboardInterrupt failures whenever a test performed an os.kill(pid, 0) probe; with it, the run completes.
  2. python -m pytest tests/test_run_tests_parallel.py tests/tools/test_zombie_process_cleanup.py -q on both Windows and POSIX.
  3. python scripts/check-windows-footguns.py --all stays green.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass — full suite via scripts/run_tests_parallel.py on native Windows now completes end-to-end (before this fix the runner itself died near 100% with ~200 collateral KeyboardInterrupt failures). The remaining failures are pre-existing native-Windows failures unrelated to this change: spot-checked files (e.g. tests/tools/test_file_tools.py, tests/tools/test_local_background_child_hang.py) reproduce identical failure counts under plain python -m pytest with no runner involved. Targeted suites for the touched files are green: tests/test_run_tests_parallel.py (all but one pre-existing Windows failure tracked in run_tests_parallel.py on Windows: ':' path-list split breaks drive letters; per-file progress lines lost to UnicodeEncodeError on piped stdout #57149), tests/tools/test_zombie_process_cleanup.py (13/13).
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11 Pro (native, no WSL), CPython 3.11.15

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — the Popen comment block documents the trap in place
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — POSIX behavior is byte-identical (start_new_session=True as before)
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A

…rl+C on Windows

scripts/run_tests_parallel.py relied on start_new_session=True for child
isolation, but on Windows that flag only maps to CREATE_NEW_PROCESS_GROUP
in CPython 3.12+ — on 3.11 it is silently ignored, so every per-file
pytest child shared the runner's console process group. os.kill(pid, 0)
on Windows is not a no-op: it routes through GenerateConsoleCtrlEvent
(bpo-14484), so a single liveness probe during a test broadcast
KeyboardInterrupt to every concurrent child AND the runner itself
(observed: runner died at 100% completion with ~200 collateral
KeyboardInterrupt failures per full-suite run).

- Pass creationflags=CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW
  explicitly on win32 (each child is its own ctrl-event group root, on
  its own invisible console); keep start_new_session=True on POSIX.
- Fix the one unguarded os.kill(pid, 0) prober that fires on Windows:
  tests/tools/test_zombie_process_cleanup.py now probes via
  psutil.pid_exists() and cleans up via Popen.kill() (the old cleanup
  used signal.SIGKILL, which does not exist on Windows).
- Add a regression test asserting the Popen isolation kwargs per
  platform.

Found during the PR NousResearch#57066 / issue NousResearch#57068 triage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for tracing the Windows process-group failure; current main still has the unsafe launch at scripts/run_tests_parallel.py:257-268 and the unguarded probe at tests/tools/test_zombie_process_cleanup.py:23-29.

Problems

  • The audit misses tests/test_live_system_guard_self_test.py:284-295: it bypasses the safety guard and calls os.kill(os.getpid(), 0). Default runner discovery includes this file (scripts/run_tests_parallel.py:117-161), while CONTRIBUTING.md:686-709 identifies signal-0 calls as Windows Ctrl+C broadcasts. The PR should make this test Windows-safe too.
  • The new comments say start_new_session changes behavior in CPython 3.12. CPython's 3.11 and 3.12 source both mark it POSIX-only; please describe it as such rather than version-gating the behavior.
  • The regression assertion enters the Windows branch only on a native Windows host, while CI's test matrix is Ubuntu-only (.github/workflows/tests.yml:41-45).

Suggested changes

  • Finish the signal-0 audit and skip or replace the remaining live-system-guard probe on Windows.
  • Correct the CPython explanation and make the Windows kwargs branch testable on Linux CI.

Automated hermes-sweeper review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low — cosmetic, nice to have platform/windows Native Windows-specific behavior or breakage sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: parallel test runner children share the console process group — one os.kill(pid, 0) probe broadcasts KeyboardInterrupt across the whole run

3 participants