fix(cron): reap the whole process group when a script job times out - #68262
Closed
Ait0u5hi wants to merge 1 commit into
Closed
fix(cron): reap the whole process group when a script job times out#68262Ait0u5hi wants to merge 1 commit into
Ait0u5hi wants to merge 1 commit into
Conversation
_run_job_script ran no_agent cron scripts via subprocess.run(timeout=), which SIGKILLs only the direct child on timeout. Backgrounded grandchildren are orphaned, and a child wedged in uninterruptible D-state hangs the reap entirely, leaking stuck processes (the cron heartbeat recovers the claim, not the OS process). Run the script in its own session/process group (start_new_session=True on POSIX) and, on TimeoutExpired, os.killpg the whole group; Windows keeps proc.kill(). Converts subprocess.run to Popen+communicate(timeout=); the (bool, str) return contract and secret-redaction path are unchanged. Adds a POSIX-guarded regression test asserting the timeout path kills the group.
alt-glitch
added
type/bug
Collaborator
Related to #59379, which addresses the same timeout process-group leak and also corrects failure classification. This PR is the focused cleanup slice; maintainers can choose the preferred scope. |
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Cron
no_agentscript jobs are run viasubprocess.run(argv, timeout=script_timeout)in_run_job_script. On timeout,subprocess.runSIGKILLs only the direct child — any process the script backgrounded is orphaned, and a child wedged in uninterruptible D-state (heavy I/O) hangs the reap entirely. The cron heartbeat/claim-TTL recovers the claim, but not the leaked OS process, so stuck processes accumulate. Observed on a Jetson/eMMC box where scripts doing heavy I/O against a largestate.dboccasionally exceedscript_timeout.Related Issue
Fixes # (no existing issue — happy to open one if preferred)
Type of Change
Changes Made
cron/scheduler.py—_run_job_script: run the script in its own session/process group (start_new_session=Trueon POSIX) and, onTimeoutExpired,os.killpg(os.getpgid(proc.pid), SIGKILL)the whole group instead of just the child. Windows keepsproc.kill(). Convertssubprocess.run→Popen+communicate(timeout=); the(bool, str)return contract and the secret-redaction path are unchanged.signal.SIGKILLviagetattr(..., signal.SIGTERM)+# windows-footgun: ok.tests/cron/test_scheduler_process_group_reaping.py— POSIX-guarded regression test.How to Test
python3 -m py_compile cron/scheduler.pypytest tests/cron/test_scheduler_process_group_reaping.py -qsleep 300 &then blocks past its timeout — after the fix,pgrep -g <pgid>is empty (grandchild reaped); before, it leaked.Checklist
Code
fix(cron):)scripts/check-windows-footguns.py cron/scheduler.py— clean