fix(cron): tree-kill script timeout descendants via agent.deadline.kill_process_tree (#85125 4a, salvage of #86791) - #93795
Conversation
…ll_process_tree The script-timeout path used a site-local process-group kill, which cannot reach a grandchild that created its OWN session (start_new_session background jobs, watchdogs). Such descendants kept running after the job reported failure (NousResearch#71148, NousResearch#59549). Migrate the timeout handler to the unified deadline layer's kill_process_tree (NousResearch#85147, d6a5cb9): psutil snapshots the descendant set before signalling, so own-session grandchildren are reached too. Fallback to the site-local group kill if the import ever fails, so the path cannot re-wedge. The explicit script-timeout message stays the classification anchor (NousResearch#85536's contract), keeping cron timeouts distinct from provider timeouts. Salvage additions on review (NousResearch#85125 Phase 4a): - migrate the sibling kill site too — the cancel_event/"ownership was lost" path orphaned setsid grandchildren the same way (whole-bug-class rule); pinned by test_cancel_path_also_tree_kills - proc.poll() early-return in _terminate_cron_script_tree so a script that exits right at the deadline doesn't log a spurious "no signal" warning (mirrors _terminate_cron_script_process); pinned by test_already_exited_proc_is_left_alone - acceptance test's script timeout 1s -> 2s: interpreter startup under CI load could eat the whole 1s window before the spawner wrote its pid file - note: kill_process_tree hard-kills (SIGKILL) immediately, whereas the old path gave a 1s SIGTERM grace window; intended for a deadline- expiry hard stop (both docstrings say "hard stop") Based on NousResearch#86791 by @ayushnangia; cherry-picked to preserve authorship. Co-authored-by: dante32683 <dante32683@users.noreply.github.com> Co-authored-by: supotato-ipj <supotato-ipj@users.noreply.github.com>
Overall: fixes a real bug class (own-session grandchildren surviving cron script timeouts) by routing through
|
Pull request was closed
#85125 Phase 4a. A cron script timeout used a site-local process-group kill that cannot reach a grandchild running in its OWN session (
start_new_sessionbackground jobs, watchdogs) — descendants kept running after the job reported failure (#71148, #59549). The timeout path now goes through the unified deadline layer'sagent.deadline.kill_process_tree(#85147): psutil snapshots the descendant set BEFORE signalling, so own-session grandchildren are reached too.Based on #86791 by @ayushnangia — cherry-picked to preserve authorship (with his Co-authored-by credits to @dante32683 (#59379) and @supotato-ipj (#59549 evidence) kept). Salvage round folds four review findings into the same commit:
cancel_event/ "cron fire ownership was lost" path killed the script with the same killpg-only helper and orphaned setsid grandchildren identically. Both kill sites now tree-kill; pinned bytest_cancel_path_also_tree_kills.proc.poll()early-return in the new helper, mirroring_terminate_cron_script_process: a script that exits right at the deadline no longer produces a spurious "reported no signal" WARNING; pinned bytest_already_exited_proc_is_left_alone.agent.deadlinenow logs as what it is instead of masquerading as a kill failure (import stays function-local so tests can monkeypatch).Behavior note:
kill_process_treehard-kills (SIGKILL) immediately, whereas the old timeout path gave a 1s SIGTERM grace window. For a deadline-expiry hard stop this is intended — both docstrings say "hard stop" — but it is a semantics change worth stating.Verification
grandchild pid 62499 survived the script timeoutand passes with the fixlive_system_guard_bypassmarker verified registered in tests/conftest.py; psutil verified a pinned core dependencyruffclean;tydiff-clean (3 stub-typing errors fixed via cast)Closes #86791. Fixes #71148. Fixes #59549.
Part of #85125 (Phase 4a — delegated item, implementation + acceptance cell by @ayushnangia).