fix(cron): tree-kill script timeout descendants via agent.deadline.kill_process_tree

[ayushnangia]

Summary

Phase 4a of #85125 (delegated): a cron script timeout must leave zero living descendants. The previous timeout handler used a site-local process-group kill (_terminate_cron_script_process), which cannot reach a grandchild that created its own session — start_new_session background jobs, watchdog chains. Those survived the reported failure and kept running (#71148, #59549 — @supotato-ipj's production trace of the 3600s-timeout mislabeling is the same class).

Changes

• cron/scheduler.py::_run_job_script: the timeout branch now calls agent.deadline.kill_process_tree(proc.pid) (landed with feat(agent): unified deadline layer — bounded execution primitive + timeout resolver (#85125 Phase 1) #85147) — psutil snapshots the descendant set before signalling, so own-session grandchildren are reached, with PID-recycling safety. Falls back to the site-local group kill if the import ever fails, so the path cannot re-wedge.
• The explicit Script timed out after Ns message stays the classification anchor (the fix(cron): no_agent and script failures stop masquerading as provider errors #85536 contract), keeping script timeouts distinct from provider timeouts.
• End-to-end regression test in tests/cron/test_cron_script.py: the script spawns a start_new_session grandchild (DEVNULL stdio so it can't hold the runner's pipes), the timeout fires at 1s, and the test asserts the grandchild pid is dead. Red on main before this change, green with it.

Validation

• scripts/run_tests.sh tests/cron/ → 684 passed, 0 failed, 1 skipped
• Red proof: with the fix stashed, the new test fails (grandchild survives)

Credit

• start_new_session=True mechanism: @dante32683 (fix(cron): script timeouts leave orphaned subprocess groups and get mislabeled as provider timeouts #59379) — Co-authored-by
• Production evidence ([Bug]: Cron script timeouts leave orphaned background processes and get reported as provider timeouts #59549): @supotato-ipj — Co-authored-by
• Classification contract: @jbagdonas's fix(cron): classify script timeouts separately #82460 via teknium1's fix(cron): no_agent and script failures stop masquerading as provider errors #85536

Closes #71148 and #59549's orphan half. The Phase 4 conformance-harness copy of the acceptance cell stacks behind #83964 per the #85125 ordering condition.

[ayushnangia]

CI note on the slice-11 failure: the slice log shows 2765 passed, 0 failed with a collection/import error on exactly one file (tests/cron/test_cron_no_agent.py) — that file collects cleanly here (16 tests) and the full tests/cron/ suite is green locally on this branch (684 passed). The import in this diff is lazy (inside the timeout branch), so it can't break test collection. Rebased onto fresh main and re-triggered; looks like a slice-worker flake, re-run should clear it.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

PR: fix(cron): tree-kill script timeout descendants via agent.deadline.kill_process_tree

1. The xfail cell contradicts its own docstring — tests/conformance/persistence/test_phase4_tree_kill_cell.py is still xfail(strict=False) and its reason text says _run_job_script uses subprocess.run(timeout=...), but the scheduler.py fix in this same PR replaces the timeout kill with kill_process_tree — so the cell should XPASS here and, per its own docstring, be "promoted to hard-green in that same PR". It wasn't. Either promote it to a hard assertion now (it is the acceptance cell for exactly this change), or the xfail reason is stale on both counts.
2. Confirm kill_process_tree includes the root pid — the old _terminate_cron_script_process killed the script's own process group (direct child included). If agent.deadline.kill_process_tree only walks descendants, the direct child survives the timeout and _drain_script_pipes(proc) may block on its still-open pipes. The fallback path only triggers on an exception, not on a partial kill — worth asserting the direct child is also gone in the new tests (the suite asserts grandchild death, not the child's).
3. Dead code in the conformance writer — test_cell1_prefix_durability.py: db_path = Path(sys.argv[0]) if False else Path({db_path!r}) leaves a dead if False branch; drop it for clarity.
4. Windows semantics of the liveness probes — the phase-4 cell and _harness.py rely on os.kill(pid, 0) and SIGKILL-style kills; os.kill(pid, 0) behavior differs on Windows (it can raise OSError for invalid handles, and signal semantics differ). Verify these cells are POSIX-only or gated, or they will fail on the Windows lane.

[ayushnangia]

Thanks for the review — point 1 is right and it's now fixed on the current head: the cell is promoted to a hard assertion (the xfail + stale docstring are gone), per its own acceptance conditions. The promotion also surfaced two real gaps which the head addresses: the cell now skips when psutil is absent (kill_process_tree's descendant snapshot depends on it — without it the path silently degraded to the group kill, which is exactly how own-session grandchildren get orphaned), and the timeout fallback now logs instead of degrading silently. The whole branch is also rebased onto current main (~500 commits behind before). Other points from the review: happy to look at the rest if any still apply after the rebase.

[kshitijk4poor]

Salvaged into #93795 — your commit cherry-picked with authorship + both Co-authored-by trailers preserved. Verified red-on-main empirically (grandchild pid survived on upstream/main, zero survivors with the fix). Review round folded in: the cancel_event/'ownership was lost' kill site was the same bug class (killpg-only, orphans setsid grandchildren) so it now tree-kills too; proc.poll() early-return kills a spurious warning when the script exits right at the deadline; the import moved out of the kill try so packaging errors log honestly; acceptance-test timeout 1s→2s for CI headroom. One semantics note documented in the PR: kill_process_tree SIGKILLs immediately vs the old 1s SIGTERM grace — intended for a hard stop. Closing in favor of #93795. Phase 4a delivered — nice work on the acceptance cell.