Skip to content

fix(cron): kill process group on script timeout, fix mislabeled failure message (#59549) - #59574

Closed
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/59549-cron-timeout-orphans
Closed

fix(cron): kill process group on script timeout, fix mislabeled failure message (#59549)#59574
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/59549-cron-timeout-orphans

Conversation

@webtecnica

Copy link
Copy Markdown
Contributor

Summary

Two bugs in cron/scheduler.py's script execution path:

Bug 1: Mislabeled timeout messages

The error classifier in _format_cron_failure matches "timed out" generically before checking the specific source. Script timeouts (which produce "Script timed out after Ns: /path") contain "timed out" and are reported as provider timeout instead of script timeout. Fix: check for "script timed out" before the generic branch.

Bug 2: Orphaned child processes on timeout

subprocess.run(timeout=N) only kills the direct child when the timeout fires; grandchildren spawned by the script survive as orphaned processes. Migrated to subprocess.Popen + preexec_fn=os.setsid (POSIX) to create a fresh process group, and on TimeoutExpired the entire group is killed via os.killpg(pid, 9).

Changes

File Δ
cron/scheduler.py +21/-9 lines (Popen migration, process group, error message fix)

Tests

  • tests/cron/test_cron_script.py: 38/38 passed

Closes #59549

…re message (NousResearch#59549)

Two bugs:

1. **Mislabeled timeout.** The error-message classifier in
   ``_format_cron_failure`` matches "timed out" generically and reports
   every timeout as "provider timeout". Script timeouts produce "Script
   timed out after Ns: /path" which also contains "timed out", so they
   are mislabeled. Add a specific "script timed out" check before the
   generic provider-timeout branch.

2. **Orphaned child processes.** ``subprocess.run`` with ``timeout=``
   only kills the direct child on timeout; grandchildren survive as
   orphans. Migrate to ``subprocess.Popen`` + ``preexec_fn=os.setsid``
   (POSIX) to create a fresh process group, and on TimeoutExpired kill
   the entire group with ``os.killpg(pid, 9)``.
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists labels Jul 6, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused cron fix. The two reported defects are present on current main: cron/scheduler.py:75-79 classifies the script runner's Script timed out ... result (cron/scheduler.py:2133-2134) as a provider timeout, and the runner uses subprocess.run(..., timeout=...) at cron/scheduler.py:2101-2109.

Problems

  • In the PR timeout handler, cron/scheduler.py:2048 limits cleanup to sys.platform != "win32". The Windows branch returns after proc.communicate(timeout=...) without terminating or reaping proc, so this migration drops direct-child timeout cleanup on a supported platform. Windows 10/11 is Tier 1 in website/docs/getting-started/platform-support.md:20.
  • The PR adds no regression tests. Current tests/cron/test_cron_script.py:184-196 only checks that a timeout is reported; it does not cover descendant cleanup or the delivery classification.

Suggested changes

  • Add a Windows direct-child terminate/kill-and-reap fallback while retaining the POSIX process-group cleanup.
  • Add focused cleanup and classifier regression tests.

Automated hermes-sweeper review.

Comment thread cron/scheduler.py

except subprocess.TimeoutExpired:
# Kill the entire process group to clean up orphaned children (#59549)
if sys.platform != "win32":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch skips every termination action on Windows and immediately returns the timeout result. Please add a Windows direct-child terminate/kill-and-reap fallback here; otherwise the Popen migration leaves the timed-out script process unmanaged on that platform.

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Closing as superseded — both bugs this PR targets are now fixed on main: (1) the mislabeled "provider timeout" for script failures was fixed by #85536 (commit d1fc204) — classification is gated on job mode (provider_reachable = not job.get("no_agent")) before any substring match, and script timeouts get their own message; (2) process-group kill on script timeout landed via #93795 (#85125 Phase 4a, merged 2026-08-27), which tree-kills descendants through agent.deadline.kill_process_tree and closed #59549/#71148. Your PR had the right ordering insight (script-timeout check BEFORE the generic matcher) — thanks for both fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Cron script timeouts leave orphaned background processes and get reported as provider timeouts

4 participants