fix(cron): contain timed-out cron workers (#18004)

[rodboev]

Summary

Cron inactivity could mark an attempt failed while its nested agent worker still executed. That allowed output, job state, durable execution history, local admission, and process-global cwd cleanup to describe a completed attempt before the worker had stopped.

The timeout path now requests cooperative interruption and retains the active attempt until the submitted worker exits. The existing terminal lifecycle then records the saved timeout once. Built-in ticks, external fires, manual runs, and direct callers share one local admission lease, while matching external fire claims remain live for long-running agent and script work.

Changes

• cron/scheduler.py: joins timed-out workers before terminal handling, preserves the active lease through cleanup, protects shutdown behavior, and snapshots TERMINAL_CWD under its lock.
• cron/jobs.py: adds matching-incarnation fire-claim refresh for live externally fired work.
• cron/scheduler_provider.py and tools/cronjob_tools.py: acquire local admission before durable fire claims.
• Focused cron tests: cover timed workers, late success, provider and tool ingress, claim liveness, shutdown, script heartbeat failure, profile context, and queued cwd writers.

Validation

Scenario	Before	After
Timed-out worker ignores interruption	Terminal state could be recorded while the worker was live.	Terminal handling waits for worker exit and retains the active attempt.
Same job enters through another local path	Admission order differed by caller.	One scheduler lease prevents local overlap.
External agent or script outlives its initial fire-claim TTL	Another replica could reclaim the job.	Only the matching live claim is refreshed until completion.
Gateway shuts down during a timed-out worker	Shutdown could clear claims before worker exit.	The live worker retains ownership until it completes.
Queued workdir writer	It could restore a stale cwd snapshot.	Snapshot and restore both occur under the writer lock.

Test plan

• python -m py_compile cron/scheduler.py cron/jobs.py cron/scheduler_provider.py tools/cronjob_tools.py
• python -m pytest tests/cron/test_claim_job_for_fire.py tests/cron/test_cron_inactivity_timeout.py tests/cron/test_execution_ledger.py tests/cron/test_run_one_job.py tests/cron/test_scheduler.py tests/cron/test_scheduler_provider.py tests/cron/test_script_claim_heartbeat.py tests/cron/test_shutdown_interrupt.py tests/cron/test_terminal_cwd_lock.py tests/tools/test_cronjob_run_immediate.py -v --timeout=0

Not in scope

Python cannot forcibly terminate an arbitrary running thread or external side effect. A supervised process boundary would be required for immediate hard cancellation.

Upstream

Closes #18004.

[alt-glitch]

Addresses #18004. Competes with open PR #18011 (alternate fix for cron inactivity-timeout execution semantics).

[rodboev]

I compared the competing #18011 branch against this PR and pulled in the part it handled better: a cron timeout must not allow the same job to overlap a still-running worker when agent.interrupt() is ignored past the grace window.

This branch is now a hybrid of the original PR and that non-overlap behavior:

1. cron/scheduler.py interrupts as soon as the inactivity timeout trips, then waits a short grace period for cooperative agents to stop before reporting the timeout.
2. If a non-workdir/profile job ignores the interrupt past the grace window, cron now calls agent.close() immediately for hard resource cleanup, records the still-running future by job id, and skips later runs of that same job until the old future exits.
3. If a workdir/profile job ignores the interrupt, cron waits for that worker to finish before restoring process-global context. That avoids returning while TERMINAL_CWD, profile home, or profile-loaded env state could still be in use by the worker thread.
4. Deferred worker completion now clears the timed-out-run registry and reaps stale async auxiliary clients after the worker's event loop is gone.

I also fixed two edge cases from review:

• The workdir/profile decision is based on validated runtime state, so a job whose workdir was deleted and falls back to default behavior does not block indefinitely as a process-context job.
• The hard cleanup path is not deferred for non-context jobs; agent.close() still runs immediately after the interrupt grace expires.

Validation on Windows:

• python -m py_compile cron\scheduler.py: passed.
• pytest tests\cron\test_scheduler.py::TestRunJobSessionPersistence::test_run_job_does_not_hang_when_timeout_interrupt_is_ignored tests\cron\test_scheduler.py::TestRunJobSessionPersistence::test_run_job_waits_for_workdir_job_when_interrupt_is_ignored -v --timeout=0: 2 passed.
• pytest tests\cron\test_scheduler.py -v --timeout=0: 133 passed.
• pytest tests\cron\test_cron_inactivity_timeout.py -v --timeout=0: 11 passed.
• pytest tests\cron -v --timeout=0: 394 passed, 1 skipped, 7 known Windows failures in test_cron_workdir.py::test_tilde_expands and chmod permission assertions in test_file_permissions.py.
• codex review --base origin/main: no discrete regression found.

[bbopen]

Thanks for pushing this forward. I tested this branch against the #18004 shape with a non-cooperative agent, and it looks like the PR fixes the important same-job overlap/state problem: after timeout it interrupts, records the still-running future, and later runs of the same job are skipped while that worker remains active.

One residual behavior seems worth calling out so maintainers can decide whether it belongs here or in a follow-up: for non-workdir jobs, the worker thread can still be alive after cron has returned the timeout failure. In my synthetic probe, run_job() returned the timeout failure, _TIMED_OUT_RUNS contained the job, and a second run was skipped, but the original worker was still running until explicitly released. So this is containment/non-overlap, not a hard cancellation boundary; non-cooperative tool/subprocess/network side effects can still continue after the failure report.

For workdir/profile jobs, this branch takes the opposite safe tradeoff and waits for the worker to finish before restoring process-global context, which avoids context corruption but can block indefinitely if the worker never exits.

That may be the right pragmatic scope for #18004. If maintainers want “timeout means no further side effects are possible” as the stronger contract, I’m happy to open a focused follow-up issue/PR around process isolation or tool/subprocess-level cancellation rather than expanding this PR.

[rodboev]

Thanks for testing the non-cooperative case. I agree with your read: this PR gives cron containment and same-job non-overlap, not a hard cancellation boundary for arbitrary worker side effects.

I rebased the branch on current main and kept that tradeoff explicit in the code/tests:

1. Non-workdir/profile jobs still return the timeout failure after the interrupt grace window if the worker ignores agent.interrupt(). The still-running future stays recorded by job id, so later runs of that same job are skipped until the old worker exits.

2. Workdir/profile jobs take the safer path and wait for the worker before restoring process-global context. That avoids TERMINAL_CWD, profile home, and profile-loaded env state being restored while the worker might still execute tools.

3. The finalizer now avoids handing a timed-out, already-closed agent back through the deferred teardown path.

4. I added the missing profile-tagged regression case beside the existing workdir/non-workdir timeout tests, so the documented profile behavior is enforced directly.

I also dropped the old Docker workflow workaround while rebasing because current main already keeps PR validation from writing the build cache. The PR body now calls out the remaining hard-cancellation gap as follow-up territory for process isolation or tool/subprocess-level cancellation if maintainers want that stronger contract.

[rodboev]

Rebased this branch onto current main and force-pushed the result.

The diff is still scoped to cron/scheduler.py and tests/cron/test_scheduler.py. I also updated the PR body to keep the contract explicit: this PR provides timeout containment and same-job non-overlap, not hard cancellation of arbitrary worker side effects.

Validation rerun on the rebased branch:

• python -m py_compile cron\scheduler.py — passed
• pytest tests\cron\test_scheduler.py tests\cron\test_cron_inactivity_timeout.py -v --timeout=0 — 233 passed, 1 existing warning

[teknium1]

Thanks for addressing a verified cron timeout-containment gap. Current main detects inactivity at cron/scheduler.py:3175, shuts down the monitor executor without waiting at cron/scheduler.py:3182, and raises after only requesting agent.interrupt() at cron/scheduler.py:3205-3211; its ordinary same-job guard is released when run_one_job() returns (cron/scheduler.py:3682-3687).

The PR's timed-out-future registry, cooperative interrupt grace, and workdir/profile wait directly cover those paths. The added tests cover cooperative interruption, non-cooperative same-job containment, no-interrupt behavior, and workdir/profile context handling. GitHub reports the rebased PR as mergeable clean.

Automated hermes-sweeper review.

[rodboev]

Rebuilt this against current main instead of replaying the old scheduler diff. After inactivity, cron requests the existing cooperative interrupt and keeps the worker's admission, context, resources, durable attempt, and cwd protection live until that worker exits. The existing terminal path then records the timeout once, so a late success cannot replace it.

The same local admission path now covers ticks, external fires, manual runs, and direct calls. Matching external fire claims stay live for long-running agent and script work, and gateway shutdown leaves a timed-out worker's claims alone until it actually stops. This remains containment rather than hard cancellation: an ignored interrupt keeps that one attempt live and blocks another run of the same job.

[alt-glitch]

This was generated by AI during triage.

Related: #18011 is the open alternate timeout-semantics repair, while this current head now retains admission, attempt state, claims, and process-global context until the worker exits. The remaining maintainer decision is whether containment is sufficient or a stronger hard-cancellation boundary is required.