fix(gateway): give in-flight cron work its own drain floor

[JoaoMarcos44]

Summary

The shutdown drain detects in-flight cron work and then kills it anyway, reporting drain took 0.00s, timed_out=True, cron_at_start=1, cron_now=1. This gives cron work its own bounded drain floor so a restart no longer destroys a running job.

This is the drain-contract half of #82161 and is complementary to #82195, not a duplicate — zero file overlap. #82195 removes the hermes update self-deadlock that triggered the reported instance. This PR fixes the behaviour the reporter asked for explicitly:

"the drain behavior should be independent of what caused the shutdown" … "The drain phase should wait for in-flight cron jobs to complete (up to some configured timeout) before proceeding with shutdown."

After #82195 alone, a plain systemctl restart hermes-gateway / launchctl kickstart -k / hermes gateway stop with any cron job in flight still kills it at 0.00s. That is what this PR closes.

Root cause

agent.restart_drain_timeout defaults to 0 and governed every class of in-flight work through a single budget in _drain_active_agents().

That default is deliberate and correct for chat turns: _notify_active_sessions_of_shutdown() tells the user the gateway is restarting, and the session is pre-marked resume_pending, so the next message resumes it. Interrupting a chat turn is cheap and recoverable.

A cron run has neither property:

• nobody is waiting on it interactively;
• mark_running_jobs_interrupted() writes it to jobs.json as a permanent failure;
• a recurring job does not retry — compute_next_run() just moves to the next schedule, so the work is lost;
• the failure notice races adapter teardown (the reporter measured Telegram disconnecting 6 ms before the job was marked failed), so no alert is delivered.

Because the budget was shared, the drain hit if timeout <= 0: return snapshot, True before entering the wait loop at all. It never waited a short time — it never waited. Hence drain took 0.00s while simultaneously reporting cron_at_start=1, and a "timed out after 0.0s" warning that printed the configured budget instead of the elapsed one.

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#8b0000', 'mainBkg': '#0a0204', 'primaryTextColor': '#ffccd5', 'primaryBorderColor': '#ff0038', 'lineColor': '#ff0038'}}}%%
graph TD
    A[Shutdown Signal] --> B[stop / drain phase]
    B --> C{In-Flight Work Detected}
    C -->|Chat Turn| D[Announced + resume_pending]
    C -->|Cron Run| E[No User, No Retry]

    subgraph BEFORE ["BEFORE - One Shared Budget"]
        F[restart_drain_timeout = 0] --> G[timeout less-or-equal 0 short-circuit]
        G --> H[Wait Loop Never Entered]
        H --> I[drain took 0.00s / timed_out=True]
        I --> J[Tool Subprocess Killed]
        J --> K[jobs.json: Permanent Failure]
        K --> L[Alert Lost to Adapter Teardown]
    end

    subgraph AFTER ["AFTER - Split Deadlines"]
        M[Chat Deadline = restart_drain_timeout] --> N[Unchanged: Immediate Interrupt]
        O[Cron Deadline = cron_drain_timeout] --> P[Clamp to Watchdog Leash minus Reserve]
        P --> Q[Bounded Grace Window]
        Q --> R{Job Finished?}
        R -->|Yes| S[Normal Completion + Delivery]
        R -->|No| T[Interrupt + Recorded, Cleanup Intact]
    end

    D --> M
    E --> O

Loading

Infographic :

The fix

Cron work drains on its own deadline, independent of the chat budget.

agent.cron_drain_timeout (default 30s; 0 opts out and restores the previous behaviour). Reads from config or HERMES_CRON_DRAIN_TIMEOUT, following the exact loader/parse conventions of the neighbouring restart_after_turn_timeout — including "0 is a deliberate value, blank falls back to the default".

Two design points that make this safe rather than merely longer:

1. The floor is clamped, not trusted. resolve_cron_drain_budget() caps the configured value at watchdog_delay − elapsed − CRON_DRAIN_CLEANUP_RESERVE_S. The shutdown watchdog hard-exits at restart_drain_timeout + 60s, and TimeoutStopSec is generated as max(60, drain + 30) from the same knob. Waiting past that leash would swap a job that is killed and recorded for one that is SIGKILLed mid-write and left wedged at last_status=running forever — strictly worse than the bug being fixed. The 10s reserve is the post-drain teardown: interrupt agents, process_registry.kill_all(), mark_running_jobs_interrupted(), disconnect adapters. With shipped defaults the effective budget is 30s inside a 60s leash.

2. Bounded, so it cannot deadlock. A cron job that triggers its own restart (the reporter's hermes update case) still loses after the floor expires. Unlike restart_after_turn_timeout's 6-hour cap, this can never wedge the shutdown.

The floor only ever extends the wait — resolve_cron_drain_budget() never returns less than drain_timeout, so an operator who deliberately configured a long restart_drain_timeout keeps it.

Secondary cleanups, both from the report

• The timeout <= 0 special case is removed. An already-expired deadline expresses "interrupt immediately" naturally, so timed_out is always computed from real state instead of asserted up front. This deletes a branch rather than adding one.
• The drain-timeout warning now logs the elapsed wait instead of the configured budget. "Gateway drain timed out after 0.0s" was the line that made this look like a phantom.

Behaviour preserved

• Chat-only shutdowns are unchanged. restart_drain_timeout: 0 still interrupts chat turns immediately — covered by a dedicated test asserting the drain returns in < 1s even with a 30s cron floor configured.
• _drain_active_agents(timeout) with one argument keeps pre-Gateway drain exits after 0.00s with in-flight cron job, killing it mid-run #82161 semantics (cron_timeout defaults to timeout), so every existing caller and test is untouched.
• /update bypasses drain and interrupts in-flight cron tool work #60432: cron work is still counted and still marked interrupted on forced kill.
• [Bug]: Planned-restart fires while cron delivery is in-flight — message silently dropped #58818: cron delivery stays cooperative with the event loop; no blocking join added.
• Gateway restart drain includes the requesting turn in its wait set — active turns force-killed at the 180s timeout #77184: the in-band after-turn wait is untouched.
• No public signature changes; all new symbols are additive.

Validation

Suite	Result
tests/gateway/test_cron_drain_floor.py (new, 13 tests)	13 passed
Drain suites: test_cron_active_work_drain, test_update_cron_drain, test_restart_drain, test_restart_after_turn, test_api_server_active_work_drain, test_cron_shutdown_drain	39 passed
Restart-helper dependents: test_gateway_shutdown, test_external_drain_control, test_restart_resume_pending, test_restart_notification, test_stuck_loop, test_restart_redelivery_dedup, test_restart_service_detection	74 passed
tests/cron/test_shutdown_interrupt.py, tests/hermes_cli/test_gateway_service.py	14 passed, 1 skipped
Ruff on all changed files	clean
git diff --check	clean

New coverage includes the exact repro (0s drain + cron in flight now waits and completes), the bounded-floor guarantee, the chat-only no-regression case, watchdog clamping, and parse/opt-out edge cases.

One existing assertion changed: test_in_flight_cron_job_marked_interrupted_on_forced_kill now sets _cron_drain_timeout = 0.01 alongside _restart_drain_timeout = 0.01, since forcing the interrupt path requires expiring both budgets. The behaviour it asserts is unchanged.

Known gap, deliberately out of scope

The undelivered-notification race the reporter noted (adapters disconnect before the cron failure is written) is a separate defect in delivery ordering. This PR reduces its blast radius — jobs that finish inside the floor deliver normally — but does not reorder teardown. Worth its own issue.

Relates to #82161.

[JoaoMarcos44]

Tracking issue #82279 explains how this PR relates to the other two fixes from the #82161 investigation (#82195, #82224, #82244) — what each one covers on its own, why none of them supersedes another, and the suggested merge order.

[spfcraze]

This was generated by AI during triage.

Summary:
PR #82282 implements the same fix as this PR — a 30s drain floor for in-flight cron work under the default restart_drain_timeout: 0 — for the same issue, #82161.

Problems:

• fix(gateway): drain timeout=0 must still wait for in-flight cron/API work #82282's description: "Apply a minimum drain budget (30s) when the only active work is background (cron/API) and the configured timeout is 0. Chat sessions still get the immediate drain behavior when timeout=0." — the same root cause this PR fixes (the timeout <= 0 early return in _drain_active_agents), the same 30s floor, under the same default config.
• Gateway drain exits after 0.00s with in-flight cron job, killing it mid-run #82161's expected behavior — "The drain phase should wait for in-flight cron jobs to complete (up to some configured timeout)" — is what both PRs implement; fix(gateway): drain timeout=0 must still wait for in-flight cron/API work #82282 declares Fixes #82161.
• This PR's "not a duplicate" sentence addresses fix(update): avoid gateway-owned restart deadlock #82195 ("complementary to fix(update): avoid gateway-owned restart deadlock #82195, not a duplicate — zero file overlap").

Solution:
Cross-linking the two PRs makes the overlap visible to whoever triages either one.

---

Checked against 82f28b9 — the PR head when this was written — and 2446c8b, main at the same moment.

[teknium1]

Merged via #86684 — both commits were cherry-picked onto current main with your authorship preserved in git log. The #82244 delivery-error recovery was rewoven into main's owner-fenced interrupted short-circuit (the completion path changed with #70638), and a follow-up ensures ownerless legacy fires still make it into the notification list. Thanks @JoaoMarcos44!