fix(cron): preserve active turns across gateway restarts (salvage #101877)

[kshitijk4poor]

Summary

Cron jobs fired by a systemd-managed Linux gateway now survive systemctl restart / hermes gateway restart: the fire is handed to a detached systemd-run --scope worker that owns the execution durably, and its final delivery is queued for whichever gateway instance is live.

Salvage of #101877 by @jayleaton — the three commits are authored by Brooklyn Nicholson (@OutThisLife) and cherry-picked with authorship preserved. Two follow-up commits fold in the review findings.

Root cause: #60711 made the shutdown drain see in-flight cron work, but a forced restart still kills the gateway cgroup, and the cron agent turn with it. start_new_session alone does not escape a systemd service cgroup.

Changes

• cron/scheduler.py: run_one_job hands a managed-gateway fire to python -m cron.scheduler --external-worker-file … via restart_safe_gateway_child_argv; parent waits on the worker process (1s wait(timeout)), never re-runs the job in-process on an uncertain handoff; worker adopts the execution row (CAS claimed+handoff_pending → running) before any side effect; restart-safe waiters are excluded from forced-shutdown interruption; stranded payload/ack files reaped once terminal; worker entrypoint sets up hermes logging.
• cron/executions.py: handoff_pending/handoff_started_at columns (via sqlite_util.add_column_if_missing), mark_execution_handoff_pending, adopt_claimed_execution, owner-fenced running/finish/recovery with a 30s adoption grace.
• cron/delivery_queue.py (new): profile-local sqlite queue; gateway claims a row before transport I/O, dead claimant → unknown never replayed; unclaimed rows stay queued across a long gateway outage (they are certainly unsent, not uncertain) and are reported as deferred, not delivery_failed; WAL via apply_wal_with_fallback; prune only where terminal rows are created.
• gateway/run.py: housekeeping thread drains the queue per profile scope (gated on the queue file existing, so non-systemd gateways never open it).
• tools/process_registry.py: restart_safe_gateway_child_argv — fail closed in managed topology when a user scope can't be created; passthrough everywhere else.
• hermes_cli/kanban_db.py: Kanban worker spawn uses the same scope wrapper and build_subprocess_env.
• tools/cronjob_tools.py: cronjob(run) reports the exact execution outcome instead of stale last_status.
• _deliver_result queues only when the worker marker matches the delivering job's own execution_id (a nested in-process dispatch inside a worker must not be keyed under the outer attempt).

Validation

	Before	After
systemd gateway restarted mid cron run	agent turn killed, false interrupted	worker survives, side effect once, delivered once (E2E: parent SIGKILLed while worker running)
gateway down > worker wait budget (300s)	delivery marked failed, never sent	row stays pending, next gateway drains it
parent ledger polling during a run	20 sqlite opens/s for the whole job	~1 read/s, wakeup on process exit
macOS / Windows / launchd / Docker gateways	—	unchanged: wrapper returns before any I/O; cron run CLI untouched
tests/cron + housekeeping + kanban handoff + cronjob_tools		1228 passed; 5 test_monitor_kind failures are pre-existing on main (env)
ruff / ty (diff vs main)		clean / 0 new diagnostics

Regression guards added: nested-dispatch queue keying (mutation-checked), pending-stays-queued, worker exit-then-recheck; test_lost_fire_claim_stops_stale_delivery had become vacuous under the new ownership CAS and is fixed (mutation-checked).

Related

• Predecessor: /update bypasses drain and interrupts in-flight cron tool work #60432 → fix(gateway,cron): make shutdown drain see in-flight cron work (#60432) #60711 (@HexLab98, @JoaoMarcos44). This changes the restart-safe subset of that contract: a parent waiting on a scoped worker is not force-interrupted once ownership has moved.
• Overlaps feat(gateway): isolate workers and queue under resource pressure #97739 (system-scope backend selection) on process_registry/scheduler/run.py. This PR is user-scope-only and fails closed when a user scope is unavailable; if feat(gateway): isolate workers and queue under resource pressure #97739 lands first, restart_safe_gateway_child_argv should consume its backend-neutral seam.
• Kanban worker scope change is intentional (same restart hazard).

Closes #101877

[jayleaton]

Live managed-restart follow-up: the original exact head's process isolation worked, but the end-to-end acceptance did not.

At 16:35 local time I restarted the sole default-profile managed gateway while cron execution eb4ef99dccd440deb5e2e88140cea024 (job c8e80ee31ad4) was active:

• gateway PID changed 908682 → 1059241;
• cron worker PID 1021707 stayed alive and running in hermes-worker-cron-c8e80ee31ad4-exec-eb4ef99dccd440deb5e2e88140cea024.scope;
• exactly one default gateway remained active;
• the replacement gateway delivered the successful result once (for_failure=0, one delivery row, status=delivered, owner PID 1059241);
• there was one execution row and one same-job execution — no duplicate fire or delivery.

However, immediately after delivery the worker terminalized that same execution as:

failed: Interrupted by shutdown before terminal completion.

The job likewise recorded last_status=error. So the scoped worker and successful delivery survived, but durable terminal ownership still produced a false shutdown failure.

The existing live test passes but misses this topology: test_managed_gateway_restart_preserves_active_worker_and_single_side_effect terminates a harness parent and manually drains through an in-process replacement loop; it does not run the real gateway's 30-second graceful drain and replacement ticker. On current origin/main, the focused file remains green (18 passed, 1 skipped), confirming a coverage gap rather than an existing red test.

This live run was on the original exact head 74d90c3198; afterward I deployed current origin/main (80fae22bf5, containing merge c3e9b28a42) through the same worktree and verified one active default gateway at that SHA. I have not repeated the production restart. QA follow-up is tracked as Kanban t_8d12dfaf to isolate whether the external worker's fire-claim heartbeat sets lost_ownership during the graceful restart/replacement window or another path rewrites the claim. No second PR was opened.