fix(cron): prevent silent ticker stall (17h no-heartbeat incident)

[dizhaky]

Incident

The cron ticker could silently stall: in one observed incident it ran ~17 hours with no jobs firing while the gateway process stayed up and healthy. No error was surfaced — cron simply stopped ticking.

Root cause

The gateway drives cron from a single background thread (gateway/run.py _start_cron_ticker). Two problems combined:

1. Unbounded await on the tick path. The standalone platform delivery send (cron/scheduler.py _deliver_result) awaited _send_to_platform(...) with no timeout. A platform HTTP send that never returns wedges the entire ticker thread — cron stops firing.
2. No liveness signal and no watchdog. The old loop caught tick exceptions only at logger.debug (gateway/run.py old _start_cron_ticker: except Exception as e: logger.debug("Cron tick error: %s", e)), and there was nothing watching whether the loop was still alive. A wedged or dead ticker was never detected or restarted.

Fix

• Heartbeat before job work. New cron/scheduler.py::run_cron_tick_once records a liveness heartbeat via record_ticker_heartbeat() at the top of every tick, before any job work, then calls tick() inside a try/except that logs escaped exceptions at ERROR (with traceback) and continues — a single bad tick can no longer kill the loop or make it look dead. The gateway loop now calls run_cron_tick_once instead of cron_tick directly.
• Bounded delivery. cron/scheduler.py::_deliver_result wraps the standalone send in asyncio.wait_for(..., timeout=_DELIVERY_TIMEOUT) (_DELIVERY_TIMEOUT = 60s), and the thread-pool fallback uses future.result(timeout=_DELIVERY_TIMEOUT + 5). A delivery that never returns now fails as an error instead of hanging forever.
• Self-healing supervisor. New gateway/run.py::_cron_ticker_supervisor watchdog runs in its own daemon thread, checks ticker_heartbeat_is_stale(interval, stale_multiplier=5.0), and restarts the ticker via a mutable holder + _spawn_cron_ticker() when the heartbeat is older than 5x the tick interval. It does not restart while the heartbeat is healthy, and does not thrash before the first tick (stale check returns False when no heartbeat has been recorded yet). The tick() file lock prevents old/new ticker threads from overlapping.

The heartbeat uses a monotonic clock for age math (immune to wall-clock jumps) and mirrors a human-readable timestamp to a .ticker_heartbeat file for operator diagnostics (best-effort; never raises into the loop).

Test plan

New TestTickerHeartbeatAndSupervisor in tests/cron/test_scheduler.py (5 tests):

1. test_tick_exception_does_not_kill_loop — an exception in tick() is caught/logged and run_cron_tick_once returns without propagating.
2. test_heartbeat_recorded_before_tick_even_when_job_hangs — heartbeat is recorded before/independent of job work (loop liveness, not job success).
3. test_heartbeat_stale_detection — ticker_heartbeat_is_stale returns False when healthy / never recorded, True once past the threshold.
4. test_supervisor_restarts_stale_ticker — supervisor calls restart_ticker when the heartbeat is stale.
5. test_supervisor_does_not_restart_when_healthy — supervisor does not restart a healthy ticker.

Full tests/cron/test_scheduler.py suite: 132 passed.

🤖 Generated with Claude Code

---

Generated by Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

🔎 Lint report: fix/cron-ticker-silent-stall vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 8615 on HEAD, 8613 on base (🆕 +2)

🆕 New issues (4):

Rule	Count
unsupported-operator	2
invalid-assignment	2

First entries

cron/scheduler.py:79: [unsupported-operator] unsupported-operator: Operator `+=` is not supported between objects of type `None` and `Literal[1]`
cron/scheduler.py:77: [invalid-assignment] invalid-assignment: Invalid subscript assignment with key of type `Literal["monotonic"]` and value of type `int | float` on object of type `dict[str, None | int]`
tests/cron/test_scheduler.py:2611: [unsupported-operator] unsupported-operator: Operator `<` is not supported between objects of type `int | float | None` and `float`
cron/scheduler.py:78: [invalid-assignment] invalid-assignment: Invalid subscript assignment with key of type `Literal["wall"]` and value of type `str` on object of type `dict[str, None | int]`

✅ Fixed issues (2):

Rule	Count
unresolved-attribute	1
not-subscriptable	1

First entries

cron/scheduler.py:753: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `_T@run & ~AlwaysFalsy` in union `(Unknown & ~AlwaysFalsy) | (_T@run & ~AlwaysFalsy)`
cron/scheduler.py:754: [not-subscriptable] not-subscriptable: Cannot subscript object of type `_T@run` with no `__getitem__` method

Unchanged: 4578 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.