feat(cron): alert when ticker supervisor restarts a stalled ticker

[dizhaky]

Motivation

A cron ticker stall once went unnoticed for 17 hours — the gateway stayed up while scheduled jobs silently stopped firing. PR #71 fixed the recovery half of that failure mode by adding a watchdog (_cron_ticker_supervisor) that detects a stale heartbeat and restarts the ticker thread. But the restart itself is silent: nothing announces that a stall happened and self-healed. The original incident was invisible precisely because nothing surfaced it — so a self-heal we never hear about leaves the same observability gap.

This PR closes that gap: when the supervisor restarts a stalled ticker, it now sends a home-channel alert.

What changed

• gateway/run.py
  • _cron_ticker_supervisor(...) (~L17925) now accepts adapters and loop (mirroring what _spawn_cron_ticker already receives). On a stale heartbeat it restarts the ticker first (recovery is priority), then sends the alert.
  • New _send_cron_supervisor_alert(adapters, loop, stall_age_s, threshold_s) (~L17869) resolves home-channel targets via cron.scheduler._iter_home_target_platforms() / _get_home_target_chat_id() / _get_home_target_thread_id() and delivers via the live adapter — reusing the existing cron home-channel delivery path.
  • New _cron_supervisor_alerts_enabled() (~L17869) reads the config knob.
  • Supervisor spawn site (~L18244) now passes runner.adapters and _cron_loop.

Config knob + cooldown

• cron.supervisor_alerts (bool, default True) in config.yaml, read via hermes_cli.config.load_config() — the same pattern as cron.script_timeout_seconds / cron.wrap_response. When False, the ticker still restarts and logs, but no alert is sent. Config-getter chosen over an env var because the cron.* config pattern is already established and trivial to follow; it fails open (a config read error still alerts).
• 15-minute cooldown (alert_cooldown=900s, tracked with time.monotonic()): a flapping ticker still restarts + logs every time, but only alerts once per cooldown window to prevent alert storms.

Thread/loop safety

Both the ticker and supervisor run in worker threads, so the alert send is scheduled onto the gateway event loop with safe_schedule_threadsafe(adapter.send(...), loop) then bounded by future.result(timeout=30) (guards future is None, future.cancel() on TimeoutError) — the exact pattern _deliver_result uses in cron/scheduler.py. The entire helper is wrapped in try/except: a missing loop, unresolved targets, or a hung send is logged and swallowed, never propagated, so the alert can never delay or block ticker recovery.

Test plan

New tests/gateway/test_cron_supervisor_alert.py (9 tests; supervisor-loop cases mock the alert helper, send-helper cases mock adapters/loop/scheduling — no real event loop or network):

1. Stale-heartbeat restart calls the alert exactly once (with correct age/threshold args).
2. Healthy ticker sends no alert and does not restart.
3. Alert raising still restarts the ticker and does not crash the supervisor (exception isolation).
4. Cooldown: two stalls within the window restart twice but alert once.
5. cron.supervisor_alerts disabled skips the alert but still restarts.
   6-9. Send-helper direct tests: quiet no-op without loop / without adapters; resolves + sends to home channel; timeout cancels the future without raising.

Results: tests/gateway/test_cron_supervisor_alert.py 9 passed; tests/cron/test_scheduler.py 132 passed (existing supervisor tests still green under the new signature).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PQKCc5mDedYAiCNyXnTezh

---

Generated by Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

🔎 Lint report: fix/cron-supervisor-stall-alert vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 8616 on HEAD, 8615 on base (🆕 +1)

🆕 New issues (1):

Rule	Count
unresolved-import	1

First entries

tests/gateway/test_cron_supervisor_alert.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`

✅ Fixed issues: none

Unchanged: 4582 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.