Decouple cron-ticker liveness from tick execution (gateway heartbeat thread)

[tjkang]

Problem

cron/scheduler.py::tick() runs on a single in-process background thread
(gateway/run.py::_start_cron_ticker, 60s interval). It acquires
fcntl.flock(LOCK_EX) on <HERMES_HOME>/cron/.tick.lock and holds it for the
entire tick, and tick() runs workdir/profile jobs sequentially
(intended invariant — those jobs mutate process-global _hermes_home/os.environ).

When a sequential job is long-running (e.g. a multi-minute browser+LLM pipeline),
the ticker thread is monopolised for that whole duration. Consequences:

• The next 60s tick fails LOCK_NB and returns early → .tick.lock mtime freezes.
• Any external liveness monitor keyed on .tick.lock mtime false-alarms
  "cron stopped" even though the gateway is perfectly healthy and the job is
  progressing normally.

The freeze is not a throughput bug (long sequential execution is by design).
The only real defect is that the liveness signal is coupled to the work thread.

Fix

A dedicated daemon thread bumps a separate <HERMES_HOME>/cron/.gateway.heartbeat
file mtime every 30s, independent of tick():

• cron/scheduler.py: add _get_heartbeat_path() and heartbeat_loop(stop_event, alive_check, interval=30).
  The bump is a path-based touch + os.utime — unaffected by the flock held
  on the different .tick.lock inode, so it stays fresh mid-tick.
• gateway/run.py: start the heartbeat thread next to the cron ticker and stop
  it on shutdown. The bump is gated on _gateway_alive(), which actively probes
  the event loop with loop.call_soon_threadsafe(event.set) + event.wait(timeout)
  — if the loop is genuinely hung the probe never returns, the heartbeat stops,
  and the monitor correctly fires.

.tick.lock keeps its original meaning ("a tick actually ran"), so tick deadlocks
remain detectable; .gateway.heartbeat answers the distinct question "is the
gateway event loop alive?".

Why a separate file (not reusing .tick.lock)

Reusing .tick.lock for the heartbeat would conflate two signals — a healthy
event loop with a stuck/deadlocked cron tick would then be indistinguishable.
A distinct file preserves both diagnostics.

Invariants preserved

• tick() lock acquisition/release and the sequential/parallel job partition are
  unchanged (byte-for-byte).
• The heartbeat never bumps unconditionally — the _gateway_alive() gate is a
  hard requirement, otherwise an always-on heartbeat would permanently mask real
  outages from the health monitor.

Tests

Unit test covers: heartbeat stays fresh while a long job holds the tick lock;
alive_check()==False ⇒ no bump (hung-loop gate); prompt exit on stop_event;
path-based auto-create.

Notes for adopters

External monitors should treat .gateway.heartbeat as the gateway-liveness
signal and .tick.lock as the tick-progress signal: a stale .tick.lock with a
fresh .gateway.heartbeat means a long job is in progress (no alarm).

[teknium1]

Thanks for the careful separation of tick progress from gateway liveness. This is now redundant on current main.

• Current gateway ticks call cron_tick(..., sync=False) in cron/scheduler_provider.py:179.
• Workdir/sequential jobs are queued to a persistent single-worker pool without blocking the ticker (cron/scheduler.py:3704-3717); async ticker mode returns without awaiting jobs (cron/scheduler.py:3759-3782) and then releases .tick.lock (cron/scheduler.py:3783-3794).
• Commit 475bb334a30318960f5cfe40e6ad7d9d910e13e8 implemented that non-blocking sequential dispatch after this PR's base, removing the lock-mtime freeze described here.
• The PR's gateway startup target has also been superseded by provider-based startup in gateway/run.py:20831-20846.

Automated hermes-sweeper review.