feat(gateway): built-in health heartbeat (adapters, dashboard, nightly sync) - #20
Merged
Merged
Conversation
…tion hotfix Production evidence (gui.log, desktop.log, live dirty tree) showed why the merged durable-session fix still wasn't enough for "it only starts working when I open the session": 1. Accepted-but-unstarted prompts (`session["queued_prompt"]`) lived only in memory and only drained from client-driven hooks. The 20s WS-orphan reaper and the TTL/LRU evictors happily tore down a detached session holding one, silently dropping a message the server had said yes to. All three reapers now treat queued work as pending work and spare the session. 2. When the end-of-turn drain lost its update-guard race (or the turn thread died between clearing `running` and draining), nothing ever retried without a client attached. A background sweeper now starts accepted prompts on idle sessions every 15s, kicking the agent build first when the prompt was queued during the build window. Work the server accepted runs whether or not anyone is watching. 3. The live install carried two hand-applied hotfixes that only existed as uncommitted edits (lost on every nightly sync): the compacted-transcript fix (include_compacted in SessionDB.get_messages + REST/api_server/compute host readers — history vanishing after in-place compression) and a client_request_id dedup bridge. The transcript fix is ported verbatim; the bridge is superseded by the request ledger, with wire compat kept: `client_request_id` is accepted as an alias for `request_id`, duplicate responses carry `client_request_id`/`deduplicated`, and `prompt.status` answers alongside `request.status`, so clients built against the hotfix keep working unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two TestFetchOpenRouterModels tests mocked the live /v1/models HTTP call but not get_curated_openrouter_models(), so preferred_ids silently tracked the DEPLOYED catalog manifest — when it dropped qwen/qwen3.7-max the tests broke on every branch (pre-existing on ace/patches, verified). The sibling fallback test already pins the manifest out with exactly this patch and a comment saying why; apply the same pin to both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y sync)
The nightly upstream sync had been failing for weeks (SSH fetch errors, then
31-file merge conflicts) with status only in last-run.json — nobody told. The
dashboard backend dies and respawns on a moving port with nothing watching
it. Hermes now monitors itself from the gateway — the always-on process —
and reports to the chat surface the user actually reads:
- gateway/heartbeat.py: config-gated HeartbeatService (off by default).
Probes: live platform adapters, the dashboard backend (via a startup
portfile — ports change every respawn, so discovery beats config), and the
nightly-sync last-run.json (failed OR silent-past-deadline both alert).
Cadence: alert immediately on unhealthy transition, re-alert every
realert_hours while broken, recovery message once, periodic all-healthy
beat every heartbeat_hours so silence is distinguishable from a dead
monitor. A failed send leaves cadence state untouched so the next tick
retries. Delivery reuses the standalone platform sender cron uses, so a
beat can still get out when a live adapter object is wedged.
- gateway/run.py: start the service after cron/housekeeping in
start_gateway, wired to runner.adapters.
- hermes_cli/web_server.py: write <hermes_home>/dashboard.portfile.json
(port + PID) once the HTTP server binds, enabling out-of-process probes.
Config:
heartbeat:
enabled: true
channel: "telegram:<chat_id>"
check_interval_minutes: 5
heartbeat_hours: 24
realert_hours: 6
nightly_status_file: /path/to/last-run.json # cross-HERMES_HOME setups
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The nightly upstream sync has been failing for weeks — Aug 2 on SSH fetch, Aug 3 on a 31-file conflict merge — and the only record was
last-run.jsonon disk. The dashboard backend dies/respawns on a moving port with nothing watching. Nobody gets told. Owner directive: health checks + heartbeats for Hermes (Ace Coder + Telegram), built into Hermes.What
gateway/heartbeat.py— config-gatedHeartbeatService(off by default) started fromstart_gateway:gatewaydashboard<hermes_home>/dashboard.portfile.json, written byweb_serverat bind — ports change every respawn, so discovery beats config): PID alive + HTTP root answersnightly_synclast-run.json:failedalerts, and silence pastnightly_max_age_hoursalso alerts (a daily job that stops running is a failure too)Cadence: immediate alert on unhealthy transition → re-alert every
realert_hourswhile broken → one recovery message → periodic all-healthy beat everyheartbeat_hours(so silence ≠ dead monitor). Failed sends leave cadence state untouched and retry next tick. Delivery uses the standalone platform sender (same as cron), so beats get out even when a live adapter object is wedged.Checks
tests/gateway/test_heartbeat.py: 22 tests (config parsing, all three probes incl. a real HTTP server, alert/realert/recovery/beat cadence, failed-send retry, render). All green; ruff clean.test_complete_path_at_filter.py/test_systemd_notify.pyfail identically on the clean base locally (env-dependent: AF_UNIX tmp-path length) — pre-existing, green in CI.Also (outside this diff): live repo
upstreamremote switched SSH→HTTPS so the 4 AM launchd fetch stops flaking without an ssh-agent.🤖 Generated with Claude Code