gateway: optional durable lifecycle journal for external health monitors

[vxcozy]

Summary

This adds an opt-in journal that records gateway lifecycle events (start, replace) to a bounded JSONL file. It is off by default and activates only when HERMES_GATEWAY_CHURN_PATH names an absolute path. Nothing else changes.

Motivation

We run Hermes unattended on a dedicated machine, with an external watchdog that pages a human when something breaks. The hardest failure to catch from outside is silent gateway churn: the process dies and relaunches (or gets replaced during an update), everything looks healthy afterward, and whatever was in flight during the gap is just gone. Log scraping is fragile across restarts and rotation, and polling the process table misses fast replacements.

A small durable journal solves this: the gateway appends one record per lifecycle event, and any monitor can read the file and apply its own idea of how much churn is too much. We have carried this as a local patch since July and re-applied it across every update. It has caught real incidents for us, including a crash loop after an OS update and a replacement that dropped an in-flight change. Upstreaming it means any install can get the same visibility without carrying a diff.

What's in the PR

• gateway/churn.py (new, ~220 lines): the writer. Bounded at 256 records with atomic rewrite at the cap, advisory file locking (fcntl on POSIX, msvcrt on Windows), and records limited to process identity: event type, timestamp, old pid, new pid. No thresholds and no policy; deciding what churn is acceptable belongs to whatever reads the file.
• Wiring in gateway/run.py (20 inserted lines, nothing removed): after the gateway owns its PID and finishes adapter startup, it writes a start record, or a replace record when it took over a running instance. Both are no-ops when the env var is unset.
• tests/gateway/test_churn.py: activation gating, the record cap, racing writers across processes, readers never seeing a partial window, and separation from the existing takeover-marker contract.

Design choices

The writer is deliberately dumb. It does not rotate by time, does not interpret events, and refuses relative paths instead of guessing a base directory. A write failure logs a warning and never affects gateway startup: visibility must not become a new way to go down. Records are one JSON object per line, so consumers need nothing beyond the standard library.

Compatibility and testing

Off by default, zero behavior change with the env var unset, no new dependencies. On current main: tests/gateway collects clean with this change (5,637 tests, same as without it) and the new file passes (5 tests). The run.py diff is insertion-only.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a9fcdbb2-161f-4b2b-9fd7-b2a061db8690

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

gateway: optional durable lifecycle journal for external health monitors

1. gateway/churn.py + gateway/run.py (~L29621): activation via a new user-facing HERMES_GATEWAY_CHURN_PATH env var is a non-secret behavioral setting. Per AGENTS.md, new HERMES_* env vars for non-secret config are rejected — behavioral settings belong in config.yaml. Consider a gateway.churn_path config key bridged to an internal env var (the gateway config loader already exists), so deployment docs point at config rather than .env.
2. _atomic_append_capped() rewrites the entire file on every event (read up to 1 MB tail, write all bytes, fsync file + dir). Events are rare lifecycle facts so the cost is acceptable today, but each write is O(file size); a plain bounded append with the cap enforced by an occasional rewrite would keep steady-state writes O(1).
3. _read_tail() drops lines[0] whenever start > 0, even when the first line in the 1 MB window is a complete record (short records, file just over 1 MB). That silently discards a valid record at the window boundary. Only drop the first line when the window actually starts mid-line (compare the window start offset against the file's line boundaries).

[vxcozy]

Thanks for taking a look so quickly @Enough1122! Both correctness points are fixed in 0a31f47.

The path moved into config.yaml as gateway.churn_path, which bridges to the internal env var. That follows the AGENTS.md rule for behavioral settings. Config wins when both are set, and the env var alone still works.

_read_tail now reads the byte just before its window and only drops the first line when the window starts mid-record. There are tests for the exact-boundary and mid-line cases.

I'm leaving the rewrite-per-event point alone. Events are process starts, so the file stays around 30 KB with the 256-record cap. The full rewrite is what makes sure a reader never sees a partial window, and a test pins that. A plain append would save little and lose the guarantee.

[alt-glitch]

This was generated by AI during triage.

Related: merged #64536 provides a separate OTLP monitoring path; this PR's opt-in local lifecycle journal is complementary rather than a duplicate.

[vxcozy]

Thanks, @alt-glitch. That matches how I'd frame it too. The one thing worth adding is where the two actually diverge.

The health export samples state on an interval, 60 seconds for metrics by default, so a gateway that dies and gets replaced between two samples still reads as running at both ends. This journal writes at the moment of the handoff instead, and records which pid replaced which. That's the silent churn case it was built for.

It also runs without a collector, which is what makes it useful on a single unattended box where the monitor is just a script reading a file.