fix(gateway): recover watchdog after transient loop stalls

[StellarisW]

Summary

This fixes a residual failure mode in the opt-in systemd event-loop watchdog introduced by #66946:

• a transient late asyncio tick reports degraded and keeps renewing the watchdog lease;
• two consecutive timely ticks are required before the service reports healthy again;
• a fully stalled loop still emits no heartbeat, so WatchdogSec remains the hard-stop restart path.

The PR has been rebased as one signed commit onto current main (c896c09c42910c584c4c7d2325b58c14713ea42c). The rebase conflict came from main's later test-pruning pass; the resolution preserves that pruning and retains only the two behavior tests specific to this fix.

Reproduction and root cause

With gateway.systemd_watchdog_seconds > 0:

1. let the watchdog task schedule its next sample;
2. block the asyncio loop longer than the internal lag tolerance but shorter than WatchdogSec;
3. allow the loop to run again.

Before this PR, the late callback permanently set _unhealthy, returned without WATCHDOG=1, and caused _run() to exit. The event loop could recover in the same process, but the notifier never restarted, so systemd later restarted an otherwise progressing gateway.

The classification was too coarse: a callback that runs late proves the loop resumed; it is different from a callback that never runs.

Behavioral contract

Condition	Notification	State	Supervisor outcome
Timely tick while healthy	WATCHDOG=1	healthy	lease renewed
Late callback while alive	WATCHDOG=1 + STATUS=degraded	degraded; timely streak reset	lease renewed
First timely recovery tick	WATCHDOG=1	degraded; streak = 1	lease renewed
Second timely recovery tick	WATCHDOG=1 + STATUS=healthy	healthy	normal operation restored
Callback cannot run before WatchdogSec	nothing	no in-process transition	systemd restarts the gateway

Implementation

• gateway/systemd_notify.py: adds the two-tick recovery state and keeps the notifier alive after a late callback.
• tests/gateway/test_systemd_notify.py: verifies the state transitions and injects a real transient loop stall.
• website/docs/user-guide/messaging/index.md: distinguishes late-but-resumed progress from a full no-callback stall.

No new task, cadence, configuration key, dependency, public API, or persistence surface is added.

Validation

• Focused recovery, real transient-stall, existing heartbeat/lifecycle, and generated-unit tests via scripts/run_tests.sh: passed, 0 failed.
• ruff check on both changed Python files: passed.
• ruff format --check on both changed Python files: passed.
• Python compile smoke test: passed.
• git diff --check origin/main...HEAD: passed.
• One signed commit; no merge commits in origin/main..HEAD.
• New GitHub CI for rebased head 365134f66990e2ac5f70bbb826247bbc271036f1: running after force-with-lease push.

Local macOS note: the full tests/gateway/test_systemd_notify.py file still contains main's pre-existing Linux abstract-socket test, whose skip condition accepts macOS AF_UNIX even though macOS does not support Linux abstract socket addresses. Validation therefore selected the watchdog behavior tests directly; Linux CI runs the full file.

Compatibility and risk

• Default-disabled watchdog installations are unchanged.
• Non-systemd platforms remain no-ops through the existing NOTIFY_SOCKET / AF_UNIX gates.
• Recurring but bounded lateness keeps the process alive in degraded; restart remains reserved for loss of progress through the full WatchdogSec window.
• Notifications remain local Unix datagrams and contain no user or session data.

Scope and non-goals

• does not change Discord WebSocket liveness or reconnect policy;
• does not recover interrupted agent sessions or retry external side effects;
• does not add a configurable recovery threshold;
• does not claim exactly-once delivery.

Crash-safe in-flight session continuation remains separate in #67078.

Rollout and rollback

No migration is needed. Rollback is a single revert of signed commit 365134f66990e2ac5f70bbb826247bbc271036f1; there is no persisted state to clean up.

Head integrity

• PR head: StellarisW:codex/gateway-watchdog-recovery
• Signed commit: 365134f66990e2ac5f70bbb826247bbc271036f1
• Base used for the rebase: c896c09c42910c584c4c7d2325b58c14713ea42c
• History: one feature commit, no merge commits

Infographic

A callback that runs renews the lease; only callback absence leaves systemd to expire it. Two consecutive timely callbacks are required before degraded state returns to healthy.

[alt-glitch]

This was generated by AI during triage.

Related to merged #66946: that work added systemd watchdog wiring, while this PR fixes the distinct residual case where one transient late tick permanently stops watchdog feeds.

[teknium1]

Thanks for the focused recovery fix. The premise is verified on current origin/main: gateway/systemd_notify.py:125-152 latches _unhealthy after a late callback, returns false, and exits _run(), while tests/gateway/test_systemd_notify.py:95-112 codifies that permanent latch. The proposed state transition addresses that exact path, and no later commit changed either target file after f57157a12.

Problems

• website/docs/user-guide/messaging/index.md:174-179 says heartbeats are sent only during timely loop progress and systemd restarts when they stop. The PR now renews WATCHDOG=1 for a late callback in degraded state, so that operator contract becomes inaccurate.

Suggested changes

• Update that watchdog paragraph to distinguish a late-but-resumed callback (degraded heartbeat, then two timely samples to recover) from a full stall with no callback, which remains systemd's restart boundary.

Automated hermes-sweeper review.

[StellarisW]

Addressed in c4babe09f:

• updated the watchdog operator docs to distinguish a late-but-resumed callback (WATCHDOG=1 + degraded status, then two timely samples to recover) from a full no-callback stall (WatchdogSec restart boundary)
• rebased the single signed commit onto the validated current main

Final local validation: 23 focused watchdog tests, 113 gateway config tests, 9,646 gateway tests, docs diagram lint (364 files), Docusaurus English/zh-Hans build, Ruff/format/compile, and git diff --check all passed.