fix(dashboard): coalesce repeat gateway restarts for a short window

[jackulau]

What does this PR do?

Stops the dashboard from starting a fresh gateway restart every time a stale
frontend asks for one, which is the flood at the top of #89034's causal chain.

_spawn_gateway_restart already refuses to start a second restart while the
first hermes gateway restart child is alive — that is what makes a
double-clicked button safe. The guard evaporates exactly when it matters most.
The child exits as soon as it has handed the restart off (to the supervisor, or
to the running gateway), which is long before the gateway is back, so:

existing = _ACTION_PROCS.get("gateway-restart")
if existing is not None and existing.poll() is None:   # False the moment the child exits
    ...
return _spawn_hermes_action(subcommand, "gateway-restart"), False

a cached frontend re-firing every few seconds clears poll() is None on every
attempt and gets a brand new restart each time. The function's own docstring
already names that frontend as a known caller ("a stale cached frontend firing
its own restart after the server already auto-restarted post-onboarding") — it
just guards the wrong half of the problem.

The reporter measured the consequence on an s6-supervised container: 77
gateway-restart started entries, 17 of them inside one minute. Each one
SIGHUPs a gateway that is still coming up, and enough of those landed
mid-FTS5-write to corrupt state.db — database disk image is malformed 203x
in agent.log, auto-repair failed, operator recreated the file by hand.

This PR coalesces same-profile requests that arrive within
GATEWAY_RESTART_COOLDOWN_SECONDS of the last spawn onto that spawn, and logs
each one. A storm becomes one restart plus a run of coalesce lines that say so.

Two design decisions worth naming, both maintainer calls if you disagree:

1. A fixed window, not "until the gateway reports healthy." Gateway restart-loop (stale dashboard frontend) + s6 finish with no backoff → state.db FTS5 corruption ("database disk image is malformed") #89034's
   Expected section asks for the health-gated version, and it is the more
   precise rule. I did not implement it because it cannot be made to fail safe
   here: a gateway that never comes back would leave the restart action
   permanently inert, and "the restart button no longer works" is a worse
   failure than the flood it prevents. A fixed window always releases. 10s is
   above the ~3.5s spacing of the reported storm and below what an operator
   waits before deliberately retrying.
2. The cooldown state lives outside _ACTION_PROCS. Completed action
   children are reaped out of that table, and a guard that disappears when the
   child exits is precisely the bug being fixed. There is a regression test for
   this, and it matters for fix(dashboard): reap action subprocesses without status polling #89060 specifically (see Overlap below).

Related Issue

Fixes #89034

Scope: this fixes Defect 1 only. #89034 reports two compounding defects and
I am deliberately not touching the second one. Defect 2 is the s6 finish
script having no death-cap, generated by
hermes_cli.service_manager.S6ServiceManager._render_finish_script. I looked at
it and did not write it, for a reason I would rather state than bury: the
suggested cap makes s6 stop supervising after N unclean exits in a window, so a
container that legitimately crash-loops for a transient reason (bad token,
unreachable provider) would stay permanently down instead of recovering when
the cause clears. That is a real product tradeoff on a shipped image and it
belongs to a maintainer, not to me. It is also worth noting that s6-supervise
already floors respawns at roughly one per second, so Defect 2 alone does not
produce the observed 3.5s-spaced storm — Defect 1 does. Fixing this half
removes the driver; the death-cap would be a second layer of protection under
it.

Overlap with open PRs

Three open PRs touch this area. I read all three; none of them rate-limits
restart requests, and I have tried to leave room for each.

PR	what it changes	overlaps?
#66595	_spawn_gateway_restart delegates to launchd/systemd/s6 when supervised, instead of spawning a child	same function, textual conflict likely. Orthogonal in behaviour: it changes how a restart is performed, not how often. In #89034's s6 container it would delegate to s6 — still 77 times. If it lands first I will rebase this on top; the guard is a few lines at the top of the function and does not care which path performs the restart.
#75845	POST /api/system/restart-hermes sends SIGUSR1 instead of spawning	different endpoint; /api/gateway/restart (the one the reported storm used) is unchanged by it
#89060	background reaper so exited action children leave _ACTION_PROCS without a status poll	composes deliberately. Their reaper removes exited entries from that table. If this cooldown had been implemented by reading _ACTION_PROCS — the obvious way — their PR would silently re-open this exact hole. test_cooldown_survives_the_action_table_being_cleared pins that. We do both edit tests/hermes_cli/test_spawn_gateway_restart_reap.py, so expect a small conflict there.

Changes Made

• hermes_cli/web_server.py
  • new module constants GATEWAY_RESTART_COOLDOWN_SECONDS (10.0) and
    _LAST_GATEWAY_RESTART, with the reasoning for both, next to
    _ACTION_PROCS.
  • _spawn_gateway_restart: after the existing in-flight reuse, coalesce a
    same-profile request made within the window onto the last spawn, log it at
    INFO, and record (monotonic, proc, command) on every real spawn. A
    different profile is never coalesced — two profiles are two services.
  • docstring extended to say why in-flight reuse alone was not enough.
• tests/hermes_cli/test_spawn_gateway_restart_cooldown.py (new, 7 tests).
• tests/hermes_cli/test_spawn_gateway_restart_reap.py: added an autouse
  fixture clearing the new module state. Required — without it the first case's
  spawn suppresses the second case's, and that file fails. It is a fixture
  rather than a per-test decorator so it collides as little as possible with
  fix(dashboard): reap action subprocesses without status polling #89060.

How to Test

1. The new tests pass:

   pytest tests/hermes_cli/test_spawn_gateway_restart_cooldown.py -q
   

   7 passed

2. Sabotage proof. Delete the recent = _LAST_GATEWAY_RESTART block from
   _spawn_gateway_restart and re-run — the three regression tests fail with
   the storm itself, while the four guardrail tests keep passing (so they are
   not just asserting the fix back to itself):

   E  AssertionError: a repeat request 3.5s later must not start a second restart
   E  assert 2 == 1
   E  assert 3 == 1     <- test_a_storm_of_requests_produces_exactly_one_restart
   E  assert 2 == 1     <- test_cooldown_survives_the_action_table_being_cleared
   3 failed, 4 passed
   

   Restore the block and all 7 pass. The 3 == 1 is the reported behaviour in
   miniature: three requests, three restarts.

3. Neighbouring suites, unchanged:

   pytest tests/hermes_cli/test_web_server.py \
          tests/hermes_cli/test_spawn_gateway_restart_reap.py \
          tests/hermes_cli/test_spawn_gateway_restart_cooldown.py -q
   

   170 passed, 4 skipped

4. Baseline comparison over every test file in the repo that mentions
   web_server (85 files, ~1060 tests), run serially with and without this
   change: 31 failures both times, identical sets (pre-existing
   Windows/environment failures — os.geteuid, resource limits, a
   non-Windows-only platform test). Zero new failures, zero fixed by accident.

5. ruff check clean on all three files.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate. Nothing open rate-limits gateway restart requests; the three PRs that touch this seam are compared in the Overlap table above, with the conflict risk for each stated
• My PR contains only changes related to this fix/feature (no unrelated commits): one commit, one production file, two test files
• I've run pytest tests/ -q and all tests pass. Not the full suite — tests/hermes_cli/ cannot be collected on Windows (test_doctor_journal_modes.py calls os.geteuid). Ran the 85-file web_server slice instead, with a with/without baseline: identical failure sets, zero new failures. CI covers the rest
• I've added tests for my changes (required for bug fixes, strongly encouraged for features): 7 tests, 3 of which fail without the fix — step 2 is the mutation proof
• I've tested on my platform: Windows 11 Pro 26200, Python 3.13

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings). _spawn_gateway_restart's docstring now explains why in-flight reuse alone was insufficient; no user-facing docs affected
• N/A, no config keys added or changed. The window is a module constant rather than a config key on purpose — see the note below
• N/A, no architecture or workflow change
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide. The change is a monotonic-clock comparison with no platform-specific behaviour; it sits above the sys.platform branching in _spawn_hermes_action and applies equally to launchd, systemd, s6 and unsupervised hosts
• N/A, no tool descriptions or schemas changed

On the constant vs. a config key: I kept it a module constant to stay minimal.
If you would rather it be tunable, agent.gateway_restart_cooldown alongside
restart_drain_timeout is the obvious home and I am happy to add it — say the
word. I specifically did not reuse restart_drain_timeout even though it
looked like the principled choice, because it defaults to 0, which would make
the guard a no-op for everyone who has not configured it.

Screenshots / Logs

What the storm looks like after the change — one restart, and the suppressed
repeats say so instead of vanishing:

=== gateway-restart started 2026-08-18 09:14:02 ===
INFO hermes_cli.web_server: Coalescing gateway restart: one was started 3.5s ago (pid 41822) and the gateway may still be coming back; not spawning another (#89034).
INFO hermes_cli.web_server: Coalescing gateway restart: one was started 7.0s ago (pid 41822) and the gateway may still be coming back; not spawning another (#89034).

From #89034, what it looked like before (excerpt):

gateway-restart started            77
database disk image is malformed   203x (agent.log), 442x (errors.log)
gateway.previous_unclean_exit      14x

[kshitijk4poor]

Merged via #90275 using rebase-merge — your commit cherry-picked with authorship preserved.

Your fix was excellent: correct root cause diagnosis (child exits before gateway is back, so poll() is None guard disappears), well-designed fail-safe cooldown (fixed window always releases, unlike health-gating which could wedge permanently), mutation-proof tests (3 fail without the fix, 4 guardrail tests still pass), and thorough audit of all 3 competing PRs for composition. No changes needed — salvaged as-is onto current main.

Closes #89034