fix(gateway): per-session quiescence + task-liveness reaper (busy-gateway deferred restart never fires)

[Kyzcreig]

What & why

A deferred safe-restart on a busy multi-session gateway never fires. The watcher's wait_for_quiescence waits for the gateway to go globally idle (active_agents == 0 across every session). A gateway serving many channels + sub-agent fan-outs + cron turns rarely/never hits global-zero, so the watcher waits the full QUIESCE_TIMEOUT, logs QUIESCE_TIMEOUT, posts a ⚠️, and gives up without restarting — the gateway keeps running stale code/config.

Proven live 2026-06-30: a config change deferred-restarted the gateway; it waited >15 min and never fired while a second conversation + a burst of sub-agent sessions held active_agents at 2–3. (The first theory "the user kept re-activating it" was falsified — the gate is global; silence wouldn't have helped.)

Secondary defect: leaked _running_agents entries have no background reaper — a turn coroutine killed without unwinding (hard SIGKILL mid-turn, handler crash) leaves a phantom entry making active_agents == 0 permanently unreachable.

Fix

• Phase 1 — publish per-session active keys. gateway/status.py write_runtime_status gains active_agent_keys (+schema_version, 0600), so a watcher can ask "is my initiating session idle?" instead of "is the whole gateway idle?".
• Phase 2 — task-liveness reaper. A periodic sweep evicts a _running_agents slot iff its owning turn task is done()/cancelled — a live long turn (a 30-min fan-out, a long fetch) is never reaped, any age. Amputation-safe by construction (captures current_task() synchronously at the slot-set, no registration window; crash-safe per-iteration; idempotent release). This is the delicate part and is heavily unit-tested.
• Phase 3pt1 — per-session quiescence gate (in the watcher scripts, shipped separately in the skills repo): proceed when the initiating session is idle even if siblings are busy.

Scope decision (Option A)

The spec originally proposed a gateway-side recheck at drain ("never interrupt the initiating turn"), but that rested on a false premise — the F2 restart-initiator breadcrumb is consumed at turn-end, before the post-quiescence kickstart ever drains, so a gateway-side recheck has nothing to read (logically vacuous). Per owner decision, Option A: accept that the shipped watcher per-session gate + the existing 180s graceful-drain + auto-resume is the complete fix. The bounded residual — a fresh >180s initiating turn landing in the poll→kickstart window — gets interrupt+auto-resume, identical to every turn on every restart today. No new code in the gateway shutdown/drain path (the highest-blast-radius zone).

Tests (22 focused, all green post-rebase)

• tests/gateway/test_quiescence_status.py — active_agent_keys published, schema_version, 0600, back-compat (missing field).
• tests/gateway/test_quiescence_task_tracking.py — the turn task is captured as the exact current_task() with no await window.
• tests/gateway/test_quiescence_reaper.py — dead-task reaped; live-long-task survives; pending-sentinel survives; registration-window survives; concurrent register/release no-KeyError; crash-safe.

Watcher-side Phase 3pt1 (both safe-gateway-restart and safe-reboot watchers) ships via the skills/home repo (already committed there) — noted so the reviewer knows this PR is the hermes-agent half.

[github-actions]

🔎 Lint report: feat/gateway-quiescence-reaper vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 12271 on HEAD, 12265 on base (🆕 +6)

🆕 New issues (4):

Rule	Count
unsupported-operator	1
unresolved-attribute	1
not-subscriptable	1
invalid-assignment	1

First entries

tests/gateway/test_quiescence_status.py:49: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["active_agent_keys"]` and `dict[str, Any] | None`
tests/gateway/test_quiescence_status.py:39: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `None` in union `dict[str, Any] | None`
tests/gateway/test_quiescence_status.py:81: [not-subscriptable] not-subscriptable: Cannot subscript object of type `None` with no `__getitem__` method
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`

✅ Fixed issues (2):

Rule	Count
unresolved-attribute	2

First entries

run_agent.py:3129: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`

Unchanged: 6426 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

[greptile-apps]

Greptile Summary

This PR fixes a long-standing deferred-restart failure on busy gateways: wait_for_quiescence waited for global active_agents == 0 (never reached under fan-out load) and gave up. It also adds a background reaper to reclaim _running_agents slots leaked by turns killed without unwinding (hard SIGKILL, handler crash), which made active_agents == 0 permanently unreachable.

• Phase 1 (status): write_runtime_status gains active_agent_keys (live session keys, not just a count) and schema_version=2; gateway_state.json is now written at 0o600 (owner-only) because the keys carry chat-id + user-id metadata.
• Phase 2 (reaper): _reap_dead_running_agents_loop runs every ~60 s and evicts _running_agents slots whose recorded asyncio Task is done()/cancelled. A 30 s grace period covers the narrow registration window between the slot-set and the synchronous current_task() capture. Live tasks (even multi-hour fan-outs) are never reaped regardless of age.
• Test coverage: 22 new focused tests cover the key invariants (dead reaped, live survives, grace window, crash isolation, idempotent double-release, heartbeat, file mode).

Confidence Score: 5/5

Safe to merge — the reaper is strictly additive, crash-isolated, and touches only genuinely dead slots; the status change is a backwards-compatible field addition with the file-mode tightened to owner-only.

The reaper predicate (task.done()) is an unambiguous signal that cannot amputate a live turn regardless of age. The registration-window grace period (30 s) is belt-and-suspenders since no await exists between the slot-set and current_task() capture. Per-entry try/except in the sweep prevents one bad entry from aborting the loop. The double-release race (reaper + turn's own finally) is handled by idempotent pop(key, None) on all three dicts. The test suite covers every stated invariant. No regressions to existing callers of write_runtime_status (field is opt-in, schema_version is a reader hint, file mode is a tightening).

No files require special attention.

Important Files Changed

Filename	Overview
gateway/run.py	Core runtime changes: adds _running_agent_tasks dict, task-liveness reaper loop (_reap_dead_running_agents + _reap_dead_running_agents_loop), task capture in _handle_message, and task-handle cleanup in _release_running_agent_state. Grace-period logic and per-entry crash isolation are sound.
gateway/status.py	Adds active_agent_keys parameter and schema_version=2 to write_runtime_status; switches _write_json_file to mode=0o600 (TOCTOU-safe atomic write). Read-modify-write semantics mean an omitted active_agent_keys retains the previous value — intentional and handled by _persist_active_agents always passing the field.
tests/gateway/test_quiescence_reaper.py	New test file covering all key reaper invariants: dead-task reaped (AC-4), live-long-running task survives (AC-5), registration-window grace (AC-5b), crash isolation (AC-6), idempotent double-release (AC-10), and the async loop's continued execution after a sweep exception.
tests/gateway/test_quiescence_status.py	New test file covering active_agent_keys persistence, schema_version=2, back-compat when the field is omitted, explicit empty-list semantics, and the 0o600 file-mode requirement.
tests/gateway/test_quiescence_task_tracking.py	New test file verifying task-handle release on slot clear, idempotent double-release, _persist_active_agents writing keys, and sentinel exclusion from the snapshot.
tests/gateway/test_session_race_guard.py	Existing race-guard test updated to include _running_agent_tasks = {} in the test runner fixture, keeping it in sync with the new dict added to __init__.
tests/gateway/test_status.py	Existing status test updated to expect mode=0o600 in the atomic_json_write call signature, correctly reflecting the new 0600 file permission.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant H as _handle_message
    participant RA as _running_agents
    participant RAT as _running_agent_tasks
    participant PAA as _persist_active_agents
    participant WRS as write_runtime_status
    participant Reaper as _reap_dead_running_agents_loop

    Note over H: Turn starts (NO await before task capture)
    H->>RA: "[key] = real_agent"
    H->>RAT: "[key] = asyncio.current_task()"
    H->>PAA: _persist_active_agents()
    PAA->>WRS: "active_agents=N, active_agent_keys=[key,...]"

    Note over Reaper: Periodic sweep (~60s)
    loop each _running_agents entry
        alt sentinel
            Reaper-->>Reaper: skip (never reap)
        else "task.done() == False"
            Reaper-->>Reaper: skip (live turn, any age)
        else "task.done() == True"
            Reaper->>RA: _release_running_agent_state(key)
            RA->>RAT: pop(key)
            RA->>PAA: _persist_active_agents()
            PAA->>WRS: "active_agent_keys=[...without key]"
        else "no task + age <= 30s"
            Reaper-->>Reaper: skip (registration window)
        else "no task + age > 30s"
            Reaper->>RA: _release_running_agent_state(key)
        end
    end

    Note over H: Turn ends (normal unwind)
    H->>RA: _release_running_agent_state(key)
    RA->>RAT: pop(key)
    RA->>PAA: _persist_active_agents()
    PAA->>WRS: "active_agents=N-1, active_agent_keys=[...without key]"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant H as _handle_message
    participant RA as _running_agents
    participant RAT as _running_agent_tasks
    participant PAA as _persist_active_agents
    participant WRS as write_runtime_status
    participant Reaper as _reap_dead_running_agents_loop

    Note over H: Turn starts (NO await before task capture)
    H->>RA: "[key] = real_agent"
    H->>RAT: "[key] = asyncio.current_task()"
    H->>PAA: _persist_active_agents()
    PAA->>WRS: "active_agents=N, active_agent_keys=[key,...]"

    Note over Reaper: Periodic sweep (~60s)
    loop each _running_agents entry
        alt sentinel
            Reaper-->>Reaper: skip (never reap)
        else "task.done() == False"
            Reaper-->>Reaper: skip (live turn, any age)
        else "task.done() == True"
            Reaper->>RA: _release_running_agent_state(key)
            RA->>RAT: pop(key)
            RA->>PAA: _persist_active_agents()
            PAA->>WRS: "active_agent_keys=[...without key]"
        else "no task + age <= 30s"
            Reaper-->>Reaper: skip (registration window)
        else "no task + age > 30s"
            Reaper->>RA: _release_running_agent_state(key)
        end
    end

    Note over H: Turn ends (normal unwind)
    H->>RA: _release_running_agent_state(key)
    RA->>RAT: pop(key)
    RA->>PAA: _persist_active_agents()
    PAA->>WRS: "active_agents=N-1, active_agent_keys=[...without key]"

Loading

_{Reviews (2): Last reviewed commit: "feat(gateway): task-liveness reaper for ..." | Re-trigger Greptile}