feat(turnover): skip registry-known-exhausted subs, and leave a durable handoff when a turn is cut

[Kyzcreig]

Why

The 2026-09-21 #apollo pattern, three times in one night: the primary sub hits its 5h/7d cap mid-turn, the gateway walks the fallback chain (claude-bpx-17 → apx-1 → … → apx-15, most also exhausted), spends ~60 s emitting "Rate limited — switching to fallback provider..." ×10, then dies. The next session reconstructs state from Discord scrollback — 15-25 min per turnover — and the work in flight at the cut is silently lost.

Two independent defects, one PR each half.

(A) The walker tries subs the usage registry already knows are dead

agent/quota_registry_gate.py reads the published usage snapshot (var/usage-portal/site/usage.json) and prunes, in one pass before any client is constructed, every chain entry whose 5h or 7d window is measurably rejected with a reset beyond a 5-minute horizon. The loop's quota-failover site emits one skipping N exhausted subs line instead of N. When the whole tail is pruned it fails fast naming the soonest reset.

Conservative by construction — missing file, corrupt JSON, unknown provider, stale observation (>6 h), a rejection with no reset time, and scoped per-model (Fable) allowances all fail OPEN to the historical walk. Only a quota reason (rate_limit / billing / upstream_rate_limit) triggers the gate; a transport timeout says nothing about quota and walks as before.

Measured against the live registry, 2026-09-21 ~02:10 PT

registry: 21 claude accounts published

BEFORE: 21 providers attempted, 21 "switching to fallback provider..." lines
AFTER:   4 providers attempted, 1 line
         ("⚠️ Rate limited — skipping 17 exhausted subs (quota registry) …")

requests avoided: 17
live config.yaml chain (5 entries): 2 pruned, 3 survive
  -> ['claude-apx-14', 'claude-apx-16', 'openai-codex']

Per-sub verdicts spot-checked against the raw usage.json windows (e.g. apx-10 five_hour 102.0 rejected 2.1h → SKIP; apx-14 five_hour 90.0 allowed_warning → ELIGIBLE).

That measurement found a real bug the unit tests could not see. _coerce_epoch parsed only numeric reset values; the registry publishes ISO-8601 Z strings, so against the live payload the first implementation pruned nothing — a vacuous pass, because the tests used epoch floats. Fixed, with ISO tests added.

(B) No durable handoff at the cut

agent/turn_handoff.py persists the in-flight turn state to $HERMES_HOME/state/turn-handoff/<session_key>.json: last user message id + text, the assistant's in-progress text, every tool call issued this turn with its result truncated to 500 chars and an explicit flag on the one that never completed (the most valuable single fact), and the open todo items.

• Written from the conversation loop's terminal provider-failure return, which appends a 3-line notice.
• Consumed by the turn prologue into the next turn, via the existing plugin/gateway user-context channel (so the ephemeral system prompt stays byte-stable).
• Consume-once; expires after 24 h; corrupt and expired files are deleted rather than retried forever.
• Session keys are sanitized + digest-disambiguated, so a key with separators or traversal cannot escape the handoff root.
• Every entry point swallows its own failures — a dying turn must not die twice.

/resume-handoff is registered in the command registry (alias resume_handoff) and dispatched in gateway/run.py, so the notice names a command the dispatcher actually knows.

Notes for the reviewer

• /resume-handoff is listed in _SLACK_VIA_HERMES_ONLY. Slack's 50-slash cap would otherwise have evicted /usage — caught by tests/hermes_cli/test_commands.py::TestSlackNativeSlashes::test_telegram_parity, and confirmed as mine by a clean-base control run (green on base, red with my registry entry).
• No live tree was deployed or synced.

Verification

scripts/run_tests.sh, sandboxed HOME, 216 passed / 0 failed, rc=0:

=== Summary: 13 files, 216 tests passed, 0 failed (100% complete) in 12.3s ===

New: test_quota_registry_gate (27), test_quota_gate_fallback_wiring (9), test_turn_handoff (28), test_turn_handoff_wiring (9), test_resume_handoff_command (10). Neighbours re-run: test_commands, test_turn_context, test_turn_context_overflow_warning, test_conversation_loop_interpreter_shutdown, test_lazy_command_exports, test_fallback_announce, test_quota_window_cooldown, test_fallback_reason_threading.

Every new module was RED first: ModuleNotFoundError / ImportError before implementation, and the ISO-8601 pair failed for the right reason before the parser fix.

[Kyzcreig]

Review round 1 addressed in 430e7b0.

1. Recovered subscriptions return on later turns

• The configured _fallback_chain is no longer destructively pruned.
• Registry-known exhausted providers are memoized in _quota_gate_skipped_providers and skipped locally before client construction.
• build_turn_context() clears every snapshot-derived gate field before primary restoration on each turn.
• Added a two-turn regression: apx-1 is exhausted on turn 1 (skipped for apx-2), healthy on turn 2, and is selected again. The source chain remains unchanged.
• RED proof: restoring the destructive mutation makes the test fail because apx-1 disappears from the cached agent.

2. Fail-fast message and one-line cascade are now live consumers

• The terminal failure path calls append_quota_exhaustion_message(); an all-dead tail now surfaces the soonest resets in 6h in final_response.
• rate_limited_status_line() emits at most once per turn. The new emitted-sequence regression drives a 21-entry chain (17 dead, 4 eligible) across all surviving failures and asserts the exact platform-status list contains one line total.
• Removed the dead _quota_gate_summary_line / _quota_gate_skipped_count attributes.
• RED proofs: removing the dedupe yields 5 emitted lines; removing the terminal consumer drops the reset text and fails the test.

3. Windows footguns fixed

• Added explicit UTF-8 to the one os.fdopen() and three Path.read_text() sites.
• python3 scripts/check-windows-footguns.py --all -> No Windows footguns found (1214 files).

Verification:

• 14 focused files: 246 passed, 0 failed via scripts/run_tests.sh with sandboxed HOME.
• Ruff: all changed Python files clean.
• git diff --check: clean.
• Live registry re-measurement: 21 Claude accounts, 17 skipped locally, 4 eligible, 17 network attempts avoided, 1 emitted status line.

No live tree deployed or synced.

[Kyzcreig]

🔁 re-enqueue for FleetReview (Apollo): head's terminal record was stamped ERROR by a router restart (release cut 20260921T104203Z / sibling cuts 10:25-10:32Z), not by a review verdict; close/reopen re-submits the unchanged head. No code change.

[Kyzcreig]

Review round 2 rework pushed at 4075faaaeb44bdc2eb14100e6c5e19bf14e5c2e9.

/resume-handoff now previews the saved payload without consuming it; the next real model-turn seam consumes and injects the original request, assistant progress, tool calls, and todos exactly once. The regression test drives the actual GatewaySlashCommandsMixin._handle_resume_handoff_command handler, then consume_handoff_context twice.

RED before implementation: next-turn context was empty after command dispatch. GREEN after implementation: focused feature test 10/10; broader turnover/turn-context/command suite 174/174. Ruff, Windows footgun scan, and git diff --check pass. No deploy performed.

[Kyzcreig]

Round-3 rework pushed at df71280579dde2839ce23550e4822b21e6c70407.

The handoff now reads the runtime-shaped _current_streamed_assistant_text, strips think blocks through the agent helper, merges it with already-materialized assistant rows without repeating overlapping text, and persists early provider cuts when only the original user request and/or open todos exist.

TDD evidence: the four new cases first failed at 4075faaaeb (no file/notice for user + partial stream; streamed suffix absent beside a materialized row; user-only early cut returned None). They now pass, including capture -> consume-on-next-turn -> consume-once round trip with original request, visible partial stream, and open todo.

Verification:

• canonical focused suite: 8 files, 125 tests passed, 0 failed
• Ruff: All checks passed!
• Windows footguns: No Windows footguns found (1214 files)
• git diff --check: clean

No live tree deployed or synced.

[Kyzcreig]

Round-4 rework @ addf4b6c0aeef6f6ecc57677dfb614332e670d9c.

Finding 4 (blocking) fixed — the gate was inert on 8 of 9 gateways. default_snapshot_path() now resolves through hermes_constants.get_default_hermes_root() instead of get_hermes_home(). Measured with the real resolver, one fresh interpreter per home:

BEFORE                                          AFTER
~/.hermes                    21 accounts        21 accounts
~/.hermes/profiles/aegis      0 accounts        21 accounts
~/.hermes/profiles/argus      0 accounts        21 accounts
~/.hermes/profiles/daedalus-opus  0 accounts    21 accounts
/opt/data/profiles/coder  -> /opt/data/profiles/coder/var/...  ->  /opt/data/var/...  (Docker rule)

Same chain pruned identically under both homes now: 21 entries -> 16 skipped / 5 eligible under the root home AND under profiles/aegis (was 0 skipped under the profile home).

Three regressions added, all RED first: profile-shaped home resolves to the root snapshot; the non-profile layout resolves byte-identically to today; one chain prunes identically under both homes. Fail-open on a genuinely missing file is pinned separately and unchanged.

Class-sweep honored: turn_handoff.py state/turn-handoff is per-profile state and was left alone, per the review.

Non-blocking item closed rather than deferred. prune_expired_handoffs() now has a production caller: the write path sweeps after a successful write, so an abandoned session_key no longer leaks its file past the 24h TTL. Guarded — a failing sweep can never cost a handoff that was already written (pinned by a test that makes the sweep raise).

Verification: scripts/run_tests.sh sandboxed HOME, 7 files / 177 passed, 0 failed; ruff All checks passed!; ✓ No Windows footguns found (1214 files); git diff --check clean. No live tree deployed or synced.

[Kyzcreig]

🤖 merged-by: apollo · lane: boil-ocean-aged · gate: BYPASS: FR PAUSED by Ace ruling 2026-09-22 (state/fleetreview-pause-20260922.md); Apollo-reviewed lands via bypass · why: 0/6 commits on main by patch-id; rebased clean onto 7ed45d9 (was 206 behind), diff limited to PR's own 14 files; 97 passed own tests; gateway/cli command-adjacent suites show the same 11 local-env failures on main and PR (no new)

[ang-prism]

FleetReview

FleetReview's daily member-call budget is spent (600/600 for 2026-09-26 UTC); review skipped.

---

FleetReview · reviewKind: skipped-budget