fix(gateway): recover persisted turns after WebSocket replacement by 100yenadmin · Pull Request #83166 · NousResearch/hermes-agent

100yenadmin · 2026-08-10T11:26:09Z

What changed

negotiate a browser-compatible 15-second application heartbeat with a 45-second acknowledgement deadline through gateway.ready
invalidate only the expired socket generation, reject its pending calls, ignore late frames, and preserve existing Desktop reconnect ownership/backoff
advance a monotonic Desktop connection epoch per profile and reactivate the routed session after every real reconnect, including React-batched reconnects
make prompt.submit non-replayable after ambiguous transport outcomes; reconcile authoritative session state or restore the draft/attachments with a retryable delivery-not-confirmed error
repair stale running/inflight state only when a registered prompt worker is confirmed dead
centralize session transport rebinding so closed, detached, and heartbeat-expired owners can be replaced while another recently live WebSocket owner receives an explicit conflict
add TUI heartbeat capability negotiation, bounded 1–30 second reconnect backoff, and intentional kill() suppression

Older backends remain compatible because heartbeat timers start only when the backend advertises the capability.

Fault-injection proof

hermes-ws-recovery-v1 passes on head 8b145ca89:

Scenario	Result
Disconnect after acceptance, before `message.complete`	Replacement-socket activation returns exactly one user row and one persisted assistant final; dead worker state returns to `idle`
Silent packet blackhole with no native `close`	Heartbeat invalidates the exact socket, one replacement opens, authoritative activation returns the final, and `prompt.submit` remains at exactly one call
Ambiguous submit accepted	No automatic replay; persisted/running state is accepted from activation
Ambiguous submit unproven	Optimistic row removed, composer payload restored, retryable delivery-not-confirmed error shown
Healthy second client	Existing recently live stream owner retains ownership; replacement receives conflict
Intentional shutdown	Zero reconnects
Older backend without capability	No heartbeat timer; connection remains compatible

Focused local checks on the exact head:

Desktop/shared Vitest: 5 files, 126 tests passed
Desktop secondary-gateway reconnect test: 1 test passed
TUI Vitest: 1 file, 18 tests passed
Gateway Python: 3 files, 545 tests passed
Desktop, shared, and TUI TypeScript checks passed
changed Desktop/shared/TUI ESLint: 0 errors (one pre-existing hook dependency warning at an unchanged line)
changed Python Ruff and git diff --check: passed

Source PR credit

This combines the still-relevant behavior and tests from:

Javier SM — fix: reconcile session state after desktop reconnect #71475 (connection epochs and dead-worker activation cleanup). The obsolete readiness-auth commit is intentionally excluded.
Tamaz Sujashvili — [codex] Fix desktop stale WebSocket prompt timeout #40168 (Desktop heartbeat/reconnect recovery behavior), preserved through a co-author trailer on the synthesized current-architecture commit.
Indigo Karasu — fix(tui): add WebSocket heartbeat + auto-reconnect for silent drops (#32997) #60727 (TUI heartbeat and reconnect behavior), preserved as the original commits plus a co-author trailer where shared integration was synthesized.

The source PRs were used as behavioral specifications and test sources; obsolete code was not blindly cherry-picked.

Scope boundary

This PR does not add a durable event outbox or delivery acknowledgement, change Relay concurrency/readiness authentication, alter proxy lifetime policy, or perform any deployment/backport. Passing proof establishes PR-head behavior only, not released or deployed customer behavior.

Fixes NousResearch#32997 The Ink TUI WebSocket client had no heartbeat and no auto-reconnect, so a silent connection drop (macOS sleep, proxy idle timeout, VPN reconnect) left the UI stranded at a dead socket with only Ctrl+C as an escape. Add a periodic ping keepalive plus dead-connection detection that forces a reconnect with exponential backoff; never reconnect after an intentional kill(). Supersedes NousResearch#30114 (which only restarted on the exit event and could not heal silent drops) by moving reconnect into the transport client and adding the heartbeat.

Co-authored-by: Tamaz Sujashvili <tamazsujashvili@labarum.onmicrosoft.com> Co-authored-by: Indigo Karasu <mx.indigo.karasu@gmail.com>

100yenadmin · 2026-08-10T11:26:39Z

Superseded by the current-head review request below; this request referred to an earlier PR head.

100yenadmin · 2026-08-10T11:30:13Z

@codex review current head 8b145ca. The final delta aligns backend stream-owner expiry with the negotiated 45-second heartbeat deadline, so a replacement socket can activate immediately instead of waiting behind a transport the client has declared dead. Please focus on message delivery and transport ownership.

100yenadmin · 2026-08-10T11:31:21Z

@OutThisLife could you approve the fork workflows and provide (or route) one current-head semantic review for message delivery and session transport ownership? The exact head is 8b145ca; both workflows are currently blocked at GitHub’s action_required approval gate, and the local disconnect-mid-turn/silent-blackhole harnesses are green without duplicate prompt or final rows.

100yenadmin · 2026-08-10T13:17:58Z

@alt-glitch @Teknium this isn't P2, remote gateways break every 1-4 hrs and become P0 without these fixes so at best this is a P1 given there are other "options" but remote gateway usage is P0-P1

100yenadmin · 2026-08-19T11:05:32Z

This PR bundles five separable concerns and is stale against current main, so rather than push a 1,682-line rebase I'm splitting it into single-concern PRs on current main (13ce0c5c67), each with its own tests.

Slice 1 is up: #89958 — feat(gateway): add gateway.ping heartbeat wire contract (the additive wire primitive; +78 lines, backward-compatible both directions). Every other slice consumes it.

Planned follow-ons:
2. server-side: expire silent stream owners + reconcile dead prompt workers
3. TUI client: heartbeat timer + bounded reconnect for silent drops
4. desktop client: the same, on the Electron gateway store
5. persisted-turn recovery on reconnect (the original headline)

I'll file them in dependency order as each is rebased and green. Once slice 1 lands (or is cherry-picked), I'll close this omnibus and track the rest against the slices. Holding this PR open only as the umbrella until then.

100yenadmin · 2026-08-19T12:55:37Z

Closing this omnibus — it's been split into single-concern PRs on current main, each with its own tests. That's a much fairer ask than this 28-file, +1682 diff, and it lets the independent pieces land without waiting on the rest:

feat(gateway): add gateway.ping heartbeat wire contract #89958 feat(gateway): add gateway.ping heartbeat wire contract — the additive wire primitive everything else reads
fix(gateway): reconcile finished prompt-run threads at session teardown #89978 fix(gateway): reconcile finished prompt-run threads at session teardown — standalone, no dependency
fix(gateway): expire silent stream owners and rebind live transports #89992 fix(gateway): expire silent stream owners and rebind live transports — the server-side half (stacks on feat(gateway): add gateway.ping heartbeat wire contract #89958)
fix(tui): heartbeat and bounded reconnect for silent WebSocket drops #89984 fix(tui): heartbeat and bounded reconnect for silent WebSocket drops — the ink TUI client half
feat(shared): heartbeat and socket-generation invalidation in JsonRpcGatewayClient #90012 feat(shared): heartbeat and socket-generation invalidation in JsonRpcGatewayClient — the shared client half

Two notes so nothing looks quietly dropped:

Scope narrowed honestly in fix(gateway): expire silent stream owners and rebind live transports #89992. The transport guard covers the session.activate / session.resume / prompt.submit ingress paths plus the queued-prompt drain. The two compute_host.py reattach sites are deliberately not guarded — they're a single-owner channel rebinding its own transport, not a competing-renderer steal, and guarding them would break the attach path. That needs a different ownership model if it's ever wanted.
The desktop store/hooks slice is not being re-filed. The remaining piece of this PR was the Electron-side connection-epoch and non-replayable prompt.submit work (~14 files). It's entangled with local restructuring of the desktop app that isn't appropriate to push upstream, and the generic behavior it was protecting is covered by the five PRs above. Better to leave it out than to send you a diff shaped around someone else's app layout.

Nothing is lost by closing this; the PRs above are the same work, current and green.

Stacks on NousResearch#89958 (the gateway.ping heartbeat wire contract), which provides the transport liveness signals this change reads (WSTransport.closed / last_inbound_at / mark_inbound). Part of NousResearch#83166. A reconnecting renderer could not reclaim its session stream while a dropped or silent socket still "owned" it. Introduce a bounded ownership window keyed to the negotiated heartbeat deadline: - _TRANSPORT_OWNERSHIP_LIVE_S (45s), _transport_is_recently_live(): a socket that is closed, unregistered, detached, or past the heartbeat window no longer blocks a reclaim. Stdio/test transports keep their historical behavior; the conflict contract is only for competing WS clients. - _bind_session_transport(): atomically rebinds a session unless another recently-live WS still owns it, returning False so callers can reject. - Route the session-stream ingress paths through the guard: session.activate, session.resume (incl. the fast-path and adopt-other-session reuse via the _reuse_live_payload -> _reuse_live_response rename), and prompt.submit. Each returns JSON-RPC 4091 "session stream is owned by another active client" when a live owner is present. - Move _live_session_payload's transport binding to the same guard. Rework note (verdict-flagged): the queued-prompt drain (server.py) is guarded inline rather than via _bind_session_transport because it already holds history_lock, which the helper re-acquires (non-reentrant) — a queued prompt from an old renderer no longer steals a recently-live owner. The compute-host reattach sites (compute_host.py) are intentionally out of this claim: they are the single-owner compute-host channel rebinding its own transport, not a competing-renderer steal vector, and refusing the bind there would break the attach path. The PR body claim is narrowed accordingly to the WS ingress paths plus the queued-prompt drain. Tests: a recently-live owner is protected from steal, an expired/released owner can be rebound (unit + WS loopback), and a queued prompt cannot steal a recently-live owner. The stale-busy-state reconciliation is a separate slice and is intentionally excluded here.

the client half of the gateway.ping heartbeat contract (NousResearch#89958); detects a silently-dropped socket via missed ping-acks and reconnects with bounded backoff; part of the NousResearch#83166 recovery series. (cherry picked from commit 57bd8fd)

…GatewayClient The shared-client half of the gateway.ping heartbeat contract (NousResearch#89958); tracks lastInboundAt, sends pings, invalidates a silently-dead socket. Part of NousResearch#83166. (cherry picked from commit 4c4a410)

Stacks on NousResearch#89958 (the gateway.ping heartbeat wire contract), which provides the transport liveness signals this change reads (WSTransport.closed / last_inbound_at / mark_inbound). Part of NousResearch#83166. A reconnecting renderer could not reclaim its session stream while a dropped or silent socket still "owned" it. Introduce a bounded ownership window keyed to the negotiated heartbeat deadline: - _TRANSPORT_OWNERSHIP_LIVE_S (45s), _transport_is_recently_live(): a socket that is closed, unregistered, detached, or past the heartbeat window no longer blocks a reclaim. Stdio/test transports keep their historical behavior; the conflict contract is only for competing WS clients. - _bind_session_transport(): atomically rebinds a session unless another recently-live WS still owns it, returning False so callers can reject. - Route the session-stream ingress paths through the guard: session.activate, session.resume (incl. the fast-path and adopt-other-session reuse via the _reuse_live_payload -> _reuse_live_response rename), and prompt.submit. Each returns JSON-RPC 4091 "session stream is owned by another active client" when a live owner is present. - Move _live_session_payload's transport binding to the same guard. Rework note (verdict-flagged): the queued-prompt drain (server.py) is guarded inline rather than via _bind_session_transport because it already holds history_lock, which the helper re-acquires (non-reentrant) — a queued prompt from an old renderer no longer steals a recently-live owner. The compute-host reattach sites (compute_host.py) are intentionally out of this claim: they are the single-owner compute-host channel rebinding its own transport, not a competing-renderer steal vector, and refusing the bind there would break the attach path. The PR body claim is narrowed accordingly to the WS ingress paths plus the queued-prompt drain. Tests: a recently-live owner is protected from steal, an expired/released owner can be rebound (unit + WS loopback), and a queued prompt cannot steal a recently-live owner. The stale-busy-state reconciliation is a separate slice and is intentionally excluded here. (cherry picked from commit b05a60b)

A finished or dead prompt-run worker thread could leave session state wedged: `running` and `inflight_turn` stayed pinned forever after the worker had already exited (e.g. a dropped transport). Reconnect callers read the live payload as their authority, so a stale busy projection stranded the client on a spinner. Reconcile that impossible state in `_live_session_payload` before it is returned: if the session is marked running but its `_run_thread` has exited, clear `running` and the inflight turn. A missing thread is left untouched (other paths briefly set `running` before registering a worker). This is a standalone fix, independent of the WebSocket heartbeat wire contract (NousResearch#89958); part of the NousResearch#83166 recovery work. Receipts: scripts/run_tests.sh tests/test_tui_gateway_server.py \ -k activate_clears_stale_busy fail-before (reconcile call removed): 1 failed pass-after: 1 passed, 0 failed (cherry picked from commit 8146527)

the client half of the gateway.ping heartbeat contract (NousResearch#89958); detects a silently-dropped socket via missed ping-acks and reconnects with bounded backoff; part of the NousResearch#83166 recovery series. (cherry picked from commit 57bd8fd)

…GatewayClient The shared-client half of the gateway.ping heartbeat contract (NousResearch#89958); tracks lastInboundAt, sends pings, invalidates a silently-dead socket. Part of NousResearch#83166. (cherry picked from commit 4c4a410)

Stacks on NousResearch#89958 (the gateway.ping heartbeat wire contract), which provides the transport liveness signals this change reads (WSTransport.closed / last_inbound_at / mark_inbound). Part of NousResearch#83166. A reconnecting renderer could not reclaim its session stream while a dropped or silent socket still "owned" it. Introduce a bounded ownership window keyed to the negotiated heartbeat deadline: - _TRANSPORT_OWNERSHIP_LIVE_S (45s), _transport_is_recently_live(): a socket that is closed, unregistered, detached, or past the heartbeat window no longer blocks a reclaim. Stdio/test transports keep their historical behavior; the conflict contract is only for competing WS clients. - _bind_session_transport(): atomically rebinds a session unless another recently-live WS still owns it, returning False so callers can reject. - Route the session-stream ingress paths through the guard: session.activate, session.resume (incl. the fast-path and adopt-other-session reuse via the _reuse_live_payload -> _reuse_live_response rename), and prompt.submit. Each returns JSON-RPC 4091 "session stream is owned by another active client" when a live owner is present. - Move _live_session_payload's transport binding to the same guard. Rework note (verdict-flagged): the queued-prompt drain (server.py) is guarded inline rather than via _bind_session_transport because it already holds history_lock, which the helper re-acquires (non-reentrant) — a queued prompt from an old renderer no longer steals a recently-live owner. The compute-host reattach sites (compute_host.py) are intentionally out of this claim: they are the single-owner compute-host channel rebinding its own transport, not a competing-renderer steal vector, and refusing the bind there would break the attach path. The PR body claim is narrowed accordingly to the WS ingress paths plus the queued-prompt drain. Tests: a recently-live owner is protected from steal, an expired/released owner can be rebound (unit + WS loopback), and a queued prompt cannot steal a recently-live owner. The stale-busy-state reconciliation is a separate slice and is intentionally excluded here. (cherry picked from commit b05a60b)

A finished or dead prompt-run worker thread could leave session state wedged: `running` and `inflight_turn` stayed pinned forever after the worker had already exited (e.g. a dropped transport). Reconnect callers read the live payload as their authority, so a stale busy projection stranded the client on a spinner. Reconcile that impossible state in `_live_session_payload` before it is returned: if the session is marked running but its `_run_thread` has exited, clear `running` and the inflight turn. A missing thread is left untouched (other paths briefly set `running` before registering a worker). This is a standalone fix, independent of the WebSocket heartbeat wire contract (NousResearch#89958); part of the NousResearch#83166 recovery work. Receipts: scripts/run_tests.sh tests/test_tui_gateway_server.py \ -k activate_clears_stale_busy fail-before (reconcile call removed): 1 failed pass-after: 1 passed, 0 failed (cherry picked from commit 8146527)

Stacks on NousResearch#89958 (the gateway.ping heartbeat wire contract), which provides the transport liveness signals this change reads (WSTransport.closed / last_inbound_at / mark_inbound). Part of NousResearch#83166. A reconnecting renderer could not reclaim its session stream while a dropped or silent socket still "owned" it. Introduce a bounded ownership window keyed to the negotiated heartbeat deadline: - _TRANSPORT_OWNERSHIP_LIVE_S (45s), _transport_is_recently_live(): a socket that is closed, unregistered, detached, or past the heartbeat window no longer blocks a reclaim. Stdio/test transports keep their historical behavior; the conflict contract is only for competing WS clients. - _bind_session_transport(): atomically rebinds a session unless another recently-live WS still owns it, returning False so callers can reject. - Route the session-stream ingress paths through the guard: session.activate, session.resume (incl. the fast-path and adopt-other-session reuse via the _reuse_live_payload -> _reuse_live_response rename), and prompt.submit. Each returns JSON-RPC 4091 "session stream is owned by another active client" when a live owner is present. - Move _live_session_payload's transport binding to the same guard. Rework note (verdict-flagged): the queued-prompt drain (server.py) is guarded inline rather than via _bind_session_transport because it already holds history_lock, which the helper re-acquires (non-reentrant) — a queued prompt from an old renderer no longer steals a recently-live owner. The compute-host reattach sites (compute_host.py) are intentionally out of this claim: they are the single-owner compute-host channel rebinding its own transport, not a competing-renderer steal vector, and refusing the bind there would break the attach path. The PR body claim is narrowed accordingly to the WS ingress paths plus the queued-prompt drain. Tests: a recently-live owner is protected from steal, an expired/released owner can be rebound (unit + WS loopback), and a queued prompt cannot steal a recently-live owner. The stale-busy-state reconciliation is a separate slice and is intentionally excluded here. (cherry picked from commit b05a60b) (cherry picked from commit d67bef8)

Eva and others added 6 commits August 10, 2026 18:24

test(desktop): reproduce websocket recovery gaps

2610b0f

fix(desktop): reconcile sessions after reconnect

afbd05d

fix(gateway): clear dead prompt worker state on activate

4b45ee5

fix(tui): acknowledge websocket heartbeats

a5dd8bb

fix(gateway): recover turns after websocket replacement

ed1706a

Co-authored-by: Tamaz Sujashvili <tamazsujashvili@labarum.onmicrosoft.com> Co-authored-by: Indigo Karasu <mx.indigo.karasu@gmail.com>

This was referenced Aug 10, 2026

fix: reconcile session state after desktop reconnect #71475

Open

[codex] Fix desktop stale WebSocket prompt timeout #40168

Draft

fix(tui): add WebSocket heartbeat + auto-reconnect for silent drops (#32997) #60727

Open

fix(desktop): await the active reconnect owner

a122d7b

100yenadmin mentioned this pull request Aug 10, 2026

Desktop/WebSocket reconnect must recover persisted turns without duplicates #83147

Closed

fix(gateway): expire silent stream owners on heartbeat deadline

8b145ca

This was referenced Aug 15, 2026

Salvage audit: rank r1–r20 Hermes fixes and decide upstream issue, PR, retain, or delete electricsheephq/evaOS-hermes-desktop-app-adapter#124

Open

feat(gateway): add gateway.ping heartbeat wire contract #89958

Open

100yenadmin mentioned this pull request Aug 19, 2026

feat(shared): heartbeat and socket-generation invalidation in JsonRpcGatewayClient #90012

Open

13 tasks

100yenadmin closed this Aug 19, 2026

Navigation Menu

Uh oh!

fix(gateway): recover persisted turns after WebSocket replacement - #83166

fix(gateway): recover persisted turns after WebSocket replacement#83166
100yenadmin wants to merge 8 commits into
NousResearch:mainfrom
100yenadmin:fix/desktop-websocket-recovery

100yenadmin commented Aug 10, 2026 •

edited

Loading

Uh oh!

100yenadmin commented Aug 10, 2026 •

edited

Loading

Uh oh!

100yenadmin commented Aug 10, 2026 •

edited

Loading

Uh oh!

100yenadmin commented Aug 10, 2026 •

edited

Loading

Uh oh!

100yenadmin commented Aug 10, 2026 •

edited

Loading

Uh oh!

100yenadmin commented Aug 19, 2026

Uh oh!

100yenadmin commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

100yenadmin commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Fault-injection proof

Source PR credit

Scope boundary

Uh oh!

100yenadmin commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

100yenadmin commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

100yenadmin commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

100yenadmin commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

100yenadmin commented Aug 19, 2026

Uh oh!

100yenadmin commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

100yenadmin commented Aug 10, 2026 •

edited

Loading

100yenadmin commented Aug 10, 2026 •

edited

Loading

100yenadmin commented Aug 10, 2026 •

edited

Loading

100yenadmin commented Aug 10, 2026 •

edited

Loading

100yenadmin commented Aug 10, 2026 •

edited

Loading