Skip to content

fix(desktop): probe half-open gateway socket on wake and reconnect - #90769

Closed
Owen-narcissus wants to merge 1 commit into
NousResearch:mainfrom
Owen-narcissus:fix/wake-half-open-socket-reconnect
Closed

fix(desktop): probe half-open gateway socket on wake and reconnect#90769
Owen-narcissus wants to merge 1 commit into
NousResearch:mainfrom
Owen-narcissus:fix/wake-half-open-socket-reconnect

Conversation

@Owen-narcissus

Copy link
Copy Markdown

Summary

Fix a desktop hang where, after sleep/wake or a silent network drop, typing a new prompt and pressing Enter appears to do nothing until the app is restarted.

Problem

macOS sleep (or a network switch) can leave the renderer's WebSocket half-open: the TCP connection dies without a close event, so connectionState still reads 'open' while every RPC hangs until its per-call timeout. The wake nudges in useGatewayBoot (reconnectNow) skipped reconnect whenever the socket reported open, so the dead socket was never rebuilt.

The first thing the user types after waking hits prompt.submit, whose request timeout is 30 minutes (PROMPT_SUBMIT_REQUEST_TIMEOUT_MS — deliberately long because turn ACKs can legitimately take minutes, see #55024). The message silently hangs for that whole window. Restarting the app rebuilds the socket, which is why only a restart appeared to fix it.

On the backend, the disconnected-but-idle session then gets reaped by the WS-orphan reaper 20s after the socket drops, compounding the client/server state mismatch.

Changes

  • Backend: add a minimal ping JSON-RPC method (tui_gateway/server.py) answered synchronously on the WS reader thread — the round-trip measures socket health, not backend load, so it works even mid-turn.
  • Desktop: on wake signals, reconnectNow now probes the open-looking socket with a 5-second-bounded ping and force-closes it on failure, letting the existing reconnect machinery (backoff, resetTileRuntimeBindings, refreshSessions) take over. A pre-ping backend answering -32601 (method not found) proves the socket is alive and is deliberately not reconnected, so version-skewed backends can't spin the reconnect loop.
  • Tests:
    • desktop: half-open socket → force-close + fresh socket dialed; healthy socket → untouched; -32601 backend → untouched (16/16 in use-gateway-boot.test.tsx).
    • backend: ping envelope contract (tests/tui_gateway/test_ping_probe.py).

Verification

  • npx vitest run src/app/gateway/hooks/use-gateway-boot.test.tsx — 16/16 passed.
  • Related prompt/submit regression suites — 219/219 passed.
  • tsc --noEmit clean; npm run pack succeeded for macOS arm64.
  • The local desktop app was rebuilt and relaunched with the fix.

Scope

This does not change the 30-minute prompt.submit timeout (that is intentional for long turns); it ensures a dead connection fails fast before a submit ever hangs against it. Unrelated working-tree files are not included.

macOS sleep/wake (or a silent network drop) can leave the renderer's
WebSocket half-open: no close event fires, so connectionState stays
'open' while every RPC hangs until its per-call timeout. prompt.submit's
timeout is 30 minutes, so the user's next message reads as "enter does
nothing until I restart the app".

- Add a minimal ping RPC (tui_gateway/server.py) answered synchronously
  on the WS reader thread.
- On wake signals, reconnectNow now probes the open-looking socket with a
  5s-bounded ping and force-closes it on failure, letting the existing
  reconnect machinery (backoff, tile rebinding, session refresh) take
  over. A pre-ping backend answering -32601 is treated as healthy.
- Tests: half-open socket force-reconnects; healthy socket untouched;
  method-not-found backend untouched; backend ping envelope contract.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/desktop Electron desktop app (apps/desktop/*) comp/tui Terminal UI (ui-tui/ + tui_gateway/) labels Aug 20, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #89092: both repair Desktop half-open sockets after wake, but #89092 force-redials on recovery signals while this PR probes liveness and retains a responsive version-skewed socket. Consider the recovery trade-off when choosing a canonical fix.

@Owen-narcissus

Copy link
Copy Markdown
Author

Thanks for the triage note linking #89092. Since both PRs target the same half-open-socket class after wake, here's how this one differs:

#89092 (force-redial on recovery): on power resume / network online / gateway reconnect, unconditionally closes the socket when it still reports OPEN and redials immediately. Simple, no backend changes — but every wake/online event tears down and rebuilds the connection even when it was healthy, which churns the session on each resume.

This PR (#90769, probe-then-act): adds a minimal ping RPC on the backend (answered synchronously on the WS reader thread, no session state touched). On wake signals, the open-looking socket is probed with a 5s-bounded ping, and force-closed only if the probe fails; a backend that answers (even -32601 method-not-found) is treated as healthy. Cost: one tiny backend RPC plus a probe round-trip. Benefit: a healthy connection is never disturbed — no disconnect flash on ordinary wakes, and the existing backoff/reconnect machinery takes over only when the socket is actually dead.

Both approaches fix the reported "enter does nothing until I restart the app" symptom; the difference is whether recovery is unconditional (redial on every wake) or verified (redial only when the probe proves the socket is dead). The two are compatible — a probe-then-redial path could sit on top of #89092's force-redial wiring if maintainers prefer that direction.

Tests: half-open socket force-reconnects; healthy socket untouched; method-not-found backend untouched; backend ping envelope contract.

@Julian-Bob

Copy link
Copy Markdown

Additional data point: Windows + WSL2 + remote gateway (same half-open socket symptom)

Same root cause confirmed on a Windows 11 host with Hermes Desktop connected to a WSL2 backend via remote gateway (http://localhost:9119, dashboard bound to 0.0.0.0).

Symptom: Desktop intermittently loses the connection; the user must click "Save & Reconnect" in Gateway settings. Typing after a silent drop appears to do nothing until the connection is rebuilt.

Backend log evidence (gui.log, tui_gateway.ws):

10:46:19  ws closed peer=127.0.0.1:49884 reason=client_disconnect(code=1005,reason=) messages=10760 ... detached_sessions=3
10:46:20  ws accepted peer=127.0.0.1:61932
10:47:48  ws closed peer=127.0.0.1:61932 reason=client_disconnect(code=1005,reason=) messages=163 ... detached_sessions=0
10:47:49  ws accepted peer=127.0.0.1:49670
10:53:48  ws closed peer=127.0.0.1:49670 reason=client_disconnect(code=1005,reason=) messages=962 ... detached_sessions=1
10:53:49  ws accepted peer=127.0.0.1:63207
  • All disconnects are code=1005 (no close frame — TCP silently gone), consistent with a half-open socket after sleep/wake or a brief Wi-Fi drop.
  • Server side is healthy: 0 service restarts, 0 event loop stalled warnings on the affected days, 0 ws write slow entries. The backend never closed these sockets.
  • The detached_sessions=N counter on the closed sockets matches the WS-orphan reaper described in this PR (session reaped ~20s after the socket drops).

Version: backend v0.20.4 (includes #55545 loopback ping fix; note the keepalive ping stays active on non-loopback binds, so a GIL stall can still false-positive there — but in this case no stalls were logged, the drops are client-side).

This confirms the half-open socket scenario also occurs on Windows/WSL2 remote-gateway setups, not just macOS local. A ping-based wake probe (and ideally a shorter PROMPT_SUBMIT_REQUEST_TIMEOUT_MS fallback when the socket is known-dead) would fix the same hang here.

@teknium1

Copy link
Copy Markdown
Contributor

Merged via #93694 (rebase-merge, your commit preserved with full authorship — thank you!). We kept your ping RPC + probe exactly as designed, and added two things on top: the backend-update success path now nudges the same reconnect (fixes the 'force-quit after remote update' report), and the old blind gateway.close() on every wake signal was removed in favor of your probe — your own healthy-socket tests showed the blind close was churning good connections. Closing as landed.

@teknium1 teknium1 closed this Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/desktop Electron desktop app (apps/desktop/*) comp/tui Terminal UI (ui-tui/ + tui_gateway/) P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants