Skip to content

fix: serialize TUI gateway websocket sends - #48446

Closed
supplefrog wants to merge 1 commit into
NousResearch:mainfrom
supplefrog:fix/48445-ws-send-serialization
Closed

fix: serialize TUI gateway websocket sends#48446
supplefrog wants to merge 1 commit into
NousResearch:mainfrom
supplefrog:fix/48445-ws-send-serialization

Conversation

@supplefrog

@supplefrog supplefrog commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • serialize per-socket TUI gateway WebSocket sends with an asyncio lock
  • keep the existing loop-stall behavior: slow writes do not latch the transport closed
  • add regression coverage proving concurrent writer paths cannot overlap send_text() on the same socket

Fixes #48445

Test Plan

  • .venv/Scripts/python.exe -m pytest tests/test_tui_gateway_ws.py -q -o addopts= — 4 passed
  • .venv/Scripts/python.exe -m pytest tests/test_tui_gateway_server.py tests/test_tui_gateway_ws.py -q -o addopts= — 281 passed, 1 warning

Notes

@alt-glitch alt-glitch added type/bug Something isn't working comp/tui Terminal UI (ui-tui/ + tui_gateway/) comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists labels Jun 18, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related: #42938 (Windows ProactorEventLoop _WS_WRITE_TIMEOUT_S too aggressive — same tui_gateway/ws.py disconnect-under-load symptom, different mechanism), #42983 (dedicated WS write executor + keep-alive on loop stall — competing fix for the same overlapping-send path, different mechanism), #39393 (WSTransport write-path test coverage). This PR takes the serialization approach: an asyncio.Lock around _safe_send() so concurrent worker paths can't overlap send_text() on one socket.

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

✅ Looks Good

  • Well-scoped fix: Only 2 files touched (ws.py + test), addressing exactly the issue described in #48445
  • Clear root cause: The comment explains the problem — concurrent send_text() calls from multiple worker paths during high-output turns can overlap and cause issues
  • Correct solution: asyncio.Lock() properly serializes writes; check of self._closed inside the lock prevents sending on a closed socket
  • Good test coverage: New test (test_ws_transport_serializes_concurrent_sends) verifies:
    • Two concurrent writes don't overlap (max_active_sends == 1)
    • Both writes complete successfully
    • Transport remains open after concurrent writes
  • No debug artifacts or secrets
  • Follows existing patterns in the file (async lock pattern consistent with other async code in the codebase)

Minor Note

  • The test uses asyncio.sleep(0.05) to simulate work — this is appropriate for a concurrency test.

Reviewed by Hermes Agent

@teknium1

Copy link
Copy Markdown
Contributor

Closing in favor of #55545, which merged the GIL-pressure WS-disconnect fix to main (commits ebb81f10c + 9d10dcd49, contributor authorship preserved).

Your PR correctly identified the disconnect symptom, but the serialize-sends approach targets the "overlapping send_text()" theory — and that turned out not to be the mechanism. WSTransport._safe_send already only latches the transport closed on a real socket error, never on a loop stall, so concurrent sends weren't the cause. The actual root cause was event-loop starvation under GIL-heavy turns: uvicorn's keepalive ping misses its pong deadline on the starved loop and the connection is declared dead.

The merged fix addresses that directly — loopback ws-ping window widened to ride out stalls, per-token frame coalescing to cut loop wakeups, and frontend-polled RPCs routed off the read loop. Thanks for the well-mapped report on the original issue (#48445) — the file/line citations made tracing this fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery comp/tui Terminal UI (ui-tui/ + tui_gateway/) P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Desktop gateway WebSocket disconnects during long foreground tasks

4 participants