fix(tui): close slash_worker sessions on WebSocket disconnect - #31474
fix(tui): close slash_worker sessions on WebSocket disconnect#31474sea-monsters wants to merge 1 commit into
Conversation
|
Related to the slash_worker leak family: #21370, #21467, #22855, #24775. Main fix PR is #22863 which also addresses lifecycle management. This PR adds both server-side cleanup (ws.py finalize on disconnect) and worker self-protection (idle timeout + parent-PID check). Worth comparing scope with #22863. |
Three-layer defence-in-depth against orphaned slash_worker subprocesses: 1. **Server-side cleanup (P0)** — when a WebSocket disconnects, sessions marked close_on_disconnect=true (sidecar/short-lived) are finalised and their worker is killed. Normal TUI sessions still fall back to the stdio transport for historical reconnect compatibility. (Design from NousResearch#21401, adapted with permission.) 2. **Parent watchdog (P1, psutil)** — a daemon thread monitors the parent's PID + create_time fingerprint every 10 s and exits if the parent disappears. Handles crashes, SIGKILL, and PID reuse (critical on Windows). (Design from NousResearch#22863, adapted with permission.) 3. **Idle timeout + getppid() poll (P1, no deps)** — the main stdin loop uses select.select() with a 60 s timeout so it can periodically check os.getppid() and a 30-minute idle deadline. Works without psutil, adding defence even when the watchdog thread is not available. Also refactors session teardown into a shared _close_session_by_id() helper so that session.close RPC, WebSocket-disconnect cleanup, and server shutdown all use the identical code path. Co-authored-by: Hermes Agent
97df2ab to
2f668de
Compare
|
Thanks @alt-glitch for the references — I've reviewed both #22863 and #21401 in detail and restructured this PR to incorporate the best of all three approaches: Updated scope (force-pushed): P0 — Server-side cleanup (adopted #21401's design)
P1 — Worker self-protection (merged #22863's watchdog + our idle timeout)
The key differentiator from #22863 is:
Would love to hear your thoughts on whether this combined approach addresses the concerns from the related issues better than each PR individually. Happy to collaborate on merging this or splitting it into follow-ups to #21401 / #22863. |
|
Superseded by #42132 (merged), which closes the slash_worker subprocess leak via two guards: process-group kill on PTY teardown + a cross-platform parent-death watchdog in the worker. Closing as resolved — thanks for tackling this; the merged fix salvaged the process-group-kill and watchdog approaches with contributor authorship preserved. |
What does this PR do?
When a TUI WebSocket disconnects (browser tab closed, network drop, OS kills the tab), the session's
slash_workersubprocess stays alive indefinitely -- it's a persistentfor raw in sys.stdin:loop that never exits unless stdin is explicitly closed. These orphaned workers accumulate over time, wasting ~100MB RAM each.This PR fixes the lifecycle disconnect in two layers:
P0: Clean up sessions on WebSocket disconnect (
ws.py)handle_ws()now finalizes the affected sessions and closes theirslash_worker/ agent when the WebSocket transport disconnects, instead of only detaching the transport.P1: Worker self-protection (
slash_worker.py)Even if the P0 fix misses a code path, the worker now exits on its own:
Related Issue
No open issue. Found during a routine environment audit: 5 orphaned
slash_workerprocesses had accumulated over 6 days, consuming ~350MB RAM.Type of Change
Changes Made
tui_gateway/ws.py: inhandle_ws'sfinallyblock, call_finalize_session, close theslash_workerand agent, and remove the session from_sessionsfor every session that owned the closing transporttui_gateway/slash_worker.py: replace blockingfor raw in sys.stdinwithselect.select-based polling that checks idle timeout and parent-PID continuity every 60sHow to Test
--tui:hermes dashboard --port 8089 --tuips aux | grep slash_workershows a worker processAlternatively, test the worker self-protection directly (run in Python):
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests pass (235 existing tests pass)Screenshots / Logs
Before: 5 orphaned
slash_workerprocesses consuming ~350MB RAM