fix(dashboard): close PTY WebSocket on child EOF to stop FD leak (#54028) by teknium1 · Pull Request #54123 · NousResearch/hermes-agent

teknium1 · 2026-06-28T09:19:42Z

Summary

Dashboard /chat PTY sessions no longer leak file descriptors when the browser socket drops uncleanly.

Root cause: the /api/pty reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. Dashboard auto-reconnect (#52962) then spawns a fresh PTY on every dropped socket, stacking on the orphaned one until the gateway hits EMFILE (Errno 24) within hours.

Changes

hermes_cli/web_server.py: the pump_pty_to_ws reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks. That unblocks the writer's ws.receive() so the existing finally runs bridge.close() and reaps the PTY. The writer loop guards ws.receive() against the RuntimeError Starlette raises once the socket is already closed.
tests/hermes_cli/test_web_server_pty_reconnect.py: regression test — a client that reads one frame and never disconnects must still see the server tear down the socket and reap the bridge on child EOF.

Validation

	Before fix	After fix
Child EOF, half-open client socket	handler blocks in `ws.receive()` forever; PTY fd + child leak	server closes WS → `bridge.close()` runs
New regression test	hangs (60s timeout)	passes in <1s
`test_web_server_pty_reconnect.py`	—	4 passed

Reported by @fifteenzhang.

Fixes #54028

Infographic

github-actions · 2026-06-28T09:21:41Z

🔎 Lint report: `hermes/hermes-bac67ebe` vs `origin/main`

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11637 on HEAD, 11637 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 6115 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@fifteenzhang

The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes #54028

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix (cherry picked from commit 6d879d4)

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix

@fifteenzhang

…sResearch#54028) (NousResearch#54123) * fix(dashboard): close PTY WebSocket on child EOF to stop FD leak The /api/pty handler's reader task returns on child EOF, but the writer loop stayed blocked on ws.receive() until the browser sent a disconnect. When the browser socket is half-open (no FIN delivered — common on macOS/launchd), that disconnect never arrives, so the handler never reaches its finally and the PTY master fd + child process leak. With dashboard auto-reconnect (NousResearch#52962), every dropped socket then spawns a fresh PTY on top of the orphaned one, exhausting file descriptors within hours (EMFILE / Errno 24). Fix: the reader task now closes the WebSocket in a finally when the child EOFs or the send side breaks, which unblocks ws.receive() so the existing finally runs bridge.close(). The writer loop also guards ws.receive() against the RuntimeError Starlette raises once the socket is closed. Reported by @fifteenzhang. Fixes NousResearch#54028 * docs: add infographic for NousResearch#54028 PTY FD leak fix

alt-glitch added type/bug Something isn't working comp/dashboard Web dashboard / control panel UI (dashboard/, landing) P1 High — major feature broken, no workaround labels Jun 28, 2026

teknium1 added 2 commits June 28, 2026 02:41

docs: add infographic for #54028 PTY FD leak fix

b31695d

teknium1 force-pushed the hermes/hermes-bac67ebe branch from 49248ee to b31695d Compare June 28, 2026 09:42

teknium1 merged commit 6d879d4 into main Jun 28, 2026

teknium1 deleted the hermes/hermes-bac67ebe branch June 28, 2026 09:42

teknium1 mentioned this pull request Jun 28, 2026

[Bug]: FD leak regression after PTY reconnect PRs (#52962 #53227) #54028

Closed

1 task

liuhao1024 mentioned this pull request Jun 28, 2026

fix(dashboard): race PTY reader against WS receive to prevent FD leak (#54028) #54047

Closed

13 tasks

Navigation Menu

Uh oh!

fix(dashboard): close PTY WebSocket on child EOF to stop FD leak (#54028) - #54123

fix(dashboard): close PTY WebSocket on child EOF to stop FD leak (#54028)#54123
teknium1 merged 2 commits into
mainfrom
hermes/hermes-bac67ebe

teknium1 commented Jun 28, 2026

Uh oh!

github-actions Bot commented Jun 28, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

teknium1 commented Jun 28, 2026

Summary

Changes

Validation

Infographic

Uh oh!

github-actions Bot commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔎 Lint report: hermes/hermes-bac67ebe vs origin/main

ruff

ty (type checker)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 28, 2026 •

edited

Loading

🔎 Lint report: `hermes/hermes-bac67ebe` vs `origin/main`