fix(dashboard): reap orphaned embedded-chat sessions to stop slash_worker leak

[benbarclay]

Summary

Since #38591 made the dashboard's embedded Chat surface unconditional, every browser refresh of /chat leaks a tui_gateway.slash_worker Python subprocess — they accumulate one-per-refresh until the box is buried in idle workers (user report: ~20 workers after a session of refreshing).

Root cause

The dashboard runs one long-lived in-process tui_gateway (the PTY child node attaches to it over /api/ws via HERMES_TUI_GATEWAY_URL rather than spawning its own Python gateway — see _resolve_chat_argv). So the topology is:

hermes dashboard  (long-lived web_server + in-process tui_gateway)
└─ /api/pty → node entry.js  (ATTACHES via /api/ws — does NOT spawn a gateway)
     in-process gateway spawns →  _SlashWorker → python -m tui_gateway.slash_worker

On every refresh the fresh Ink boot calls session.create (new sid + a fresh _SlashWorker via _deferred_build). The old tab's /api/ws disconnect, however, only detaches the transport in tui_gateway/ws.py (sess["transport"] = _stdio_transport) to allow reconnects — it never closes the old session or its slash worker. Because the dashboard process never exits, the detached _SlashWorker.proc.stdin write-end stays open forever, the worker never hits EOF, and it leaks.

This was dormant before #38591 because the whole /api/ws + session.create + _SlashWorker path was gated off on a plain dashboard.

Note: this is not a PTY signal-timing / atexit problem. Verified empirically that a worker whose owning process dies (even via os._exit(0) or SIGKILL) is always reaped via stdin-EOF; the leak is specifically the long-lived dashboard keeping detached sessions parked indefinitely.

Fix

Reap orphaned embedded-chat sessions at the session-lifecycle layer. On WS disconnect, schedule a grace-delayed reap of any session left orphaned (transport detached to stdio, not mid-turn). A quick reconnect / session.resume / prompt.submit rebinds a live transport and cancels the reap, preserving the intentional detach-for-reconnect window.

• tui_gateway/server.py: extract _teardown_session() (shared with session.close), add _ws_session_is_orphaned() + _schedule_ws_orphan_reap(). Gated by HERMES_TUI_WS_ORPHAN_REAP_GRACE_S (default 20s; 0 disables = pre-fix park-forever behaviour).
• tui_gateway/ws.py: schedule the reap for each detached session on disconnect.
• tests: reap-closes-worker, spares reattached / mid-turn / finalized sessions, disabled-when-grace-zero.

Testing

• New unit tests pass; tests/test_tui_gateway_server.py + tests/tui_gateway/ (305) + tests/gateway/test_ws_auth_retry.py + tests/hermes_cli/test_web_server.py (234) all green.
• Real-code-path reproduction: drove tui_gateway.ws.handle_ws through 6 simulated refreshes against the real in-process gateway (stubbed _make_agent so the real _SlashWorker subprocess still spawns).
  • Pre-fix (grace=0): 6 refreshes → 6 leaked workers, none reaped.
  • With fix (grace=3): only 1 worker alive at any time during the refreshes, 0 remaining after the grace window.
  • Separate test confirms a session that rebinds a live transport within the grace window is not reaped.
• Manual browser verification: ran hermes dashboard from this branch, refreshed /chat repeatedly while watching pgrep -f tui_gateway.slash_worker — count bumps to 2 briefly after a refresh and settles back to 1 within the grace window; no accumulation. Confirmed working.

Review note

Touches tui_gateway/, so this needs @teknium1 review (outside the Docker lane).

Fixes the slash-worker accumulation introduced as fallout from #38591.

[github-actions]

🔎 Lint report: fix/dashboard-slash-worker-leak vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9850 on HEAD, 9850 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 5107 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.