fix(gateway+codex): desktop/dashboard event-stream reliability — detached-session black hole, turn-end persistence, live codex events

[vashkartik]

Summary

Four reliability fixes for gateway-served UIs (dashboard / desktop SPA) plus a small installer override, found while running the gateway under an embedding desktop app that reloads webviews and respawns the backend frequently. Each fix is independent; together they take websocket chat from "first turn works, everything after a reconnect dies silently" to fully durable.

1. Detached-session black hole (tui_gateway/server.py)

Problem. When a websocket drops (page reload, transient disconnect, host respawn), handle_ws parks every session it owned on the module-level _stdio_transport — which in dashboard mode has no reader. Nothing ever re-binds the session. The agent finishes the turn (rollouts show task_complete with real answers), but every event — deltas, message.complete — streams into captured stdout. The client's busy state sticks forever and all later sends are silently swallowed.

Repro. Open a dashboard session, send a prompt, reload the page mid-turn, reconnect: the turn never completes client-side and subsequent sends do nothing.

Fix. Any session-scoped RPC arriving over a live WS upgrades a stdio-parked session to that transport (_rebind_ws_transport in _sess_nowait). Only stdio-parked sessions are upgraded: a second live client cannot hijack another websocket's stream, and real stdio gateways (Ink TUI, where stdio has a genuine peer) dispatch with _stdio_transport as the current transport and are left untouched.

2. Missing turn-end persistence (tui_gateway/server.py)

Problem. Turn results lived only in process memory until the next prompt.submit persisted them. Any backend restart between turns (crash, idle reap, host-driven respawn) erased the completed exchange: the session row kept its generated title but message_count stayed 0 and session.resume painted an empty transcript.

Repro. Send one prompt, wait for the reply, kill the gateway process, restart, resume the session: empty transcript despite the titled session row.

Fix. _persist_session_history writes replace_messages into the session's own state.db right after the turn's history lands (after any compression key rotation), mirroring _ensure_session_db_row's profile-home routing. Best-effort: a DB failure never kills the turn thread.

3. Silent codex app-server turns (agent/codex_runtime.py, agent/transports/codex_app_server_session.py)

Problem. Codex app-server turns were silent in every connected UI: message.start then message.complete and nothing in between — no deltas, no reasoning, no tool activity — because (1) CodexAppServerSession's on_event hook was never wired, (2) the projector drops delta notifications by design (history-only), and (3) the return dict omitted last_reasoning even though the projector stashes reasoning on every spliced assistant message.

Fix. _codex_live_event bridges notifications to the agent's existing display callbacks (the gateway re-binds them every turn): item/agentMessage/delta → _fire_stream_delta, item/reasoning/delta → _fire_reasoning_delta, item/started|completed for command/fileChange/mcp/dynamic tools → tool_start/tool_complete, mirroring the projector's names and deterministic call ids so live tool cards correlate with persisted history. run_codex_app_server_turn now returns last_reasoning so message.complete carries payload.reasoning like the default chat-completions path. The session pre-drain (approval round-trips) fires on_event too, so deltas aren't dropped while an approval is pending. Display is best-effort: nothing in the bridge can break a turn.

4. Duplicate user echo (agent/codex_runtime.py)

Problem. Codex re-emits the submitted input as its own userMessage item while run_conversation has already appended it — splicing both stored every user message twice, which showed up as doubled user bubbles in any persisted/resumed transcript.

Fix. Drop the projector's leading user echo (matched by role + exact content) before splicing.

5. Installer repo override (scripts/install.sh)

HERMES_REPO_URL / HERMES_REPO_URL_SSH let embedders and fork maintainers install from a pinned fork branch without carrying a divergent copy of the script. When only the HTTPS override is set it is used for the primary clone attempt too, so a fork-only branch never 404s against the default repo first.

Test coverage

• tests/test_tui_gateway_server.py: WS re-bind upgrade, no-hijack of a live transport, stdio gateways untouched, turn-end persist writes messages, persist survives DB failure, and the truncate test now asserts both the pre-turn rewrite and the turn-end persist.
• tests/agent/test_codex_runtime_live_events.py (new): delta → stream callback, reasoning delta → reasoning callback, tool started/completed → tool events with exit-code-tagged results, call ids match the projector's history ids, non-tool/junk items ignored, callback exceptions never escape the bridge.

282 passed, 1 deselected
(tests/test_tui_gateway_server.py, tests/agent/test_codex_runtime_live_events.py,
 tests/agent/transports/test_codex_event_projector.py)

Also verified live against a running dashboard: tool.start with the real command, tool.complete with result+duration, token-by-token message.delta stream, then message.complete; and send → kill backend → fresh process → session.resume returns the full transcript with no duplicated user rows.

🤖 Generated with Claude Code

[teknium1]

Thanks for the focused reliability work. The Codex portions remain relevant on current main: agent/codex_runtime.py:376-400 forwards only tool-progress notifications, while agent/transports/codex_event_projector.py:109-110 can still materialize a userMessage that agent/codex_runtime.py:448-449 appends after the standard user turn.

Problems

• The PR's new on_event assignment would replace current main's _on_codex_event callback, dropping its existing tool_progress_callback behavior (agent/codex_runtime.py:376-400).
• The websocket and persistence hunks are based on superseded mechanics: current disconnects use _detached_ws_transport (tui_gateway/server.py:820-824), and current finalize persistence is marker-based (tui_gateway/server.py:570-587, 1a2f3aea9).

Suggested changes

• Salvage the Codex live-event bridge by composing it with _on_codex_event, then retain the user-echo dedup with an actual projected-userMessage regression test.
• Rework the gateway portions against current transport and persistence paths instead of cherry-picking them directly.

Automated hermes-sweeper review.

[teknium1]

When salvaging onto current main, compose this bridge with the existing on_event callback rather than replacing it: current agent/codex_runtime.py:376-400 uses that callback to emit tool_progress_callback breadcrumbs for item/started events.