feat(dashboard): isolate heavy turns in compute host

[Kyzcreig]

Related work

This is independent of #63082. That PR offloads blocking dashboard SessionDB handlers; this PR isolates GIL-heavy agent turns in a compute-host process. They touch different runtime boundaries and can be reviewed or merged in either order.

Problem

A dashboard/desktop chat turn runs the synchronous agent loop in a serving-process thread. Pure-Python work can hold the GIL long enough to stall unrelated WebSocket and REST requests in that same process. Thread-pool dispatch therefore does not provide a hard responsiveness boundary.

What changed

This ports the full fork mechanism hunk-by-hunk onto current upstream rather than cherry-picking the heavily diverged gateway file:

• adds a persistent line-JSON compute-host child that owns live AIAgent turn state
• adds a serving-process supervisor with bounded respawn and startup orphan reconciliation
• streams RPC/event deltas from the child back through the existing session transport
• routes interrupt, reload-MCP, compression, and host-owned slash mutations across the control protocol
• mirrors read-only agent/session metadata back into the serving process without constructing a second agent
• exits orphaned children when the PPID changes or becomes init
• fails open to the historical inline turn path if compute-host dispatch fails synchronously
• keeps the feature dark by default behind dashboard.turn_isolation: false
• adds tui_gateway/synthetic_turn.py, a GIL-heavy test seam that is inert unless HERMES_ISO_CERTIFY_SYNTH_TURN=1
• adds scripts/iso-certify.py, which launches an isolated scratch dashboard, drives concurrent synthetic turns, probes WebSocket session.list plus REST /api/status, validates the load, and emits a bounded PASS/FAIL/INCONCLUSIVE verdict

Safety invariants

• The child sets HERMES_COMPUTE_HOST_CHILD=1, preventing recursive isolation.
• The parent process remains authoritative for transport and request routing; the child owns live agent/history mutation only while isolation is active.
• A failed pipe send does not emit a duplicate terminal turn error before the inline fallback runs.
• The config gate defaults off at both the merged config schema and the raw runtime read site.
• The certify harness uses a temporary HERMES_HOME, loopback-only spare port, and synthetic model seam; it does not touch the live dashboard or make model calls.

Verification

Latest upstream base: fc232f8ce648645b4df96d8be3fba2dc7cfd12d5.

scripts/run_tests.sh tests/tui_gateway tests/test_tui_gateway_server.py -q
684 passed, 0 failed

ruff check <all changed Python files>
All checks passed!

Full upstream-tree AC-4 run (6 concurrent lanes, 360s, 12s synthetic turns, 500ms probe cadence):

• verdict: PASS
• load: valid; 35 completed turns across all 6 lanes
• median/min turn duration: 12.12s / 12.00s
• worst-lane minimum deltas: 222
• REST: 720 probes, p99 7.079ms, max 56.21ms
• WebSocket: 720 probes, p99 1.688ms, max 13.73ms
• combined serving p99: 7.079ms
• probes over 1s / serving stalls: 0 / 0
• lane errors / event-loop-stall log lines: 0 / 0

An immediate second 360s run measured 5.929ms serving p99 and 0 stalls, but correctly returned INCONCLUSIVE rather than PASS because one lane reported a zero minimum turn/delta sample. That result is not counted as certification; it demonstrates the harness's load-validity gate fails closed instead of producing a false green.

Short local port-validation A/B on the same checkout (2 concurrent lanes, 15s window, 3s GIL-heavy turns; this is not the full AC-4 duration):

mode	load	serving p99	stalls >1s	verdict
isolation OFF	valid, 7 turns	5656.9 ms	3	FAIL
isolation ON	valid, 6 turns	24.85 ms	0	PASS

Both runs had 0 lane errors and 0 event loop stalled log lines. The isolated run completed 30 WebSocket and 30 REST probes with 0 over-threshold probes.

Prior fork measurements (not re-measured on upstream)

These are fork-side acceptance/deployment measurements from the original rollout, included for provenance and explicitly not represented as measurements from this upstream branch:

• dark-code AC-4 p99: 13.87s
• baseline: 3.33s
• real fix pass 1: 0.41s
• real fix pass 2: 0.44s
• gate: <1s
• live before arm: 55 stalls / 24h, worst 11.09s
• live after arm: 0 stalls

Origin

This is the upstream extraction of the full mechanism required by fork PR #307, including the isolation core that fork PR #307 itself assumed had already landed there. The upstream port deliberately includes the core, delta/control protocol, PPID guard, fail-open fallback, synthetic seam, and reusable certify harness in one reviewable feature commit.

[teknium1]

Thanks for carrying the process-boundary approach through transport, controls, and a dark config gate. The current-main premise is real: tui_gateway/ws.py:388 dispatches through a thread, and tui_gateway/server.py:8547 / 9103 run AIAgent.run_conversation() in the serving process.

Problems

• tui_gateway/server.py:1168 only routes while the parent agent is absent. Current tui_gateway/server.py:5314 already schedules a deferred parent build at session creation, so normal first-prompt timing can bypass isolation; the opposite race can construct both parent and host agents.
• tui_gateway/server.py:1190 sends parent history, but 1217-1247 mirrors only metadata. After tui_gateway/host_supervisor.py:465-466 respawns a crashed host, tui_gateway/compute_host.py:433-513 seeds it from stale parent history, losing host-only context for the next turn.

Suggested changes

• Establish host ownership at session creation and suppress/cancel the parent build when isolation is enabled.
• Recover transcript state from an authoritative persisted source or mirror it atomically; add crash-then-continuation E2E coverage.

Automated hermes-sweeper review.

[teknium1]

This gate is race-dependent: session creation still schedules the deferred parent agent build (_schedule_agent_build on current main). If that build finishes before the first prompt, the session never isolates; if the prompt wins, it can leave both a parent and host agent alive. Establish ownership at session creation and suppress or cancel the parent build for isolated sessions.

[teknium1]

The parent history is never updated by the completion mirror, which only carries metadata/version. After a host crash and respawn, the replacement host is seeded from this stale frame history, so its next turn can lose all prior host-owned context. Recover from SessionDB or mirror the authoritative transcript atomically, and cover crash → respawn → continuation.

[OutThisLife]

Rebased onto current main in #65895 (your commits preserved via Co-authored-by). 624 commits stale but remarkably clean — one conflict in server.py's info dict where main added "approval_mode" and this PR reads tools/skills from the compute-host mirror; kept both (compatible — with isolation dark, mirror.get("tools") is absent → {}, same as main). Gate + tests auto-merged.

Verified: tests/tui_gateway + test_tui_gateway_server.py = 718 pass / 0 fail on the rebased tree. Kept separate from #65893 (SessionDB offload) as you intended — different runtime boundary.

Superseding here; will close once #65895 merges. Nice work on the dark-by-default gating + fail-open + certify harness — made this an easy review.

[OutThisLife]

Landed via #65895 (merged) — commits preserved with Co-authored-by. Closing as superseded. Thanks @Kyzcreig.