fix(tui_gateway): keep Stop from stranding a session on a dead compute host or an in-flight agent build (supersedes #71825)

[briandevans]

This supersedes #71825 (same bug, current code)

#71825 reported this bug correctly, and its shape is the right one — this PR carries it over. It cannot land as filed, for a structural reason rather than a staleness one:

• Its production hunk is tui_gateway/server.py @@ -10252,14 +10252,44, but commit f67ca22 moved the session RPC bodies out of server.py into tui_gateway/methods_session.py. On main today, git show origin/main:tui_gateway/server.py | grep -c 'method("session.interrupt")' → 0. The handler it patches is not the handler that runs; server.py:10252 is now unrelated display.tool_progress focus-config code.
• That is why GitHub currently reports fix(tui_gateway): stop dead compute-host Stop from leaving the session forever queued #71825 as mergeable: CONFLICTING / mergeStateStatus: DIRTY.

Its HostSupervisor.has_pending_turn() addition still applies cleanly and is kept here essentially verbatim. Its four regression cases are ported by hand (its test hunk anchors at @@ -322 of a file that is now 15,282 lines).

What does this PR do?

session.interrupt on a turn-isolation session calls HostSupervisor.interrupt() and, on failure, returns 5019 immediately:

if _session_uses_compute_host(session):
    sid = str(params.get("session_id") or "")
    if session.get("running"):
        try:
            _get_compute_host_supervisor().interrupt(sid, request_id=f"interrupt-{rid}")
        except Exception as exc:
            return _err(rid, 5019, f"compute-host interrupt failed: {exc}")   # ← early return
    with session["history_lock"]:
        session["_turn_cancel_requested"] = True
        session["queued_prompt"] = None
    _clear_pending(sid)
    ...

HostSupervisor.interrupt() calls start() and then _send_frame(), so a host that is gone — broken pipe, respawn exhausted — makes it raise. The early return then skips every recovery step below it: _turn_cancel_requested is never latched, queued_prompt is never cleared, _clear_pending(sid) never runs, pending approvals are never denied, and session["running"] is left True.

Because the host is dead, no turn.end will ever arrive to clear busy. Every later prompt.submit therefore falls into _handle_busy_submit and queues forever. The user pressed Stop, got an error, and the session is bricked until the backend is restarted.

The repo already documents the expectation this branch fails to honour

The in-process branch of the same function, ~15 lines below, carries this exact safety net:

Safety net: if the turn's run thread is already gone but running stayed stuck (a crash/desync that skipped the run loop's finally), force-clear it so the session can't be permanently bricked at 4009 "session busy" — every send/restore/resume would otherwise reject until a full backend restart.

run_thread = session.get("_run_thread")
run_thread_alive = run_thread is not None and run_thread.is_alive()
...
if not run_thread_alive:
    with session["history_lock"]:
        if session.get("running"):
            session["running"] = False
            _clear_inflight_turn(session)

The fix is derived from that idiom. The only thing that differs is the liveness probe: not run_thread_alive becomes not supervisor.has_pending_turn(sid).

The same handler's in-process branch had a second way to strand Stop

Everything above is the turn_isolation path. turn_isolation defaults to False in hermes_cli/config_defaults.py, so the branch almost every session actually takes is the in-process one directly below it — and it stranded Stop too, for an unrelated reason.

It resolved its session through _sess:

session, err = _sess(params, rid)
if err:
    return err

_sess is _sess_building plus _wait_agent, and _wait_agent blocks up to 30 seconds on session["agent_ready"]:

def _wait_agent(session: dict, rid: str, timeout: float = 30.0) -> dict | None:
    ready = session.get("agent_ready")
    if ready is not None and not ready.wait(timeout=timeout):
        return _err(rid, 5032, "agent initialization timed out")
    err = session.get("agent_error")
    return _err(rid, 5032, err) if err else None

That is the same Event the deferred build has not set yet. Stop waited on the build it was cancelling.

This is the default path, not an edge case: session.create unconditionally seeds "agent": None alongside an unset agent_ready, and prompt.submit marks the session running and returns before the build completes.

Three consequences, all user-visible:

situation	result
build finishes inside 30s	Stop is late — and session.interrupt is not in _LONG_HANDLERS, so it runs inline on the socket reader thread and every RPC queued behind it is stalled with it.
build outlives 30s	_wait_agent returns 5032 and the handler returns before _turn_cancel_requested is ever set. _wait_agent_for_prompt then polls a flag nobody set, for up to agent.build_wait_timeout (default 600s), and goes on to call _run_prompt_submit. The turn the user cancelled runs anyway.
build already failed	_wait_agent returns 5032 from its agent_error branch even for a set agent_ready. Stop errors out before the cleanup below it, and nothing else clears running — the session is permanently busy, which is exactly the trap the compute-host half of this PR guards against, reached through the other branch.

The repo states this remedy against itself

_sess_building exists precisely for handlers in this position, and its own docstring makes the argument:

Resolve a session and warm its agent build WITHOUT waiting for it. For handlers that need the session record but not the agent. … _sess's _wait_agent was buying nothing and charging up to 30 seconds for it.

_LONG_HANDLERS' own comment names this RPC as one that must not be left waiting:

keep them off the main stdin loop so a slow portal can't stall approval.respond / session.interrupt / other RPCs.

And _wait_agent_for_prompt's docstring records that the flat 30s _wait_agent ceiling was "a message-eating cliff (#63078)" — which was fixed for prompt.submit and left in place for session.interrupt.

The commit that introduced _sess_building ("fix: attach RPCs no longer wait on the deferred agent build") converted the six attach RPCs — clipboard.paste, image.attach, image.attach_bytes, pdf.attach, file.attach, image.detach — and did not sweep the control RPCs. This is that sweep, for the one control RPC where the wait is self-defeating.

Note this is not fixed by adding session.interrupt to _LONG_HANDLERS. That would unblock the reader thread but leave the 30s delay and the dropped _turn_cancel_requested exactly as they are.

Scope

Deliberately narrow: session.interrupt's in-process branch only. Nothing in that branch needs a built agent — its single agent use, request_hard_interrupt(session["agent"]), is now guarded, because agent is None while the build is in flight becomes newly reachable once we stop waiting for it. When the agent already exists, _sess_building returns immediately and behaviour is byte-identical.

Other handlers share the "needs the session record, not the agent" shape (approval.*, rollback.list / rollback.diff, process.kill / process.list). They are not included here: they are a different concern from Stop must not leave the session stuck, and they belong in their own change.

Related Issue

No issue is filed for this — an issue search across four phrasings returned nothing. Supersedes #71825.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• tui_gateway/host_supervisor.py — add HostSupervisor.has_pending_turn(sid). _pending_turns already existed but had no accessor; the new method reads it under the same self._lock that guards every mutation of that dict (submit_turn, _complete_turn, _fail_pending_turns).
• tui_gateway/methods_session.py — session.interrupt's compute-host branch no longer returns early on interrupt failure. It records the inflight snapshot, logs the failure, falls through to the shared recovery, and force-clears the stuck busy flag under four guards (below). Stop still returns {"status": "interrupted", "turn_isolation": True} — from the user's point of view the turn is over either way.
• tests/test_tui_gateway_server.py — four regression cases ported from fix(tui_gateway): stop dead compute-host Stop from leaving the session forever queued #71825, plus one direct unit test for the new probe. Placed beside the existing compute-host cluster (test_prompt_submit_dispatches_to_compute_host_when_turn_isolation_enabled, ..._fails_open_inline_when_compute_host_dispatch_breaks, test_compute_host_turn_end_updates_metadata_mirror) and reusing its monkeypatch.setattr(server, "_get_compute_host_supervisor", ...) scaffolding.

For the in-process branch:

• tui_gateway/methods_session.py — session.interrupt's in-process branch resolves through _sess_building rather than _sess, and guards its single agent use for the now-reachable agent is None state.
• tests/test_tui_gateway_server.py — test_interrupt_before_agent_ready_prevents_late_turn_start and test_cancelled_turn_before_agent_ready_emits_error_event stop monkeypatching _wait_agent and now run against a real, never-set agent_ready, so they can actually fail; plus two new cases, test_interrupt_never_waits_on_the_deferred_agent_build and test_interrupt_recovers_a_session_whose_agent_build_failed.

The force-clear cannot clobber a live turn

All four must hold:

guard	what it rules out
host_unreachable	a successful interrupt — normal teardown still owns the turn
not pending_for_sid	the supervisor still owes this sid a completion callback; that waiter will clear busy itself
session.get("running")	already cleared between the probe and the lock
inflight_now is inflight_at_interrupt (and not None)	a successor prompt.submit / drain replaced the inflight turn and owns running now

The probe fails closed: an exception from has_pending_turn sets pending_for_sid = True, so an unreadable supervisor leaves teardown to the waiter rather than guessing.

The probe is issued before history_lock is taken. That honours the invariant stated at the other supervisor-interrupt call site: "never call a provider or compute-host method while holding history_lock: an interrupt can wait behind the very operation it is trying to cancel." The guards are then re-evaluated inside the lock, which is what closes the race the probe cannot.

Sibling sweep — resolved in both directions

Two supervisor-interrupt call sites exist in the corpus (grep -rn '\.interrupt(' tui_gateway/ hermes_cli/ apps/), plus the other two _session_uses_compute_host branches that return _err on host failure:

site	verdict
methods_session.py, session.interrupt	IN — the defect above.
server.py, _interrupt_busy_session	OUT, four reasons.
methods_session.py, session.compress	OUT — returns _err on host failure, but does not own the running busy flag, so it cannot strand the session.
methods_session.py, session.save	OUT — same: no running ownership.

_interrupt_busy_session swallows the same failure with a bare except Exception: pass, so it looks like the same bug from the other direction. It is deliberately not changed:

1. It is a best-effort interrupt in service of an already-queued prompt. Its caller runs _enqueue_prompt(session, text, transport) under history_lock and returns _ok(rid, {"status": "queued"}); teardown there is owned by _drain_queued_prompt, not by the interrupt.
2. The failure is swallowed inside a daemon thread whose finally only resets _busy_interrupt_pending. There is no rid / RPC context in that thread to report a recovery from.
3. Force-clearing running from it could clobber a live successor turn — _drain_queued_prompt claims running = True under the lock, and by the time the daemon thread runs the drain may already have started the queued turn.
4. Its caller's own comment states the invariant the site is built around: "never call a provider or compute-host method while holding history_lock." Adding state mutation there pushes work into exactly the place the file forbids it.

session.interrupt is the opposite case in every respect: an explicit user Stop, on the RPC thread, with an rid, where ending the turn is the intent.

Related work that is not superseded

#45001 (fix(desktop+gateway): make "session busy" lock recoverable — AbortSignal + server watchdog) is complementary, not overlapping. It adds a time-based stall watchdog that force-clears running + inflight_turn after session.inflight_watchdog_seconds (default 120s) of silence. It is strictly weaker on this path and does not replace this fix:

• it clears only running / inflight_turn, so _turn_cancel_requested, queued_prompt = None, _clear_pending(sid) and the approval resolution are still skipped by the dead-host early return;
• the user waits two minutes for a Stop they already pressed.

Its hunks (server.py@3648/3663/5311/5537) are pre-refactor and never touch the compute-host branch. Both changes can land independently.

How to Test

Compute-host half:

uv run --with pytest --with pytest-asyncio python3 -m pytest \
  tests/test_tui_gateway_server.py -k "compute_host_interrupt_failure or compute_host_supervisor_reports_pending" -v

In-process half:

uv run --with pytest --with pytest-asyncio python3 -m pytest \
  tests/test_tui_gateway_server.py -k "before_agent_ready or interrupt_never_waits or interrupt_recovers_a_session" -v

Red-before / green-after, measured rather than predicted. Each baseline was restored with git checkout <parent-commit> -- <path> — not git stash, which is a no-op on an already-committed file and yields a silent false pass:

case	prod hunk reverted, tests kept	with this PR
dead host clears stuck running	❌ 5019 returned, running stays True	✅
pending turn ⇒ leave running	❌	✅
post-teardown ⇒ don't clobber drain	❌	✅
replaced inflight ⇒ don't clobber successor	❌	✅
has_pending_turn probe semantics	❌ AttributeError	✅
test_interrupt_before_agent_ready_prevents_late_turn_start	❌ 5032 agent initialization timed out	✅
test_cancelled_turn_before_agent_ready_emits_error_event	❌ 5032 agent initialization timed out	✅
test_interrupt_never_waits_on_the_deferred_agent_build	❌ _wait_agent called	✅
test_interrupt_recovers_a_session_whose_agent_build_failed	❌ 5032 provider metadata fetch failed	✅

The two end-to-end interrupt cases each spend a real 30 seconds before failing against the reverted file (60.79s for the pair) and pass in under a second with the fix — the wait is the defect, so the clock is part of the evidence. The two new cases fail in 0.43s, because they assert the invariant directly instead of one of its symptoms.

test_session_not_running_before_agent_ready_emits_error_event carries the same _wait_agent stub but is deliberately left untouched: it never calls session.interrupt (it clears running directly), so the stub is unreachable in it. It stays green on both sides of the change, which makes it the control for the two that flip.

Full file: 580 passed. tests/tui_gateway/: 472 passed. ruff check clean on both touched files.

Manual reproduction, compute-host half: enable dashboard.turn_isolation, start a turn, kill the python -m tui_gateway.compute_host child, press Stop. Before: compute-host interrupt failed, and every subsequent message is silently queued forever. After: Stop succeeds, the composer is usable again, and the failure is logged.

Manual reproduction, in-process half: leave turn_isolation at its default, start a fresh session so the deferred build is still running (a cold MCP discovery makes this easy to hit), send the first message, and press Stop immediately. Before: the UI is unresponsive for up to 30 seconds, Stop reports agent initialization timed out, and the message you cancelled is then sent anyway once the build lands. After: Stop returns immediately and the turn never starts.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 15 (Darwin 25.4), Python 3.11

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Contract Protected

Invariant: a compute-host session.interrupt never leaves session["running"] set when no host completion can ever clear it.

• Known-bad input: HostSupervisor.interrupt() raises (dead child, respawn exhausted, broken pipe) while running is True — previously returned 5019 and stranded the session.
• Future-input coverage: the recovery keys off the failure of the interrupt call itself, not off any specific exception type or message, so a new failure mode inside start() / _send_frame() is covered without a code change.
• Negative cases: a live pending completion, an already-torn-down turn, and a replaced inflight turn each leave running untouched; a failed probe fails closed to the same outcome.

Second invariant (in-process branch): session.interrupt never blocks on the deferred agent build, and records _turn_cancel_requested regardless of the build's state.

• Known-bad inputs: an unset agent_ready (build in flight), which cost up to 30 seconds and then dropped the cancel flag entirely on timeout; and a set agent_ready with agent_error populated (build failed), which dropped it immediately.
• Future-input coverage: the handler no longer consults build state at all on this path, so a new build outcome cannot reintroduce the failure. test_interrupt_never_waits_on_the_deferred_agent_build asserts _wait_agent is not called, which holds independently of whatever _wait_agent itself later does.
• Negative case: when the agent is already built, _sess_building returns immediately and this path is byte-identical to before — covered by the existing in-process interrupt tests, which are unchanged and still green.

[teknium1]

Thanks for carrying the fix through the handler split. The premise is present on current main: tui_gateway/methods_session.py:2715-2719 returns 5019 if the compute-host interrupt raises, before the cancellation and pending-prompt cleanup at tui_gateway/methods_session.py:2720-2730. HostSupervisor.interrupt() can propagate failures from start() or _send_frame() (tui_gateway/host_supervisor.py:269-271), while the normal completion callback is the path that clears running (tui_gateway/server.py:1519-1545).

The proposed accessor is synchronized with the existing pending-turn mutations, and the recovery guards avoid clearing a pending or replacement turn. The added focused tests cover the dead-host, pending-completion, post-teardown, and replacement-inflight cases.

Automated hermes-sweeper review.

[Copilot]

Pull request overview

Fixes a TUI gateway turn-isolation failure mode where session.interrupt could leave a session permanently “busy” (forever queueing prompt.submit) if the compute-host child was already dead and HostSupervisor.interrupt() raised.

Changes:

• Add a HostSupervisor.has_pending_turn(sid) probe to determine whether a compute-host completion callback is still pending for a session.
• Update session.interrupt (compute-host branch) to avoid early-return on interrupt failure, and to safely force-clear session["running"] + inflight_turn only when guarded conditions indicate no completion will ever arrive.
• Add regression tests covering interrupt-failure recovery and its guarded “do not clobber” cases.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tui_gateway/methods_session.py	Ensures Stop recovers local busy state even when compute-host interrupt fails, with race-safe guards.
tui_gateway/host_supervisor.py	Adds a lock-protected accessor to check whether a session still has a pending host completion.
tests/test_tui_gateway_server.py	Adds regression coverage for dead-host interrupt recovery and negative/race cases.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.