Skip to content

fix(tui_gateway): stop dead compute-host Stop from leaving the session forever queued - #71825

Open
fangliquanflq wants to merge 5 commits into
NousResearch:mainfrom
fangliquanflq:fix/tui-compute-host-interrupt-stuck-running
Open

fix(tui_gateway): stop dead compute-host Stop from leaving the session forever queued#71825
fangliquanflq wants to merge 5 commits into
NousResearch:mainfrom
fangliquanflq:fix/tui-compute-host-interrupt-stuck-running

Conversation

@fangliquanflq

@fangliquanflq fangliquanflq commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

When turn isolation routes a session through the compute-host child and that host is already gone, session.interrupt used to return 5019 without clearing running or latching cancel. Later prompt.submit calls only queued forever until a backend restart. This mirrors the in-process dead-thread busy recovery for that failure path, with guards so a still-pending or successor turn is not clobbered.

Bug Cause

The compute-host branch of session.interrupt early-returned on HostSupervisor.interrupt failure before latching _turn_cancel_requested or clearing session["running"]. With no live host, turn.end never arrives to clear busy, so _handle_busy_submit kept queueing new prompts.

Reproduction Steps

  1. Enable dashboard.turn_isolation and leave a session with _compute_host_active=True and running=True.
  2. Make HostSupervisor.interrupt raise (host not running / respawn disabled).
  3. Call session.interrupt, then prompt.submit.

Expected: Stop recovers local busy state so a later prompt can start a new turn.
Before fix: interrupt returned 5019 with running still true and cancel unset; subsequent submits only queued.

Fix

On compute-host interrupt failure, latch cancel, clear queued_prompt, and force-clear busy only when safe:

  • no pending host completion for that sid (HostSupervisor.has_pending_turn)
  • inflight_turn is still the same object observed before interrupt()

Otherwise leave running for the crash waiter / successor. Added regression tests for clear, pending, post-teardown, and replaced-inflight cases.

Related Issue

No issue

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Changes Made

  • tui_gateway/server.py - recover stuck busy on compute-host interrupt failure with pending/inflight guards
  • tui_gateway/host_supervisor.py - add has_pending_turn(sid)
  • tests/test_tui_gateway_server.py - regression coverage for the four interrupt-failure cases

How to Test

  1. Manual: with turn isolation on, kill/desync the compute host mid-turn, press Stop, then send another message - it should start a new turn instead of queueing forever.
  2. Automated (already run locally):
scripts/run_tests.sh tests/test_tui_gateway_server.py -k "compute_host_interrupt or compute_host_turn or prompt_submit_dispatches_to_compute or prompt_submit_fails_open" -q

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature
  • I've run scripts/run_tests.sh on relevant tests and they pass
  • I've added tests for my changes
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation - N/A
  • I've updated cli-config.yaml.example if I added/changed config keys - N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows - N/A
  • I've considered cross-platform impact - N/A (Python session state only)
  • I've updated tool descriptions/schemas if I changed tool behavior - N/A

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tui Terminal UI (ui-tui/ + tui_gateway/) sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Jul 26, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #68146, but not a duplicate: this patch repairs the compute-host interrupt-failure path, while #68146 addressed the missing settled event after stale-interrupt recovery.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the dead compute-host Stop path; the failure premise remains valid on current main.

Problems

  • The production hunk now targets a moved handler. Commit f67ca22 moved the session RPC bodies from tui_gateway/server.py into tui_gateway/methods_session.py; current session.interrupt still has the early return _err(...5019...) at tui_gateway/methods_session.py:2719. The added recovery in the old server.py location therefore needs to be transplanted to the active handler.

Suggested changes

  • Apply the guarded recovery to tui_gateway/methods_session.py:2705 and retain the HostSupervisor.has_pending_turn() guard plus the regression coverage. The current failure branch is still reachable because HostSupervisor.interrupt() calls start() and _send_frame() (tui_gateway/host_supervisor.py:269-271).

Automated hermes-sweeper review.

Comment thread tui_gateway/server.py Outdated
@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users area/sessions Session lifecycle, resume, persistence, history labels Jul 30, 2026
@fangliquanflq

fangliquanflq commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the transplant in 4987d75 -- the prior sweeper note is addressed.

I re-checked the live path: session.interrupt in tui_gateway/methods_session.py no longer early-returns 5019 on HostSupervisor.interrupt failure. It latches cancel, clears queued_prompt, and force-clears busy only when has_pending_turn is false and inflight_turn is still the pre-interrupt object. Fail-closed on probe errors is correct. The four regression cases plus the has_pending_turn unit test cover the important branches.

When the host is gone, interrupt used to return 5019 without clearing
running or latching cancel, so later prompt.submit only queued forever.
…ing turn

Interrupt failure with a still-registered completion must leave running
set so the crash waiter can tear down without racing a successor submit.
…interrupt

Sample has_pending_turn only while holding history_lock, fail closed on
lookup errors, and force-clear busy only when the dead turn's inflight
snapshot is still present so a drain-owned successor is not clobbered.
…nflight

Force-clear busy only when inflight_turn is still the same object observed
before interrupt(), so a successor submit that replaced inflight is not
clobbered while pending registration is still in flight.
@fangliquanflq
fangliquanflq force-pushed the fix/tui-compute-host-interrupt-stuck-running branch from 331dbe6 to b294423 Compare August 2, 2026 08:42
…sion

Rebase dropped the merge transplant; keep recovery on the active
session.interrupt path after handlers moved out of server.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/tui Terminal UI (ui-tui/ + tui_gateway/) P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants