fix(tui_gateway): stop dead compute-host Stop from leaving the session forever queued - #71825
Open
fangliquanflq wants to merge 5 commits into
Open
Conversation
Collaborator
teknium1
reviewed
Jul 30, 2026
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for isolating the dead compute-host Stop path; the failure premise remains valid on current main.
Problems
- The production hunk now targets a moved handler. Commit f67ca22 moved the session RPC bodies from
tui_gateway/server.pyintotui_gateway/methods_session.py; currentsession.interruptstill has the earlyreturn _err(...5019...)attui_gateway/methods_session.py:2719. The added recovery in the oldserver.pylocation therefore needs to be transplanted to the active handler.
Suggested changes
- Apply the guarded recovery to
tui_gateway/methods_session.py:2705and retain theHostSupervisor.has_pending_turn()guard plus the regression coverage. The current failure branch is still reachable becauseHostSupervisor.interrupt()callsstart()and_send_frame()(tui_gateway/host_supervisor.py:269-271).
Automated hermes-sweeper review.
19 tasks
Contributor
Author
|
Thanks for the transplant in 4987d75 -- the prior sweeper note is addressed. I re-checked the live path: |
When the host is gone, interrupt used to return 5019 without clearing running or latching cancel, so later prompt.submit only queued forever.
…ing turn Interrupt failure with a still-registered completion must leave running set so the crash waiter can tear down without racing a successor submit.
…interrupt Sample has_pending_turn only while holding history_lock, fail closed on lookup errors, and force-clear busy only when the dead turn's inflight snapshot is still present so a drain-owned successor is not clobbered.
…nflight Force-clear busy only when inflight_turn is still the same object observed before interrupt(), so a successor submit that replaced inflight is not clobbered while pending registration is still in flight.
fangliquanflq
force-pushed
the
fix/tui-compute-host-interrupt-stuck-running
branch
from
August 2, 2026 08:42
331dbe6 to
b294423
Compare
…sion Rebase dropped the merge transplant; keep recovery on the active session.interrupt path after handlers moved out of server.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When turn isolation routes a session through the compute-host child and that host is already gone,
session.interruptused to return 5019 without clearingrunningor latching cancel. Laterprompt.submitcalls only queued forever until a backend restart. This mirrors the in-process dead-thread busy recovery for that failure path, with guards so a still-pending or successor turn is not clobbered.Bug Cause
The compute-host branch of
session.interruptearly-returned onHostSupervisor.interruptfailure before latching_turn_cancel_requestedor clearingsession["running"]. With no live host,turn.endnever arrives to clear busy, so_handle_busy_submitkept queueing new prompts.Reproduction Steps
dashboard.turn_isolationand leave a session with_compute_host_active=Trueandrunning=True.HostSupervisor.interruptraise (host not running / respawn disabled).session.interrupt, thenprompt.submit.Expected: Stop recovers local busy state so a later prompt can start a new turn.
Before fix: interrupt returned 5019 with
runningstill true and cancel unset; subsequent submits only queued.Fix
On compute-host interrupt failure, latch cancel, clear
queued_prompt, and force-clear busy only when safe:HostSupervisor.has_pending_turn)inflight_turnis still the same object observed beforeinterrupt()Otherwise leave
runningfor the crash waiter / successor. Added regression tests for clear, pending, post-teardown, and replaced-inflight cases.Related Issue
No issue
Type of Change
Changes Made
tui_gateway/server.py- recover stuck busy on compute-host interrupt failure with pending/inflight guardstui_gateway/host_supervisor.py- addhas_pending_turn(sid)tests/test_tui_gateway_server.py- regression coverage for the four interrupt-failure casesHow to Test
scripts/run_tests.sh tests/test_tui_gateway_server.py -k "compute_host_interrupt or compute_host_turn or prompt_submit_dispatches_to_compute or prompt_submit_fails_open" -qChecklist
Code
scripts/run_tests.shon relevant tests and they passDocumentation & Housekeeping
cli-config.yaml.exampleif I added/changed config keys - N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows - N/A