Skip to content

fix(tui_gateway): isolated turns no longer fence themselves out of their own session lease (#101416, salvage #103737) - #113974

Merged
teknium1 merged 5 commits into
mainfrom
fix/sno-isolated-lease
Sep 17, 2026
Merged

teknium1 merged 5 commits into
mainfrom
fix/sno-isolated-lease

Conversation

@teknium1

@teknium1 teknium1 commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

With dashboard.turn_isolation: true, a new Desktop session's first message no longer fails with "This chat is open in another Hermes window/terminal" — the compute-host child now runs under the parent's lease, and that lease stays singular through compression rotation and through session.close.

Fixes the compute-host variant of #101416 (the child fenced out by its own parent's registry entry). VIPKaiser's cross-process route-split variant in that thread is a different bug and is handled by another lane, so this PR does not close the issue.

Changes

  • Salvage of fix(tui_gateway): stop isolated turns fencing out the parent's session lease #103737 (@jakobbjelver, authorship preserved): the parent vouches on the turn.start frame and the child installs an inert ActiveSessionLease(enabled=False) before _run_prompt_submit, so _admit_prompt_turn sees the slot as held upstream instead of re-claiming from the child pid (session_lifecycle._install_borrowed_lease, compute_host.ComputeHost._run_real_turn).
  • Review blocker 2 (@andrexibiza) — rotation A→B split authority: the borrow is no longer released=True (a released token made transfer_active_session return False and the fallback claimed a REAL lease for B under the child pid). The vouch is qualified — {lease_id, session_id} — and the child only borrows when it names the admitted stored id; the parent re-anchors its real lease A→B where it adopts the child's rotated session_key (compute_host_bridge._compute_host_adopt_frame_meta, on turn.end and compress acks). A stale A lease never vouches for B (compute_host_bridge._active_session_lease_vouch).
  • Review blocker 1 — close released the lease while the child was still writing: session_lifecycle._teardown_popped_session now interrupts the isolated turn, waits the same close grace, and if the child has not settled moves the real lease out of finalize's reach; the correlated turn.end/turn.error (child death fires it via _fail_pending_turns) releases it (_settle_isolated_turn_before_close, _release_deferred_active_session_lease). Deferred leases stay live authority for the orphan sweep (_own_live_lease_ids).
  • Tests trimmed from 383 lines / 10 tests to 4 invariants (one per fix + the fail-closed negative control); 3 red on origin/main, 4 green here.

Root cause

The parent claims the registry lease in prompt.submit before routing, but the child's freshly built session record had no lease, so it re-claimed from its own pid and _is_same_writer (same pid AND same live id) refused it.

Live evidence (real compute-host child, temp HERMES_HOME, HERMES_ISO_CERTIFY_SYNTH_TURN=1)

Probe origin/main @ bbaf7af this head
First isolated turn child: Refused active session stored-A: already held by pid=<parent> surface=desktop → client gets This chat is open in another Hermes window/terminal… on every attempt message.complete '[synthetic heavy turn] …'; registry = exactly one entry {pid: <parent>, session_id: stored-A} across 2 turns
A→B rotation inside the child (B2) (pick-only) registry = [{parent, stored-A}, {child, stored-B}]; replacement child ran under B on the stale A vouch registry = [{parent, stored-B}] after rotation, after a second turn, and after a child restart; parent lease.session_id=stored-B
session.close while child turn live (B1) close returned in 0.5s, registry [], foreign try_acquire_active_session succeeded mid-turn close returned after the 5s grace with holding lease for stored-A until the child settles; foreign acquire refused (SESSION_NOT_OWNED); after turn.end registry [] and foreign acquire succeeds
Control: turn_isolation: false in-process turn completes, one parent lease unchanged

Suite: scripts/run_tests.sh tests/tui_gateway tests/hermes_cli → 14020 passed, 5 failed — test_dashboard_auth_gate (4, also red on bare origin/main on this host) and one test_profiles_sidebar_cache / test_auth_commands timing flake under 40 workers that passes in isolation; none touch this PR's files.

Not covered

  • An interrupt frame does not stop an isolated synthetic turn in the child (session["running"] reads False in the child mid-turn, so _interrupt_session_turn no-ops there) — pre-existing on origin/main, orthogonal to lease ownership; the deferred-release path is what guarantees ownership here.
  • Mid-turn compression rotates agent.session_id before the post-turn _sync_session_key_after_compress; the window until turn.end is the same one the in-process path has today.

Credits

Infographic

sno-isolated-lease

Independent review (pre-merge) — dispositions

  • Attribution check red: jakobbjelver@gmail.com unmapped → mapping file added (1c60aab). The inherited Closes #101416 trailer was reworded to a Related reference (tree byte-identical) so this PR does not close the issue's route-split half (Desktop: pooled-profile chats no longer land on the local primary and get refused SESSION_NOT_OWNED after reload (#101416 class) #113956).
  • Known limitation (kept): a compute-host child that never settles after session.close holds the deferred lease until the parent exits (no supervisor stall-kill exists today); a same-window re-open during that window is refused with the "open in another Hermes window" copy. Bounded second grace + force-terminate is a follow-up, not folded in here.
  • Edge (same as main today): if the parent-side A→B transfer fails for a liveness-tracked lease, session_key advances while the lease stays on A; the next turn has no vouch and takes the legacy self-claim path (fail-closed refusal, never a silent second writer).
  • Reviewer verified: SIGKILL of the child → _fail_pending_turns → _on_compute_host_turn_done → lease released (registry empty, foreign acquire succeeds); no double release on the ws-orphan path; 806 lifecycle/orphan tests + 71 sibling tests green on head.

…n lease

With dashboard.turn_isolation enabled, every NEW desktop session routes its
turns through the compute-host child process. The parent claims the session's
active-session lease in prompt.submit, but the child's freshly built session
record carried no lease — so _admit_prompt_turn re-claimed from the child's
pid and was fenced out by the parent's own registry entry (_is_same_writer
requires the same pid AND the same live_session_id). Result: 'Session ...
already has a live owner (desktop, pid N, running 0m)' on the first message
of every new session, plus one unreclaimable lease leaked per attempt (the
owner pid is the immortal dashboard process, so _prune_dead never reclaims
it).

Fix: the parent vouches on the turn frame (parent_owns_active_session_lease,
derived from the session's actual lease state) and the child installs an
inert borrow — ActiveSessionLease(enabled=False, released=True) — before the
turn pipeline runs. Admission sees the slot as held upstream; the child can
never release or transfer the parent's slot; _is_same_writer and the
fail-closed refusal are untouched. Without the vouch (parent predates the
field) the child keeps the legacy self-claim path.

Fixes the compute-host (turn_isolation) variant of #101416; the cross-process
route-split variant in that thread is a separate Desktop bug. Related #101416.
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

૮ >ﻌ< ა ci review

ran on 1c60aab — chore: map contributor email for @jakobbjelver

debug info

CI timings

CI timings · View report · View job

Wall time 8m40s vs 46m37s (-81.4%). 5 job(s) slower, 6 faster, 1 unchanged.

  • Python tests / Run tests: +153.0s
  • Python lints / Windows footguns (blocking): -52.0s
  • OS-specific tests / Windows-only tests: +20.0s
  • OS-specific tests / macOS-only tests: +11.0s
  • Python lints / ruff enforcement (blocking): -8.0s

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tui Terminal UI (ui-tui/ + tui_gateway/) sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Sep 17, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Related: #103737 is the earlier parent-vouch/borrowed-lease attempt for #101416. This maintainer salvage retains that direction while adding qualified lease/session vouching, rotation re-anchoring, and close-settlement handling.

teknium1 and others added 4 commits September 17, 2026 01:25
… compression rotation

The borrowed token from #103737 was released=True, so when the child compressed
and rotated the stored id A->B, transfer_active_session() returned False and
_transfer_active_session_slot fell through to a REAL registry claim under the
child pid: parent owned A, child owned B (split authority). A bare-boolean vouch
then let a replacement child borrow B on the strength of the stale A lease.

- The borrow is enabled=False but NOT released: release() is a no-op and
  transfer only retargets the token locally, so the child never writes the
  registry.
- The parent vouches with {lease_id, session_id} and the child installs the
  borrow only when that session_id is the admitted stored id.
- The parent re-anchors its real lease A->B where it already adopts the child's
  rotated session_key (_compute_host_adopt_frame_meta: turn.end and compress
  acks), so authority stays singular and keyed on the live continuation.

Owner-side re-anchor + qualified admission identity follow the design in
Bergmann89's #101501.

Co-authored-by: Bergmann89 <info@bergmann89.de>
Co-authored-by: Jakob Bjelvér <jakobbjelver@gmail.com>
…solated turn settles

_teardown_popped_session only waited for session["_run_thread"]; an isolated
turn runs in the compute-host child, so session.close reached
_finalize_session and released the parent's REAL lease while the child was
still writing — a second backend could acquire the stored session mid-turn
(the double-writer state #99719 closed).

Close now interrupts the child turn and waits the same grace; if the turn has
not settled, the real lease is moved out of the session (finalize releases
nothing) and released by the turn.end/turn.error completion callback — which
child death also fires via _fail_pending_turns. Deferred leases stay live
authority for the orphan sweep (_own_live_lease_ids). The RPC close stays
bounded; ownership ends with the child's last write.

Deferred canonical-lease lifetime follows Bergmann89's #101501.

Co-authored-by: Bergmann89 <info@bergmann89.de>
Replaces the 383-line suite from #103737 with four invariants: the real
child turn path admits under the parent's lease (registry unchanged) and still
fails closed on a vouch for another stored id; a child-side A->B rotation
never claims and the parent re-anchors its real lease; session.close keeps the
lease refusing a distinct writer until the isolated turn settles.
@teknium1
teknium1 force-pushed the fix/sno-isolated-lease branch from 9bb904f to 1c60aab Compare September 17, 2026 08:25
@teknium1
teknium1 merged commit c993ba1 into main Sep 17, 2026
59 of 61 checks passed
@teknium1
teknium1 deleted the fix/sno-isolated-lease branch September 17, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tui Terminal UI (ui-tui/ + tui_gateway/) P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants