Skip to content

fix(tui-gateway): WS disconnect/reconnect TOCTOU fix + real regression test (RAH-05 + RAH-06) - #77212

Closed
JoaoMarcos44 wants to merge 2 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/rah05-06-ws-toctou-and-reattach-race
Closed

fix(tui-gateway): WS disconnect/reconnect TOCTOU fix + real regression test (RAH-05 + RAH-06)#77212
JoaoMarcos44 wants to merge 2 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/rah05-06-ws-toctou-and-reattach-race

Conversation

@JoaoMarcos44

@JoaoMarcos44 JoaoMarcos44 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #77192 (RAH-06), closes #77191 (RAH-05).

Combines the WS disconnect/reconnect TOCTOU fix (RAH-06) with its regression test fix (RAH-05) into one PR — RAH-05's test exercises exactly the revalidation logic RAH-06 introduces, so splitting them into two PRs meant the second one couldn't pass CI on its own without the first merged first. Together they tell one coherent story: a real race, a real fix, and a test that actually proves the fix works.

RAH-06 — WS disconnect/reconnect TOCTOU race

While validating a prior investigation's claim that this race was already fixed, found that the fix commit existed only on branch fix/ws-disconnect-reconnect-transport-race — never merged into main:

$ git merge-base --is-ancestor 13241a6a3 main
# not an ancestor

main's _close_sessions_for_transport() had the original bug: a session snapshot taken under _sessions_lock, released, then each session mutated (closed or re-pointed to the detached sentinel) without re-validating transport ownership — racing against session.resume's warm-reuse rebind of session["transport"] under _session_resume_lock. A reconnect landing in that window could be silently undone: the disconnecting transport's teardown either force-closed the just-reattached session, or stomped its new transport back to the detached sentinel, making a live reconnect look orphaned and eligible for grace-reap.

Fix: cherry-picked the existing, already-authored fix (13241a6a3) — applies cleanly onto identical surrounding code. Revalidates transport ownership under the same _session_resume_lock_sessions_lock ordering already used by the orphan-reap timer, immediately before claiming (close) or repointing (detach) each session. Slow teardown work still runs after releasing both locks.

RAH-05 — the existing regression test didn't exercise the race

The pre-existing regression test for this exact fix
(test_close_sessions_for_transport_skips_session_reattached_mid_teardown)
started both test sessions already pointing at new_transport. Since
_close_sessions_for_transport() filters owned_sids by the old
transport, those sessions never entered the snapshot in the first place
— the revalidation-under-lock logic RAH-06 adds was never reached. The
test passed identically before and after RAH-06's fix and proved
nothing about it.

Fix: rewrote the test to start the session on old_transport (so
the snapshot captures it) and inject the reattach — via a thin wrapper
around the real _session_resume_lock — strictly between the snapshot
and the per-sid claim, matching the actual race window.

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#00f0ff', 'mainBkg': '#0a0a16', 'primaryTextColor': '#ffffff', 'primaryBorderColor': '#ff007f', 'lineColor': '#00f0ff'}}}%%
graph TD
    A[🔌 WS disconnect] -->|snapshot owned sids| B[⚡ _sessions_lock]
    C[🔁 session.resume reattach] -->|rebind transport| D[⚡ _session_resume_lock]
    B --> D
    D -->|revalidate ownership per sid| E{Still old transport?}
    E -->|yes| F[🚀 Claim: close or detach]
    E -->|no: reattached| G[✅ Skip — live reconnect preserved]

    H[🧪 Old test: sessions start on new_transport] -->|never in owned_sids| I[🚫 Revalidation branch never runs]
    J[🧪 New test: session starts on old_transport] -->|_RaceLock reattaches mid-claim| D
Loading

Infographic :

infographic

Test plan

  • tests/test_tui_gateway_server.py -k close_sessions_for_transport — 2 passed
  • New test confirmed to fail against the pre-RAH-06 implementation (reaped == 1, not 0)
  • python -m py_compile tui_gateway/server.py tests/test_tui_gateway_server.py

Supersedes #77205 and #77206 (closed — split into separate PRs originally, recombined here since one depended on the other).

JoaoMarcos44 and others added 2 commits August 2, 2026 21:42
…d session.resume reattach

_close_sessions_for_transport() snapshotted sessions owned by the
disconnecting transport under _sessions_lock, released the lock, then
mutated each session (close or repoint to the detached sentinel)
without re-checking ownership. session.resume's warm-reuse path
(_reuse_live_payload -> _live_session_payload) rebinds
session["transport"] under _session_resume_lock independently, so a
reconnect landing in that window got silently undone: the old
transport's teardown either force-closed the just-reattached session
or stomped its new transport back to _detached_ws_transport, making a
live reconnect look orphaned and eligible for grace-reap.

Revalidate transport ownership under the same resume_lock ->
sessions_lock ordering already used by the orphan-reap timer,
immediately before claiming (close) or repointing (detach) each
session. Slow teardown work still runs after releasing both locks.

Fixes #HPA-01
…H-05)

The existing regression test for the WS disconnect/reconnect TOCTOU fix
started both sessions already on new_transport, so they never entered
owned_sids (filtered by old_transport) and the revalidation-under-lock
logic the fix added was never exercised. The test passed identically
before and after the fix and proved nothing about it.

Rewrite it to start the session on old_transport (so the snapshot
captures it) and inject the reattach strictly between the snapshot and
the per-sid claim under _session_resume_lock, via a thin wrapper around
the real lock that performs the reattach on first acquire — modeling
session.resume winning the lock race before teardown's revalidation
runs. Confirmed this fails against the pre-fix implementation
(reaped == 1, not 0) and passes against the fix in this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/tui Terminal UI (ui-tui/ + tui_gateway/) P2 Medium — degraded but workaround exists needs-decision Awaiting maintainer decision before any implementation sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages area/sessions Session lifecycle, resume, persistence, history labels Aug 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #77129: it carries the same core repair, while this PR adds the real regression interleaving identified in #77191. Please choose or consolidate the implementation and coverage.

@JoaoMarcos44

Copy link
Copy Markdown
Contributor Author

Duplicate root cause with #77129 (opened independently for the same TOCTOU bug, tracked as #77127 there). Consolidated: pushed this PR's real regression-race test on top of #77129's branch (fix/ws-disconnect-reconnect-transport-race), replacing its shallow version. #77129 now carries the fix + the real interleaving test, and closes #77127, #77191, and #77192. Closing this one in favor of #77129.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/tui Terminal UI (ui-tui/ + tui_gateway/) needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

2 participants