fix: recover agent sessions and reconnect clients after a server restart

[tusharbhardwaj-bk]

Restarting the server for a deploy broke every running agent session. Sessions sat on "Working" forever, automatic recovery never finished, and the only way out was right-clicking the session and reconnecting by hand. It turned out to be two independent faults.

Server — nothing recovered sessions at boot. Provider processes are children of the server, so they die with it, but nothing re-derived their state on startup: the projection kept status: running with a non-null activeTurnId, so the thread rendered as working indefinitely. Recovery was lazy — it only ran when the next operation happened to route to that thread — and the stuck state defended itself: the reaper skips threads with an active turn, settle is rejected while running, and Stop respawned the agent only to no-op silently on a session that had no turn. Even a clean shutdown left this behind, because the reactors are torn down before the adapters emit their exit events, so nobody is listening.

A boot-time sweep now marks those sessions interrupted, clears the stuck turn, releases approval and user-input requests whose in-process callbacks died with the server, and restarts the interrupted turn. The restart prompt restates the original request and tells the agent to check for partially completed work before redoing anything — the turn may have already edited files before it died. The agent resumes from its persisted resume cursor, so a deploy no longer costs a session.

Also fixed on the way: Stop settles a thread whose session is gone instead of respawning it, session start is bounded at 60s so one wedged child cannot hang the shared provider command worker, bindings are reconciled at boot so a SIGKILL is survivable, and shutdown stops adapters concurrently so it fits the launcher's 5s kill deadline.

Client — a half-open socket looked healthy forever. The connection supervisor only leaves the connected phase on a close event or an explicit signal, and had no timer while connected. A restart behind a proxy or tunnel can leave a socket that never delivers a close, so the client believed it was connected indefinitely — which is exactly what the manual Reconnect button was working around. Subscriptions that lost their transport made it worse by logging and waiting for a session that was never coming.

The connected phase now probes an idle connection every 20s, reusing the same guarded probe a foreground wake already runs, so a dead transport is caught in ~30s and reconnects on the normal backoff ladder. Healthy connections are untouched. Subscriptions that lose their transport report the session instead of waiting on it, gated on it still being the live session so duplicates collapse and stale reports drop. Heartbeat failures deliberately do not skip the first backoff rung — nobody is waiting on the app, and that is what stops a flapping server becoming a reconnect storm.

Upstream merge cost

Nearly all of the new logic is in one fork-owned file, ProviderSessionRestartSweep.ts, wired into ProviderCommandReactor.start() with three lines. Nothing upstream was refactored or extracted; where the sweep needed logic that already existed in decider.ts it mirrors it locally, and a test pins the two against the real decider so the duplication cannot drift silently. The remaining edits to upstream files are single-line or few-line insertions. Touch points to re-verify on each upstream merge:

• ProviderCommandReactor.start() — sweep construction, capture, and parked run
• ProviderCommandReactor.processTurnInterruptRequested — the catchCause
• ProviderService — interrupt guard, allowRecovery: false, start timeout, boot reconcile, stopAll bounding
• CodexSessionRuntime.interruptTurn — the no-active-turn failure
• supervisor.ts — heartbeat hunk in monitorConnectedLease, notifySessionSuspect
• rpc/client.ts — the escalation call

These are upstream bugs rather than fork behaviour, so the intent is to offer both fixes upstream and delete the fork copies once they land.

Verification

• New unit suite for the sweep, including a test that runs the real decider to prove a swept thread becomes settleable again.
• New integration test: a session stranded mid-turn is swept, the turn restarts automatically on a session recovered from its resume cursor, runs to completion, and a second sweep is a no-op.
• New supervisor tests: half-open socket reconnects, healthy idle connection does not churn, heartbeat failure takes the normal first rung, and suspect reports are gated on the live session.
• Server + client-runtime typecheck clean; 213 server unit, 13 integration, 563 client-runtime tests passing locally.

Auto-restart is one exported constant (AUTO_RESTART_INTERRUPTED_TURNS) away from off. Worth knowing for the first deploy: any thread currently stuck in this state will be swept and restarted on startup.

Model: Claude Opus 5 (1M context). Harness: Claude Code in T3 Code.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	11.4 KiB	—	15.1 KiB	✅
Codex	Thread snapshot wire	—	5.7 KiB	—	7.3 KiB	✅
Codex	Live turn WebSocket wire	—	5.7 KiB	—	7.8 KiB	✅
Codex	Live turn WebSocket decoded	—	50.4 KiB	—	66.4 KiB	✅
Codex	Live turn messages	—	11	—	21	✅
Claude	Total thread wire	—	11.5 KiB	—	15.1 KiB	✅
Claude	Thread snapshot wire	—	5.7 KiB	—	7.3 KiB	✅
Claude	Live turn WebSocket wire	—	5.7 KiB	—	7.8 KiB	✅
Claude	Live turn WebSocket decoded	—	51.2 KiB	—	66.4 KiB	✅
Claude	Live turn messages	—	11	—	21	✅

Baseline: unavailable · PR result: 778d72a · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 95.6 KiB
• Claude decoded thread snapshot: 96.3 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.