Skip to content

fix(server): block V2 replacements until session cleanup completes - #9806

Open
shivamhwp wants to merge 354 commits into
pingdotgg:t3code/codex-turn-mappingfrom
shivamhwp:fix/v2-session-release-timeout
Open

fix(server): block V2 replacements until session cleanup completes#9806
shivamhwp wants to merge 354 commits into
pingdotgg:t3code/codex-turn-mappingfrom
shivamhwp:fix/v2-session-release-timeout

Conversation

@shivamhwp

@shivamhwp shivamhwp commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

V2 reports a provider session as stopped after a 30-second cleanup timeout, even when its scope is still closing. Removing the live entry also lets a replacement open before cleanup finishes.

Keep the pending release until cleanup completes. Timeouts return an error and record error status instead of stopped. Close and detach retries wait for the same cleanup. New sessions and new thread attachments covered by pending cleanup stay blocked; a thread already attached to another live session can keep reusing it. Failed cleanup keeps the replacement block until the environment restarts.

Targets the Orchestration V2 branch in #2829, rebased onto 3ed8c4d. The conflict resolution preserves the base's workspace-folder validation alongside the cleanup locks. This does not change shared-session detach policy or solve the separate shared-Codex-process account handoff problem. Related #8580 and #9211 change V1 switching, not this V2 release path.

The live-session reuse review finding is fixed in 87d7bcf. Two new regression cases reproduce the prior rejection and cover pending cleanup, timeout, late success, and permanent cleanup failure. All 76 focused tests across 11 files passed, with one optional fixture-recording test skipped. Server typecheck and targeted lint/formatting passed. The earlier rebased head passed 363 tests across 20 files, including provider adapters, replay, restart recovery, and independent lifecycle probes.

CI repeatedly failed or timed out fetching Ubuntu packages over HTTP, including on fresh-runner retries. A separate commit, b0ccd56, changes the runner's Ubuntu package downloads to HTTPS without changing package sources or test commands. Full CI passed on this head: https://github.com/pingdotgg/t3code/actions/runs/33936834660.

Cursor Bugbot, Macroscope correctness, and Effect Service Conventions pass on b0ccd56. All review threads are resolved. Macroscope requires human approval because of the production lifecycle and credential-handling scope.

Screenshots

Before: no warning while cleanup is still running.

Before: cleanup times out without warning

After: the PR shows the cleanup timeout and blocks replacement sessions.

After: visible cleanup timeout warning

Recovery: the warning clears automatically after cleanup succeeds.

Recovery: cleanup completed and warning cleared

Captured in a real Chromium client connected to an isolated local server and the real Codex provider. Both the V2 base and PR head used a temporary five-second idle timer and a 120-second cleanup delay. The production 30-second cleanup timeout was unchanged.

  • Before, V2 base 3ed8c4df: the session was marked stopped after the timeout, with no warning while cleanup was still running.
  • After, PR head b0ccd56f: the session stayed in error and the app showed that replacement sessions were blocked until cleanup completed.
  • Recovery: cleanup finished, the stored error cleared, and the warning disappeared without dismissal or a page reload.

Temporary test hooks were removed after capture and are not included in this PR. Original PNGs and the captured session-state evidence are retained with the test artifacts.

Full test report.

Implemented and tested by GPT-6 using Codex in T3 Code for shivamhwp.


Note

High Risk
Changes core provider session lifecycle, MCP credential revocation timing, and open/close concurrency in Orchestration V2; failed or slow cleanup can block replacements until restart.

Overview
Orchestration V2 no longer treats a hung provider session close as successfully stopped. ProviderSessionManagerV2 keeps a releasing record after removing a session from the live map, runs scope cleanup on a background fiber, and waits up to 30 seconds; on timeout callers get a release error, projections move to error, and cleanup keeps running.

While that release is pending, open fails with ProviderSessionOpenError for the same session id or for threads tied to the releasing session (already-attached live sessions on other ids can still be reused). close/detach join the in-flight cleanup instead of starting parallel teardown; MCP credential revocation is deferred until after scope close, with guards for reservations and live holders.

CI apt steps on Blacksmith runners now rewrite mirror URLs from HTTP to HTTPS before apt-get update. Provider-switching docs describe the new cleanup and blocking rules.

Reviewed by Cursor Bugbot for commit b0ccd56. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Block V2 provider session replacements until cleanup completes

  • Adds a releasing registry in layerWithOptions that retains removed session entries, their thread IDs, and a completion Deferred until cleanup finishes, keyed by provider session and thread.
  • Reworks releaseEntry in ProviderSessionManager.ts into an uninterruptible coordinator: it removes the live entry, starts cleanup under thread and provider-session locks, waits up to 30 seconds, records a timeout error if cleanup is still running, and keeps the entry blocked on failure.
  • open now rejects replacement sessions and new thread attachments while a related cleanup is pending, but still returns an already-attached live session for the same thread.
  • detach requests targeting a session in releasing now wait on the existing cleanup instead of starting a competing release, optionally revoking the thread credential first.
  • Adds documentation in provider-switching-and-context.md covering the timeout, blocking rules, and failed-cleanup persistence.
  • Risk: cleanup entries that fail remain permanently blocked in the releasing registry; releaseEntry revokes credentials only when no live session or reservation holds the same credential — verify the isProviderSessionReleaseError classifier and credential-revocation guard do not skip a still-attached session.

Macroscope summarized b0ccd56.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Sep 4, 2026
Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts Outdated
Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts Outdated
Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts
Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts Outdated
@shivamhwp

Copy link
Copy Markdown
Collaborator Author

Note: GPT-6 on behalf of shivam (@shivamhwp).

The first CI run found failures in unchanged V2-base files: Check reports formatting in seven files, Test Server 1 expects 58 migrations but the base registers 59, and Release Smoke fails on an unused expo-notifications@57.0.15 patch. None of those files are changed by this PR. Keeping those fixes separate from session cleanup.

The review fixes are in d68d735. All 34 focused session-manager and turn-start tests pass locally, along with targeted lint and server typecheck. Waiting for the new CI and review results.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — The production session manager now coordinates asynchronous cleanup, blocks replacements and thread attachments during cleanup, changes persisted session error states, and alters MCP credential revocation timing. These cross-cutting lifecycle and credential-handling changes have substantial runtime impact beyond the accompanying tests, CI adjustment, and documentation.

You can add or adjust custom eligibility rules. Learn more.

@shivamhwp

Copy link
Copy Markdown
Collaborator Author

Note: GPT-6 on behalf of shivam (@shivamhwp).

Latest head bf93d87 passes Bugbot and Macroscope correctness. All reported review threads are resolved. Locally, 58 focused tests across six session-manager consumer test files pass, as do targeted lint and server typecheck.

CI remains blocked outside this change: seven unchanged files fail formatting, the migration test expects 58 entries instead of the registered 59, and Release Smoke rejects the unused Expo notifications patch. Earlier Test Server 3 logs also show stale replay expectations for the runtime-info prompt suffix. I reproduced the migration failure, the seven formatting failures, and the tool_call_read_only/cursor replay mismatch locally. The relevant adapter, migration, fixture, and release files match the V2 base.

Leaving these base fixes out of the cleanup PR pending maintainer direction. Nothing has been merged.

@shivamhwp
shivamhwp force-pushed the fix/v2-session-release-timeout branch from c61d233 to c86ae9d Compare September 4, 2026 23:41
Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts Outdated
Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts
juliusmarminge and others added 20 commits September 4, 2026 17:43
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
…4457)

Update every deterministic Codex turn/start expectation with the explicit user approvals reviewer emitted by the adapter, restoring exact replay coverage and unblocking the merge lane.
…ojects (pingdotgg#3640)

Restore versioned shell-cache hydration, reconcile repository enrichment across multiple environments, and decode non-null settled shell timestamps correctly.
)

Normalize subagent result disclosure across timeline rendering and collapsed/expanded states.

Co-authored-by: codex <codex@users.noreply.github.com>
…dotgg#3754)

Expose MCP tools for worktree handoff and status inspection, with focused server and contract coverage.

Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Reset snooze metadata when creating forks and provider subagent children, with focused regressions for both inheritance paths.

Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Wake replay stream waiters when outbound validation fails and ensure interrupted recordings close when no tool use arrives.

Co-authored-by: codex <codex@users.noreply.github.com>
Remove the unreferenced V1 activity payload projection that arrived from main and assert only the public serialized SchemaError shape under Effect beta.102.

Co-authored-by: codex <codex@users.noreply.github.com>
- Defer continuation offers for background completions that land while a
  root turn is active and un-finalized; finalize offers exactly one wake
  only when unhandled completed work remains
- Stop late monitor-event mutations from erasing in-turn handled marks, so
  injected-turn ack chatter is never retained as wake evidence
- Clear stale wake buffer frames at non-continuation user-turn start
- Make the late-mutation suppress logic's running/terminal branches
  mutually exclusive
This PR introduces the only wakeBuffer clear at the ACP startTurn site, and it
also clears midTurnUnreportedCompletedTaskIds there, both on the reasoning that
a user turn owns the conversation.

An orchestrator-injected wake is not a user turn. pingdotgg#4499
fix/delegated-task-parent-wake dispatches app-owned delegated-child wakes as
creationSource "server", which is non-continuation, so once both land such a
wake would discard this session's pending native wake frames and its kept
mid-turn completion marks, losing agent output and a continuation offer that
nothing else can recover.

acpIsAppOwnedWakeTurn exempts orchestrator-injected wakes from both clears,
matching what ClaudeAdapterV2 already does with its own buffer. Inert without
pingdotgg#4499, since nothing else dispatches creationSource "server". Kept as its own
commit because the concern originates in a different PR.
An app-owned delegated child wrote its result into the parent projection but
never started a parent run, so an async delegation's result sat unread until the
user's next message. Offer a provider continuation request when a delegated
child terminalizes, with a per-task completionWake policy separating async
delegations from blocking waits.

The wake must also reach the model. ProviderContinuationRequests is a
provider-native mechanism whose dispatch only triggers ingestion of output the
adapter already buffered, and both ClaudeAdapterV2 and AcpAdapterV2 discard the
message text when it is marked creationSource provider. An app-owned child
buffers nothing, so a wake marked that way settled instantly having prompted
nothing. App-owned wakes now dispatch as message_text via a shared
delegatedTaskWakeRequest helper used by both producers.

Co-authored-by: codex <codex@users.noreply.github.com>
A Claude V2 turn whose terminal SDK result carried origin.kind
"task-notification" had that result discarded, so the run stayed running
forever and every run queued behind it stalled. The drop is now limited to
results reporting zero model turns, which is what the stale debris the guard
was written for actually looks like.

ClaudeAdapterV2 dropped any task-notification-origin result unless the turn was
an agent/provider continuation. An app-owned delegated-task wake run is
agent/server, so its terminal results were discarded and finalizeActiveTurn
never ran. Any turn whose final result carries that origin hangs, not just
wakes. Observed in production against a real session: three terminal success
results with num_turns 154, 60 and 7 were all dropped, the session went silent
for 8h27m, and five queued runs behind it never dispatched.

The guard could not simply be deleted. After interrupt recovery a stale stopped
task_notification and its origin result interleave ahead of the real root
assistant stream and must not terminalize the turn or leak their text.
Zero-turn results are still dropped, so that path is unchanged: this drops
strictly fewer results than before, so no previously working path can regress.

num_turns separates the two cases cleanly. The existing stale fixture sets
numTurns 0 explicitly, makeResultFrame defaults to 1, and the production
results all carried positive turns. The predicate is num_turns === 0 rather
than < 1 or a truthiness check on purpose: num_turns is required on both result
variants in 0.3.205, but if an older emitter ever omitted it, undefined === 0
is false, so the result is accepted rather than dropped. That fails toward
terminalizing rather than toward the hang class this fixes.

Both drops and accepts are logged with num_turns, stop_reason, terminal_reason,
result uuid, session id and message provenance. The drop is debug level because
interrupt recovery produces it routinely; the accept is info, because a
positive-turn task-notification result settling a non-continuation turn is both
the hang fix working and the exact shape a stale positive-turn result would
take.

Co-authored-by: codex <codex@users.noreply.github.com>
juliusmarminge and others added 18 commits September 4, 2026 17:43
Renumbers the v2 migrations 044-052 to 045-053 behind main's
044_ClearAutomaticProjectModelDefaults, and adopts main's sticky
new-thread selection (pingdotgg#9164), local-only worktree bases in the v2 launch
path (pingdotgg#8751), the PR summary read for settlement (pingdotgg#9176), Claude per-cwd
skills (pingdotgg#9210), the provider editor redesign with the branch's dedicated
environment fields re-grafted (pingdotgg#8508), and the client half of
continue-threads-across-restart (pingdotgg#9167). The server-side continuation
markers stay unported: they live in the v1 session directory, and v2
recovery terminalizes running runs on restart, so the capability is
withheld until the v2 runtime carries them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Index event sequence lookups, recover only threads with pending work, and page projection verification and rebuilds. Bound provider event logging and omit turn histories when resuming Codex threads.

Allow delegated thread identifiers through relay routes. Add focused regression coverage and document the performance constraints.
Retain main's composer, work-log, settings, mobile and performance changes through c8f77e0 while preserving v2 runs, queued messages, provider handoffs and durable history.

Port native compaction and logout, asynchronous Codex questions, provider usage accounting, automatic settlement and PR refresh into the v2 services. Bound live event retention during replay and delivery, measure thread replay before decoding, and read checkpoint metadata without loading transcripts or patches.

Keep main migrations through 047 and move the v2 migrations to 048–058. Preserve the existing branch history and the pre-rebase backup.

Model: GPT-6. Harness: Codex.
- Preserve Codex turn identity while suppressing duplicate diff notifications
- Optimize settlement projections and isolate thread visit handling
- Add concurrency and regression coverage across server and mobile
Restore the completed work timer divider and text size from main. Keep todo-list progress in the composer and omit it from web and mobile timeline entries, including completed task lists.

Verified pending, running, and completed task projection; 187 focused web tests and 35 mobile tests pass. Web and mobile typechecks pass.
Show Send when a running thread has draft content. Separate submission follow
from first-message anchoring so later sends do not reserve extra blank space.
Restore Android initial composer insets and iOS focus-aware dictation insets.

Keep opening and final assistant replies visible around completed folds,
anchoring Worked for at the first hidden item while preserving v2 relationship
cards and execution-attempt behavior.

Validation: 107 focused tests and the mobile typecheck pass. Formatting passes;
scoped lint and React Doctor report warnings but no errors. No simulator run.
…rk (pingdotgg#4793)

Co-authored-by: Julius Marminge <julius0216@outlook.com>
Retain main's changes while preserving v2 orchestration, queue/steer controls,
composer-only tasks, timeline timers, and mobile scrolling fixes.

Port opt-in restart continuation through durable v2 effects, with shutdown
race guards, activation gating, retry deduplication, and native Codex resume.
Use narrow projection reads for control effects and runtime-request replies.
Surface Claude fallback notices without failing the turn or hiding the notice.
Report missing workspace folders before provider startup.

Carry over custom models and prices, bounded client caches and stream cleanup,
lazy image loading, persistent changed-file trees and sidebar filters, Safari
cookie import, theme fixes, POSIX file-link case, private-host favicon filtering,
native provider update paths, and platform portability updates.
Migration ids remain unchanged.

Validated scoped typechecks and focused server, web, mobile, client-runtime,
contracts, desktop, shared, SSH, script, and resource-monitor tests. Preserved
all 347 original commits and checked the final tree against both saved tips.

Model: GPT-6. Harness: Codex.
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch from f2e4a0d to 3ed8c4d Compare September 5, 2026 00:45
@shivamhwp
shivamhwp force-pushed the fix/v2-session-release-timeout branch from 8098b97 to bc10499 Compare September 5, 2026 01:05

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit bc10499. Configure here.

Comment thread apps/server/src/orchestration-v2/ProviderSessionManager.ts
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 2 times, most recently from b82facd to 2ac9bfe Compare September 5, 2026 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants