Skip to content

fix(server): keep interrupted threads resumable after restarts - #10421

Merged
maria-rcks merged 2 commits into
pingdotgg:mainfrom
maria-rcks:t3code/fix-thread-resume-restarts
Sep 7, 2026
Merged

fix(server): keep interrupted threads resumable after restarts#10421
maria-rcks merged 2 commits into
pingdotgg:mainfrom
maria-rcks:t3code/fix-thread-resume-restarts

Conversation

@maria-rcks

@maria-rcks maria-rcks commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

restart and update recovery could mark active threads failed when the provider directory held an older turn id; codex resume could also fail while decoding historical errors from another provider version. recovery now uses orchestration's active turn, and codex validates only the resume metadata it consumes. excludeTurns trims the response while preserving the model's saved context; the request interface no longer needs a generic method or test cast.

verified 204 scoped tests, server typecheck, lint, and formatting on the final change. both persisted restart/update regressions fail on the original guards and pass here.

additional end-to-end checks with real codex:

  • desktop-mode restart during a blocked shell command recalled a key and number from an earlier turn, wrote their exact value, read it separately, and finished ready.
  • web-mode update recovery used the real preparation hook with automatic continuation disabled; the marked thread recalled its earlier key and number, retained its provider thread id, and cleared its update marker.
  • a second interruption of that same remote thread using sigkill recovered the original values again and finished ready.
  • an unmarked thread with continuation disabled stayed interrupted; an explicitly stopped thread stayed stopped; a completed thread kept its original timestamp and message count across subsequent restarts.

verification limits: native electron quit/relaunch and the update package installer were not exercised. the web client intermittently retained "syncing messages" after reload and completion; sending and completing the follow-up worked, but the indicator's cause remains unverified and is outside this server fix.

desktop restart preserves the earlier verification key and number

remote update preserves the earlier verification key and number

model: gpt-6-astra. harness: codex.

@cursor

cursor Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Sep 6, 2026
@macroscopeapp

macroscopeapp Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Would Approve

Macroscope's review found this PR approvable — This is a localized server bug fix that preserves interrupted provider work across restarts and makes Codex resume tolerant of unrelated historical payloads. Existing continuation remains gated by the prior opt-in or update-marking flows, with focused tests covering the changed recovery cases.

Not approved because:

  • Monthly spending limit reached (workspace setting). Approvability relies on correctness review in order to determine eligibility

Review your spending limits in Billing settings. You can add or adjust custom eligibility rules. Learn more.

@cursor

cursor Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@maria-rcks
maria-rcks merged commit 1abc717 into pingdotgg:main Sep 7, 2026
24 checks passed
github-actions Bot added a commit to omarcresp/t3code-flake that referenced this pull request Sep 7, 2026
## What's Changed
* fix(clients): show feedback results in composer banners by @juliusmarminge in pingdotgg/t3code#10398
* fix(server): stop Windows terminal polling from spiking CPU by @UtkarshUsername in pingdotgg/t3code#9476
* fix(web): onboarding installs agents without needing Node or npm by @t3dotgg in pingdotgg/t3code#10402
* fix(server): allow settling threads with unanswered async questions by @t3dotgg in pingdotgg/t3code#10400
* feat(ci): ship stable releases from the latest nightly commit by @t3dotgg in pingdotgg/t3code#10410
* feat(marketing): add a nightly channel to the download page by @t3dotgg in pingdotgg/t3code#10408
* fix(web): keep settings inputs focused during IME composition by @Lucenx9 in pingdotgg/t3code#10262
* fix(server): preserve Codex reset credits during usage updates by @yashranaway in pingdotgg/t3code#10308
* docs: link the repository security reporting policy by @yashranaway in pingdotgg/t3code#10303
* fix(web): only show auto balance errors after failed checks by @maria-rcks in pingdotgg/t3code#10407
* fix(web): improve preview recording frame delivery by @maria-rcks in pingdotgg/t3code#10403
* fix(server): preserve inline provider secrets on redacted saves by @maxwellyoung in pingdotgg/t3code#10054
* fix(web, mobile): replace Apple desktop machine labels by @extoci in pingdotgg/t3code#10396
* fix(web): hide browser when the right panel starts closing by @Neel2107 in pingdotgg/t3code#10385
* fix(web): keep settings section headings description-free by @maria-rcks in pingdotgg/t3code#10415
* fix(usage): read and redeem hub reset credits through CLIProxyAPI by @juliusmarminge in pingdotgg/t3code#10395
* fix(web): deduplicate expanded tool labels and keep errors expandable by @Yash-Singh1 in pingdotgg/t3code#10420
* fix(server): skip git status scans while the index is locked by @Gigioxx in pingdotgg/t3code#9845
* fix(mcp): allow text-only preview snapshots by @juliusmarminge in pingdotgg/t3code#10232
* fix(claude): name the expired login or usage limit instead of a generic API error by @vitalyiegorov in pingdotgg/t3code#10321
* feat(mobile): queue a message while its attachment is still uploading by @juliusmarminge in pingdotgg/t3code#10404
* feat(mobile): show when an existing thread has a message waiting in the outbox by @juliusmarminge in pingdotgg/t3code#10405
* fix(codex): accept misalignment policy errors on thread resume by @realbakari in pingdotgg/t3code#10373
* fix(server): skip disabled settlement lookups by @t3dotgg in pingdotgg/t3code#10424
* fix(server): run OpenCode CLI commands sequentially by @t3dotgg in pingdotgg/t3code#10427
* feat(web): name the drop action while dragging sidebar threads by @SunkenInTime in pingdotgg/t3code#10378
* perf(web): keep the sidebar responsive during bulk thread updates by @t3dotgg in pingdotgg/t3code#10413
* fix(web): onboarding wizard now supports light mode by @t3dotgg in pingdotgg/t3code#10432
* feat(threads): dismiss async questions without replying by @t3dotgg in pingdotgg/t3code#10431
* fix(web): stop collapsing the composer when it loses focus by @t3dotgg in pingdotgg/t3code#10437
* fix(server): keep interrupted threads resumable after restarts by @maria-rcks in pingdotgg/t3code#10421

## New Contributors
* @Neel2107 made their first contribution in pingdotgg/t3code#10385
* @realbakari made their first contribution in pingdotgg/t3code#10373

**Full Changelog**: pingdotgg/t3code@v0.0.39-nightly.20260906.1316...v0.0.39-nightly.20260907.1325

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.39-nightly.20260907.1325
patroza added a commit to patroza/t3code that referenced this pull request Sep 7, 2026
## Problem

After a deploy/restart, `reconcileProviderSessions` resumes interrupted
threads by preparing the orchestration session as `starting` with
`activeTurnId: null`, then calling `sendTurn`. It previously left the
session in that starting state until the provider emitted
`turn.started`.

Codex resume (`thread/resume`) often does not emit `turn.started` until
the replacement turn finishes. Meanwhile `thread.queue.steer` refuses
with “still starting a turn”, so queued messages and Send now stay
blocked for the whole recovered turn.

This showed up after the #457/#458 upgrade: upstream `pingdotgg#10421` made
restart resume fire without requiring the directory `activeTurnId` to
match the projection turn, so more in-flight threads recover after a
service restart. The stuck-in-starting gap was already in fork recovery;
the update just made resume kick in.

## Fix

After a successful recovery `sendTurn`, dispatch `thread.session.set`
with `status: "running"` and `activeTurnId: result.turnId`. Prepare
still uses starting/null (short window). Failed `sendTurn` still settles
error.

## Test

`serverRuntimeStartup.reconcile.test.ts` now asserts both the prepare
`starting` sets and the post-`sendTurn` `running` sets with the admitted
turn ids.

---------

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
aorwall added a commit to aorwall/t3code that referenced this pull request Sep 7, 2026
Merges 141 upstream commits (`b438447f6..8b2838e`) into the fork,
following the
`fork-upstream-merge` skill. Landed as a merge commit; conflicts
resolved by the
path-policy verdicts in `docs/fork/inventory.json`.

## Merge shape

563 files landed (`git diff --stat HEAD^1 HEAD`) against 563 in the
upstream
range, so no upstream work was dropped. Fork delta 720 files.

## Conflicts

14 files, resolved by concern. The merge commit message names each. Two
are worth
carrying forward:

- Upstream extracted the project action rows into
`ProjectActionsList.tsx`. The
fork's Edit gate now rides an `editable` prop that defaults to
upstream's
  always-editable behavior, so the gate survives the next extraction.
- Upstream moved the `agent-browser-access` setting onto its new
`/settings/projects` page. `settingsSearch.ts` points that item there
and drops
  a fork delta.

## Two things a clean merge did not show

**Upstream can delete an export the fork still imports.** pingdotgg#10225 removed
`ClientTracingLive` as unused. `apps/web/src/lib/runtime.ts` still
installs it,
and neither file conflicted, so the typecheck failed four ways on a
merge git
called clean. Restored with a `// Fork:` comment naming the consumer.

**A green test step can hide a suite that never finished.** `vp run -r
test`
kills the packages still running when one of them fails. A
`@t3tools/desktop`
failure truncated `apps/web` and `@t3tools/mobile` after each had
reported
hundreds of passing files, and `verify.mjs` counted any package with
labeled
output as tested. Four failing web tests went unreported. The check now
keys on
the closing `Test Files` line and runs every unfinished package alone.

## Unsupported methods

`unsupported-methods.mjs` reports 0 ADD, 0 DROP, 2 KEEP, against 61
dispatched
backend methods and 131 contract methods.

Getting there took a fix. The backend moved its dispatch from
`crates/t3code/src/lib.rs` to `crates/t3code/src/rpc/dispatch.rs`, where
every
arm is a one-line call into a handler below the match. The script read
the old
path and reported zero dispatched methods, then read the new one and
called
`vcs.switchRef` a DROP, because the `unsupported_exit` that refuses it
had moved
out of the arm. It now tries both paths and follows an arm two calls
deep.

Contract changes: `provider.consumeResetCredit` and
`server.getHostResources`
gained `UnsupportedMethodError`; `server.getUsageSummary` lost it, which
closes
the item the previous merge left open.

## Feature classification

**Usable as-is** — client-only, nothing new on the wire. Sidebar drag
across
sections with destination cues and a named drop action (pingdotgg#9731, pingdotgg#9750,
pingdotgg#10378,
pingdotgg#10453, pingdotgg#10464). Composer behavior: a multiline draft survives timeline
scrolling (pingdotgg#10444), the composer stops collapsing on blur (pingdotgg#10437) and
regains
focus when you tab back (pingdotgg#10463). Panel and preview chrome: resize the
floating
preview from any edge (pingdotgg#10467), toolbar controls stay anchored (pingdotgg#10478),
the
stuck resize cursor clears (pingdotgg#10461), the browser hides as the right
panel closes
(pingdotgg#10385), manual panel choices hold during a turn (pingdotgg#10113). Settings and
accessibility polish (pingdotgg#10177, pingdotgg#10262, pingdotgg#10415, pingdotgg#10258, pingdotgg#10124, pingdotgg#10125,
pingdotgg#10127,
pingdotgg#10128, pingdotgg#10175). Performance (pingdotgg#10413, pingdotgg#10190, pingdotgg#10118). Plus the GitHub
mark on
`github.com` links (pingdotgg#10324), the Tux icon for WSL (pingdotgg#8511), a remembered
usage
page selection (pingdotgg#10189), project settings in the legacy sidebar menu
(pingdotgg#10021),
and text-only preview snapshots (pingdotgg#10232).

**Unsupported in Moatless** — resolves to a refusal, or falls through to
its own
empty state. Each is recorded in `docs/fork/gaps.md`:

- Reset credits through the hub and CLIProxyAPI (pingdotgg#10462, pingdotgg#10395,
pingdotgg#10308).
`provider.consumeResetCredit`, new union entry. `UsageLimits.tsx`
catches the
  refusal and shows "Could not use the reset credit."
- Balancing new threads across connected machines (pingdotgg#9895, pingdotgg#10433,
pingdotgg#10407).
`server.getHostResources`, new union entry. Nothing polls until a user
picks
automatic routing, and the composer then reads "Auto balance
unavailable."
- Onboarding: import grouped by repository (pingdotgg#10493), the shared
multi-computer
  wizard (pingdotgg#10465), agent install without Node or npm (pingdotgg#10402). All ride
  `agentSessions.scan` and `.import`, an existing gap.
- Shared project defaults and scoped overrides (pingdotgg#9754). The page reads,
and every
  write goes through `server.updateSettings`, which the backend does not
  dispatch.
- Two new `orchestration.dispatchCommand` types: `thread.active.reorder`
(pingdotgg#9729)
and `thread.user-input.dismiss` (pingdotgg#10431). Both are ordinary controls, a
sidebar
drag and a Dismiss button, and a dispatched command cannot be refused
per type.
That is the standing _A command cannot be refused_ gap, now 26 members
wide.

**Backend behavior to consider reproducing in Moatless** — upstream
server fixes
whose behavior the fork's client assumes:

- Invalid script IDs no longer crash threads (pingdotgg#10019). The fork ships
project
  scripts, so this one is worth reading first.
- Settlement: settle inactive threads without a PR lookup (pingdotgg#10103), skip
disabled
settlement lookups (pingdotgg#10424), settle threads with unanswered async
questions
  (pingdotgg#10400).
- Interrupted threads stay resumable after a restart (pingdotgg#10421).
- Completed requests stay closed across clients (pingdotgg#10123).
- Placeholder branches are followed after a checkout updates (pingdotgg#10441).
- A thread's PR links without an open client (pingdotgg#10101), and checkpoints
are
  captured before a PR status refresh (pingdotgg#10347).
- Adapters declare their own context compaction (pingdotgg#10112).
- Transcripts with oversized tool records import (pingdotgg#10430), and git
status scans
  are skipped while the index is locked (pingdotgg#9845).
- Usage limits pool per provider across accounts and environments
(pingdotgg#10300). The
client renders what `server.getUsageSummary` returns, so this shows
something
  only if the Moatless payload carries per-account limits.

Mobile, marketing, desktop, provider adapters and release tooling are
not this
fork's surface and are not classified.

## Also fixed here, and not upstream's doing

- Three `browser-*` search items still routed to
`/settings/integrations`, which
the fork owns for its Moatless administration page. A non-administrator
who
  searched for them was redirected away from the result.
- `moatless/listSearch.ts` carried no fork-only declaration.
- `pnpm fmt:check` failed on 294 files, 293 of them orval output. The
generator
  now formats what it writes through an `afterAllFilesWrite` hook.
- `@t3tools/moatless-api` exported `./generated`, a barrel that is never
checked
  in.

## Verification

`inventory-check.mjs` clean. `verify.mjs` green on seven checks:
duplicate-adds,
tripwires, resolution-check, unsupported-methods, `fmt:check` (3876
files),
`lint` and `typecheck`.

`test` is red on one package, and it is the machine.
`@t3tools/desktop`'s
`bundled libsecret helper` shells out to `pkg-config` for `libsecret-1`,
which
this sandbox does not have; it fails the same way when retried alone.
Everything
else passes: `apps/web` 369 files, `t3` 291, `@t3tools/mobile` 149,
`t3code-relay` 27, `@t3tools/client-runtime` 71, plus the smaller
packages.

`spec:check` cannot run in a sandbox: it needs a sibling `moatless`
checkout or a
deployment URL and has neither.

Written by Claude Opus 5 in Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---
Moatless task:
https://moatless.soaplabstest.com/tasks/8d348ecf-f9cb-4e65-b96b-6c1054a8ed33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant