Skip to content

fix(server): restarts no longer stall on old subagent result deliveries - #19631

Merged
t3dotgg merged 1 commit into
mainfrom
t3/speed-up-fleet-recovery
Oct 14, 2026
Merged

t3dotgg merged 1 commit into
mainfrom
t3/speed-up-fleet-recovery

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Oct 14, 2026

Copy link
Copy Markdown
Member

After an "Update everything", one fleet host took 78 s to become ready. 70 s of that was delegated-task recovery at startup. Startup holds the effect worker until recovery ends, so agents could not resume during that time.

Recovery finalized every terminal run in a parent thread whose message had ever been a delegated delivery. Each call reloaded the thread's runs, messages, subagents, and turn-item positions, then returned without writing. Only the run that holds a cohort's current delivery message can settle anything. On that host, one parent had 190 past deliveries, which caused 188 serial reloads (about 66 s, no writes).

Recovery now reads only the thread's runs. It finalizes only terminal runs whose input is a cohort's current delivery message. finalizeDelegatedCompletionDelivery already ignored every other run, so delivery behavior does not change. Startup spans now also record the recovery candidate counts.

Proof

  • New test: a parent with 40 settled deliveries and 1 current one. The current delivery settles and a second recovery changes nothing. Before this change, recovery read the delivery message 41 times. Now it reads it once.
  • Local fixture with 190 past deliveries: recovery went from 320 ms and 1,347 SQL statements to 7 ms and 13 statements.
  • All 27 test files that use delegated recovery pass (847 tests).

Not in this PR: the time before the new process starts (likely the full DB backup on ext4, see #18633), and the slow restart continuations after ready, which wait on the SQLite connection behind background sweeps.

Reviewed with sol-loop: 1 round with GPT-6.1-Sol on high.

Created with Claude Opus 5.5 in Claude Code (T3 Code).

🤖 Generated with Claude Code

Recovery finalized every terminal run whose input was ever a delegated
delivery. Each call reloaded the parent thread and returned early. A parent
with 190 past deliveries held startup for about 66 s. Recovery now finalizes
only runs that consumed a cohort's current delivery message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. label Oct 14, 2026
@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 14, 2026
@github-actions github-actions Bot added the size:M 30-99 changed lines (additions + deletions). label Oct 14, 2026
@coderabbitai

coderabbitai Bot commented Oct 14, 2026

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Team
  • Run ID: 0ea0100a-3425-4a80-8d8b-fcd0f7ecfd43
📥 Commits

Reviewing files that changed from the base of the PR and between 348f217 and edbfce1.

📒 Files selected for processing: 2
  • apps/server/src/orchestration-v2/DelegatedCompletionDelivery.test.ts
  • apps/server/src/orchestration-v2/Orchestrator.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📝 Walkthrough

Recovery now matches terminal runs to current delegated-completion deliveries instead of scanning messages for historical delegated-completion payloads. Tests verify the query count and confirm a second recovery does not change run or subagent state.

Changes: Delegated completion recovery

Layer / File(s) Summary
Current delivery matching and regression coverage
apps/server/src/orchestration-v2/Orchestrator.ts, apps/server/src/orchestration-v2/DelegatedCompletionDelivery.test.ts
Recovery records candidate thread counts, selects terminal runs whose userMessageId matches a current delegated-completion delivery, and refreshes runs after finalization. The regression test checks that recovery queries the delivery-message projection once and that a second recovery leaves runs and subagents unchanged.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to edbfc

This change speeds up restart recovery by skipping old, already-settled subagent deliveries while still finalizing the current ones. No concrete merge-blocking risk was identified.

Pre-merge checks: Passed 4
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: preventing startup stalls caused by old subagent result deliveries.
Description check Passed The description clearly covers the problem, the implementation, the verification results, known limitations, and the tools used. It does not provide a separate scope-and-approval statement, but the fo…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@macroscopeapp

macroscopeapp Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at edbfce1

Macroscope's review found this PR approvable — This is a focused, well-tested server recovery fix that removes redundant historical thread reloads while preserving current delegated-delivery settlement. The only production impact is improved startup recovery performance; the companion changes are test-only.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 4.9 KiB −23 B (−0.5%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB −23 B (−1.9%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 20.9 KiB 20.8 KiB −41 B (−0.2%) 29.3 KiB ✅
Codex Live turn messages 2 1 −1 (−50.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB −31 B (−0.6%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB −31 B (−2.5%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.2 KiB 21.2 KiB −41 B (−0.2%) 29.3 KiB ✅
Claude Live turn messages 2 1 −1 (−50.0%) 8 ✅

Baseline: 348f217 · PR result: edbfce1 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@t3dotgg
t3dotgg merged commit 22af947 into main Oct 14, 2026
33 checks passed
@t3dotgg
t3dotgg deleted the t3/speed-up-fleet-recovery branch October 14, 2026 22:53
github-actions Bot added a commit to omarcresp/t3code-flake that referenced this pull request Oct 15, 2026
## What's Changed
* fix(server): only Claude progress updates before a tool call show as text by @maria-rcks in pingdotgg/t3code#18923
* fix(pr): keep known state in sync across controls by @maria-rcks in pingdotgg/t3code#18408
* fix(web): review summary only says optional when it is by @flamboh in pingdotgg/t3code#18451
* fix(web): subagent lists in the composer banner no longer start lines with a comma by @flamboh in pingdotgg/t3code#19170
* fix(web,mobile): pull request filters no longer offer the Scratch folder by @flamboh in pingdotgg/t3code#19179
* fix(web,mobile): pull request project filter follows the sidebar order by @flamboh in pingdotgg/t3code#19180
* feat(settings): inactivity auto-settle can keep threads with a PR by @tris203 in pingdotgg/t3code#14642
* fix(server): Claude works next to an enterprise managed-mcp.json by @SunkenInTime in pingdotgg/t3code#18706
* fix(web,mobile): muse code uses meta's official blue logo by @maria-rcks in pingdotgg/t3code#19562
* fix(desktop): import CA trust variables from the login shell by @maria-rcks in pingdotgg/t3code#19604
* fix(server): preview copy only forwards text from the viewer's copy keys by @maria-rcks in pingdotgg/t3code#19561
* fix(muse): Supervised threads ask before commands again by @maria-rcks in pingdotgg/t3code#19558
* fix(server): restarts no longer stall on old subagent result deliveries by @t3dotgg in pingdotgg/t3code#19631
* fix(clients): a thread comes back when its subagent needs approval by @t3dotgg in pingdotgg/t3code#19630
* fix(usage): cold usage scans no longer stall the server by @t3dotgg in pingdotgg/t3code#19446
* fix(server): "Always allow this session" stops Codex asking for every command by @t3dotgg in pingdotgg/t3code#19635
* fix(web): keep vertical cursor stops outside styled markers by @Yash-Singh1 in pingdotgg/t3code#19651
* fix(web): mermaid code toggle stays in place when switching views by @maria-rcks in pingdotgg/t3code#19601


**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261014.3118...v0.0.46-nightly.20261014.3132

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261014.3132
github-actions Bot added a commit to davidvanderklay/t3code-flake that referenced this pull request Oct 15, 2026
## What's Changed
* fix(server): only Claude progress updates before a tool call show as text by @maria-rcks in pingdotgg/t3code#18923
* fix(pr): keep known state in sync across controls by @maria-rcks in pingdotgg/t3code#18408
* fix(web): review summary only says optional when it is by @flamboh in pingdotgg/t3code#18451
* fix(web): subagent lists in the composer banner no longer start lines with a comma by @flamboh in pingdotgg/t3code#19170
* fix(web,mobile): pull request filters no longer offer the Scratch folder by @flamboh in pingdotgg/t3code#19179
* fix(web,mobile): pull request project filter follows the sidebar order by @flamboh in pingdotgg/t3code#19180
* feat(settings): inactivity auto-settle can keep threads with a PR by @tris203 in pingdotgg/t3code#14642
* fix(server): Claude works next to an enterprise managed-mcp.json by @SunkenInTime in pingdotgg/t3code#18706
* fix(web,mobile): muse code uses meta's official blue logo by @maria-rcks in pingdotgg/t3code#19562
* fix(desktop): import CA trust variables from the login shell by @maria-rcks in pingdotgg/t3code#19604
* fix(server): preview copy only forwards text from the viewer's copy keys by @maria-rcks in pingdotgg/t3code#19561
* fix(muse): Supervised threads ask before commands again by @maria-rcks in pingdotgg/t3code#19558
* fix(server): restarts no longer stall on old subagent result deliveries by @t3dotgg in pingdotgg/t3code#19631
* fix(clients): a thread comes back when its subagent needs approval by @t3dotgg in pingdotgg/t3code#19630
* fix(usage): cold usage scans no longer stall the server by @t3dotgg in pingdotgg/t3code#19446
* fix(server): "Always allow this session" stops Codex asking for every command by @t3dotgg in pingdotgg/t3code#19635
* fix(web): keep vertical cursor stops outside styled markers by @Yash-Singh1 in pingdotgg/t3code#19651
* fix(web): mermaid code toggle stays in place when switching views by @maria-rcks in pingdotgg/t3code#19601


**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261014.3118...v0.0.46-nightly.20261014.3132

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261014.3132
AdEx-Partners-DE pushed a commit to AdEx-Partners-DE/t3code that referenced this pull request Oct 15, 2026
…es (pingdotgg#19631)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 22af947)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:M 30-99 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants