fix(server): tool calls cut off by a lost provider stream no longer wait forever

[Byntham]

Problem

When a run's provider event stream is lost mid tool call, the run is marked failed but its open tool call stays running. t3_worktree_handoff does this every time, because it detaches the session that is running it. Because the run failed, later runs on the same provider thread inherit the item as background work, so the thread shows "Waiting on background task …" after every later run until a server restart. Because the item counts as pending work, the sidebar keeps the thread at Waiting and auto-settle skips it. Fixes #15993.

Change

RunExecutionService already tracks the ids of open background items to decide when to stop ingesting. It now keeps each item's latest update as well. When the provider's own stream fails, as when its session is released, the lost-stream fallback cancels the run's own open command_execution and dynamic_tool items in the same write that records the run as failed. This matches what restart recovery does when a provider process is gone. The cancellation is built from the adapter's item, so it gets the same unserved image stripping the ingestor applies to stored updates. A failure to store an event goes through the same fallback, but there the provider may still be running the tool, so its items are left as before. Subagent rows already end through the existing cascade.

Terminals reported by the provider are unchanged. Their process is still alive, so open items still pass to the next run through selectInheritedBackgroundTurnItems.

The cleanup is in run finalization rather than in a change to session shutdown order, because a crashed provider can't emit its own cancellations.

The fake adapter that was inline in BackgroundWorkStop.integration.test.ts moved into a scriptedAdapter helper, so the new integration test reuses it instead of copying it.

Scope and approval

• Triaged bug issue: [Bug]: Stale "Waiting on background task T3 Worktree Handoff" banner persists after a Claude worktree handoff #15993 (triage). This takes the first direction listed there, limited to runs whose provider stream was lost, so inheritance still works for live processes.
• Not covered, and still cleared by restart recovery as today:
  • A stream lost after the run's terminal, while it is still waiting on background items.
  • Items inherited from earlier runs when a later run's stream is lost.
• fix(orchestration): resume continuations after planned worktree handoffs #15203 finalizes a planned workspace detach as an interrupted terminal, which bypasses this fallback. If both land, that detached terminal should pass the same orphanedTurnItems.

Verification

• New integration test, BackgroundWorkStop.integration.test.ts › "cancels a tool call cut off when a workspace change detaches its session". It uses the real orchestrator, session manager, and run execution with a scripted adapter that has Claude's single-thread session capabilities. A dynamic tool is running when ProviderSessionManager.detach releases the session, as a workspace change does.
  • Before: the item stays running (expected 'running' to equal 'cancelled'). This matches the persisted state of the affected thread: the handoff call from a failed run was still running after 63 later runs.
  • After: the item is cancelled, and derivePendingBackgroundWork returns nothing.
• New unit tests in RunExecutionService.test.ts:
  • A lost stream cancels the open tool call and leaves a finished one alone. This fails before the change.
  • A provider-reported interrupted terminal still leaves an open command for the next run.
  • A failure to store an event, while the provider stream stays up, leaves the open tool call for the next run.
  • A cancelled tool call keeps only the image metadata of an unserved image, as normal ingestion stores it.
• vp test run on RunExecutionService, BackgroundWorkStop.integration, ProviderSessionManager, ProviderTurnStartService, ProviderFailure, SelectionRestart.integration, SteeringCompletion.integration, and testkit/OrchestratorReplayFixtures.integration: 290 passed.
• tsc --noEmit for apps/server: no errors. vp lint and vp fmt --check on the changed files: clean.
• Not checked: a live Claude session in a real client. The server logs from the original incident had rotated, so the diagnosis came from the thread's persisted projection.

Fixed by Claude Opus 5.5 in Claude Code, running inside T3 Code.

🤖 Generated with Claude Code

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Path: .coderabbit.config.ts
• Review profile: CHILL
• Plan: Advanced
• Run ID: ceae154a-6617-479c-8033-6d3cb0153e10

---

📥 Commits

Reviewing files that changed from the base of the PR and between 3cd7adf and 2f44ec9.

---

📒 Files selected for processing (3)

• apps/server/src/orchestration-v2/BackgroundWorkStop.integration.test.ts
• apps/server/src/orchestration-v2/RunExecutionService.test.ts
• apps/server/src/orchestration-v2/RunExecutionService.ts

---

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

---

📝 Walkthrough

Walkthrough

RunExecutionService now tracks active background-item snapshots. When the provider event stream fails, finalization cancels eligible current-run command and dynamic-tool items. Tests cover stream loss, other termination paths, and Claude session detachment.

Changes

Background work cleanup

Layer / File(s)	Summary
Track and cancel orphaned items apps/server/src/orchestration-v2/RunExecutionService.ts	The active-background tracker stores the latest item snapshots. If the provider event stream fails, finalization emits cancellation updates for tracked current-run command_execution and dynamic_tool items.
Validate background-item cleanup apps/server/src/orchestration-v2/BackgroundWorkStop.integration.test.ts, apps/server/src/orchestration-v2/RunExecutionService.test.ts	Tests cover cancellation after stream loss, image-data removal from canceled tool payloads, no cancellation after event-storage failure or a provider-reported interrupted terminal, and a Claude worktree handoff after session detachment. The integration test setup uses a scriptedAdapter helper.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested reviewers: juliusmarminge

---

Merge Risk

Merge Risk: ⚪ Minimal · up to 2f44e

The change addresses pending tool calls after provider-stream loss, and no actionable merge-blocking risk remains after normal checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 2f44e

The change is narrowly scoped to abandoned work and retains existing ownership checks. No expanded tool permissions or new externally exposed interface was identified. Remaining uncertainty concerns interruption timing and completion of provider teardown.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — Cancellation is bounded per execution to eligible snapshots carrying that execution's run ID. A shared session failure can notify multiple subscribers, but each execution applies its own run and active-attempt ownership gate rather than cancelling arbitrary session-wide items.

Trust Boundaries and Controls

• observed — The lost-stream fallback checks run identity, thread identity, running status, and active attempt inside the existing guarded write. Superseded or stopped executions cannot commit this cancellation batch through that gate. Adapter-provided cancellation payloads also undergo image-byte stripping before persistence.

Resilience and Maintainability Implications

• observed — Cancellation events and failed-run finalization use one event batch. The existing transaction covers the ownership check, event append, projection application, and effect enqueue, containing durable partial-write risk; publication occurs afterward.

---

---

---

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title is concise, uses the repository's conventional commit format, and clearly describes the primary fix for tool calls left open after a lost provider stream.
Description check		The description includes complete Problem, Change, Scope and approval, and Verification sections. It explains the affected behavior, implementation, linked issue, scope limits, test coverage, observed…
Linked Issues check		Issue [#15993] requires the orphaned handoff tool item to stop appearing as pending work after its provider session is released. The PR cancels run-owned dynamic_tool and command_execution items w…
Out of Scope Changes check		The scriptedAdapter extraction supports the new session-detachment integration test. The added unit tests verify stream-loss cancellation, image sanitization, and the required exception paths. The r…

---

---

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

---

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Approvability

Verdict: Approved at 2f44ec9

Macroscope's review found this PR approvable — This is a scoped server bug fix that cancels only run-owned tool and command items when their provider stream is actually lost, while preserving existing background-work behavior for live streams. The production change is localized and supported by targeted unit and integration coverage, with no schema, default, deployment, or static-analysis configuration impact.

^{You can add or adjust custom eligibility rules. Learn more.}

[maria-rcks]

Note

Written by gpt-6.1-sol on behalf of Maria

Thanks for tracing the tool calls left running after their provider stream is lost. Right now we are only taking focused bug fixes from outside contributors, and our limited review capacity means we are declining unvouched submissions over 500 changed lines. This PR changes 555 lines across run finalization and integration test restructuring. Please keep the reproduction documented in issue #15993 and request reconsideration with a smaller fix or an agreed maintainer takeover.