Skip to content

fix(server): subagents no longer stay Running after an account restart stops their session - #334

Merged
bk-agent-01 merged 3 commits into
stagefrom
fix/bkt3-stuck-subagents-on-session-stop
Oct 20, 2026
Merged

bk-agent-01 merged 3 commits into
stagefrom
fix/bkt3-stuck-subagents-on-session-stop

Conversation

@bk-agent-01

@bk-agent-01 bk-agent-01 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Claude subagents sometimes show "Working" for more than an hour, and their parent session shows running subagents. The subagents do no work in that time.

On 2026-10-20, "Review skills/docs PR 1135" showed "Working for 1h 24m". Its transcript ends with a model request at 09:21:22 UTC and has nothing more until 10:45. The bkt3 log shows the cause:

09:21:15 claude.account.rebalance {threadId: 963b0316…, from: 'sam', to: 'default', reason: 'not-placeable'}
09:21:23 orchestration V2 provider event ingestion failed … Provider session released: manual_shutdown
09:21:25 claude.account.session-stopped {threadId: 963b0316…, why: 'rebalance'}
  1. The parent turn ended at 09:20:36, but its two background subagents (Agent tool) still ran in the same Claude process.
  2. ClaudeAccountsService.threadIsIdle checks only the parent turn, so the account service stopped the process to move the session to another account. This ended both subagents in the middle of their work.
  3. ProviderSessionManager.writeReleaseRecords records only the session stop. The subagent rows, the parent's subagent turn items and the child's root node stayed running, so the child showed "Working" and the parent showed "2 Working".
  4. The subagents started again only when the user sent the parent "update ?".

The same thing happens on every account-service stop path: rebalance, mode-changed-after-turn (it stopped this fix's own session at 10:52:33 while its background code reader ran) and hard-limit. From 2026-10-13 to 2026-10-20, the account service stopped 124 Claude sessions (80 hard-limit, 44 rebalance). 29 of these stops hit a session with open subagent rows. At 10:47 UTC, 9 subagent rows in 6 live sessions were stale running.

PR #301 does not cover this: it handles a subagent end that the live Claude process reports. Here the process is gone, and the server does not read Claude's "background agents didn't finish before the previous session ended" notice on the next start.

Fix

B. A released session settles the subagents it leaves open. writeReleaseRecords (one marked block in ProviderSessionManager.ts) calls the new releasedSessionSubagents.expbkt3.ts.

  • It marks open provider_native subagents of the released driver and instance, started at or before the release, as cancelled. It writes existing events only: subagent.updated, turn-item.updated for the parent's card, and node.updated for the subagent node. In each child thread, it also cancels the runless root turn and its open items, and it follows nested children.
  • It skips threads that another live session still serves (replacement sessions, hand-offs). It also skips server_shutdown, because startup recovery and its restart wake handle that case. A plain detach settles nothing.
  • It reads the thread's subagent rows first. It does the full recovery read only when one of them is open. On production, a cold recovery read of a 10,734-item thread took 2.6 s on the event loop, so an idle release must not pay for it.
  • It never fails the release. On error, it logs orchestration-v2.provider-session-release-subagents-failed, and startup recovery stays the fallback.
  • A later SendMessage resume still reopens a settled subagent, because its new events overwrite the status.

A. Optional restarts wait for background work.

  • ForkProviderSessions.hasLiveBackgroundWork asks each live session of the thread for the adapter's hasPendingBackgroundWork. A failed check counts as "work pending".
  • threadIsIdle (rebalance) now also requires no live background work, so a rebalance skips that thread.
  • Mode-change restarts and completePendingRestart defer while work runs. The 15 s status poll (refreshStatus → recheckBackgroundWaits) checks them again, because a subagent end gives no turn event. After BACKGROUND_WORK_RESTART_CAP_MS (60 min), the restart runs anyway, and fix B cleans up.
  • Access-revoked restarts do not wait. Hard-limit stops and rotation do not change, so fix B covers them.

No new event types, schema changes or migrations. The fork-marker check passes.

How I tested

This host runs no tests (the dev-server compute policy). CI runs the checks below on head ffa176f511. After the merge, I check the change on stagebkt3.

  • New ProviderSessionManager.releasedSubagents.expbkt3.test.ts:
    • A release cancels the subagent, its card, its node, and the child's root turn and its running command.
    • These stay untouched: a subagent started after the release, one on another session, an app_owned one, another driver, and a finished one.
    • A subagent on a provider thread that the recovery read did not load stays untouched (Codex shared sessions).
    • An idle release with no open subagent skips the full recovery read.
    • A detach that leaves the shared session open settles nothing. A thread that another live session serves is skipped. server_shutdown is skipped. A failed read does not fail the release.
  • ClaudeAccountsService.test.ts:
    • Rebalance skips while a subagent runs, and it moves the thread after the work ends.
    • A deferred mode change restarts on the poll after the work ends.
    • The 60-minute cap restarts anyway.
    • Access revocation still stops at once while background work runs.
  • Before the fix: production evidence above (journal, transcripts, and projection rows read with a read-only connection).
  • After the merge: on stagebkt3, I look for claude.account.restart-deferred {waitingFor: 'background-work'} and orchestration-v2.provider-session-release-subagents-cancelled in the log, and for subagent rows that stay running with no update.

Review

A fresh-context review of the diff found no P0 or P1 findings. Two P2 findings are fixed in ffa176f511: a subagent whose provider thread was not loaded is no longer cancelled, and a race in completePendingRestart no longer leaves a stale wait start. These P2 findings stay open as follow-ups:

  1. A user's own account pin waits for all background work (background commands too, not only subagents), for up to 60 minutes. A narrower check for user-chosen mode changes could wait only for subagents.
  2. If a replacement session for the same thread is already live when the release records are written (a rare order), the thread is skipped, and its subagents stay running until that session is released. A release after a detach (archive, hand-off) settles nothing.
  3. An open subagent row with a closed card and a finished run passes the cheap check but is not in the recovery read. Such a row is never settled, and each release of its thread pays the full read.

🤖 Generated with Claude Code

…t stops their session

The Claude account service stops a thread's provider session between turns
(rebalance, mode change, hard limit). Its idle check looked only at the
parent turn, so it also stopped sessions whose background subagents were
still working, and the session release recorded nothing for them: the
subagent rows, the parent's subagent cards and the child's root turn stayed
running until the parent got another message (84 minutes on 2026-10-20).

- A released provider session now settles the provider-native subagents it
  leaves open (cancelled), for every release reason except server shutdown,
  which startup recovery already handles. A cheap subagent-row check first
  keeps idle releases from paying for the full recovery read.
- Rebalance and mode-change restarts now wait for the session's background
  work, re-checked on the 15 s status poll and capped at 60 minutes.
  Access-revoked and hard-limit stops are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L labels Oct 20, 2026
bk-agent-01 and others added 2 commits October 20, 2026 13:06
Use Effect.catchTags (t3code/prefer-catch-tags), give the test store's
getThreadRecords the projection's subagents, and cover an idle release
with no open subagent skipping the full recovery read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tart waits

- Skip a subagent whose provider thread the recovery read did not load: it
  may still run in another shared session.
- A completePendingRestart fiber that lost the race no longer leaves a
  stale background-wait start behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added size:XL and removed size:L labels Oct 20, 2026
@bk-agent-01
bk-agent-01 merged commit 3dc90ea into stage Oct 20, 2026
25 of 27 checks passed
@bk-agent-01

Copy link
Copy Markdown
Collaborator Author

Merged as bk-agent-01, pinned to the approved head ffa176f511d0700f664dacaf4a6898b3f4c4ffce.

  • Go: Tushar answered "Merge when approved" in the session, and @tusharbhardwaj-bk approved ffa176f511 on GitHub (2026-10-20 13:23 UTC).
  • Checks on the head: all passed or were skipped. A newer run replaced the cancelled "Native fingerprint diff" run, and that run passed. "Transfer report artifact" stays queued on the Blacksmith runner, and this job never runs on this repository. The stage branch has no ruleset.

Next: I check stagebkt3. Then I open one stage → bkmain PR with #328, #330 and #334.

bk-agent-01 added a commit that referenced this pull request Oct 20, 2026
#328's shell-read reader worker is upstream's (loadShellSnapshotParts,
readerWorker) on this branch, so its stage-only copies are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01

Copy link
Copy Markdown
Collaborator Author

Production check: this change went live through promotion #335 (bkmain 806b6390cc, 2026-10-20 13:41 UTC). The post-deploy check and the 24-hour check are on #335. The latest measurement with every fix of this goal live is in this #344 comment: 0.44 freezes per hour, against 1.43 before, and 0 stuck subagents.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants