Skip to content

fix(server): account policies no longer freeze the server reading the shell - #338

Merged
bk-agent-01 merged 2 commits into
stagefrom
fix/bkt3-account-service-shell-reads
Oct 21, 2026
Merged

bk-agent-01 merged 2 commits into
stagefrom
fix/bkt3-account-service-shell-reads

Conversation

@bk-agent-01

@bk-agent-01 bk-agent-01 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The 24-hour check after PR #335 failed (evidence).

  1. Freezes. bkt3 logged 98 "event loop stalled" warnings in 24 h, against 37 before. There were 1–5 per hour until 10:00 UTC on 2026-10-21, then 12–22 per hour, when the Claude accounts hit their limits (38 hard-limit events, 31 place-none). The main-thread gaps in the trace match full shell reads of 2.1–2.2 s with no parent span (for example, at 14:04:57.7, 14:09:27.4 and 14:13:27.4 UTC). The 5-minute probe gave p90 611 ms, p99 1,071 ms and 12.5% of replies slower than 0.5 s.
    • ForkProviderSessions.listSessions read the full shell (active and archived, about 1,800 threads) on the event loop, and then it ran one record read per thread.
    • ClaudeAccountsService.maybeRebalance called it at least twice at each turn end of a thread on a full or not-placeable account: once for the live check, and once through threadIsIdle. It did this before it checked whether another account had room.
  2. Stuck subagents. 9 subagent cards in one session stayed running. At 12:42:48 UTC, the hard-limit path stopped the Claude session, and at 12:42:49 it placed a new session for the same thread, before the old release finished at 12:42:53. settleReleasedSessionSubagents (fix(server): subagents no longer stay Running after an account restart stops their session #334) skips every thread that a live session serves, so it settled nothing.

Fix

  1. listSessions reads only the threads that live sessions serve. A new fork-marked, optional ProviderSessionManager.listLive lists the live sessions from memory. Each attached thread then costs one small record read. The old full-shell path stays only as a fallback when listLive is missing.
  2. maybeRebalance runs the in-memory room checks (pruneExhausted, disallowedFor, pendingCounts, roomElsewhere) before any session read. When all accounts are full, a turn end costs no session read.
  3. A released Claude session also settles its process-bound subagents (providerThreadId = null, started at or before the release) when a replacement session already serves the thread. A Claude subagent lives only in the process that started it. A SendMessage resume re-stamps startedAt, so the replacement's own work is not touched. Other drivers keep the old skip.

No new event types, schema changes or migrations. The fork-marker check passes.

How I tested

This host runs no tests (the dev-server compute policy). CI runs these on the head:

  • New ForkProviderSessions.expbkt3.test.ts: listSessions keeps the same field mapping (including the legacy "claude" driver), lists archived threads, and skips deleted and missing threads. It never calls getShellSnapshot.
  • ClaudeAccountsService.test.ts: a turn end on a full, not-placeable account with no room elsewhere makes zero listSessions calls. When another account has room, the thread moves as before.
  • ProviderSessionManager.releasedSubagents.expbkt3.test.ts:
    • With a Claude replacement session live, the process-bound subagent, its card and its child are cancelled.
    • A subagent bound to a provider thread, or started after the release, stays open.
    • A Codex session still skips a thread that another session serves.

After the merge into stage, I check stagebkt3. The bkmain promotion waits until after 2026-10-22 09:00 IST, because PR #337 is under test.

Review

A fresh-context review of c57071a1ce found no P0 or P1 defects. CI on that head was green for the typecheck, all server test shards and the fork-marker check. I fixed one P2 finding in the next commit: listSessions now logs fork-provider-sessions.list-sessions-full-shell-fallback when it falls back to the full-shell read. These stay open as follow-ups:

  1. A small race: if the replacement session resumes a subagent between the settle's read and its write, the card shows cancelled until the subagent's next event.
  2. When an account has room, a turn end still lists sessions twice (now a cheap read).
  3. An older gap that this PR did not cause: a release that a detach starts (a workspace hand-off) does not settle the detached thread's subagents.

Risks

  • listSessions no longer calls sessions.get for every session. The old calls reset each session's idle timer, so sessions now idle-release on their normal schedule.
  • A thread detached from a shared session that is still live no longer appears in the list. Only attached threads appear.
  • If a replacement session starts a Claude subagent before the old release starts (for example in a workspace hand-off), that subagent's card can show cancelled until its next event.

🤖 Generated with Claude Code

… shell

On bkt3 (2026-10-21) the server logged 98 event-loop stalls in 24 h once
the Claude accounts hit their limits. ForkProviderSessions.listSessions read
the full shell (active and archived) on the event loop and then one record
read per thread, about 2.2 s, and maybeRebalance called it at least twice at
every turn end of a thread on a full account before checking for room.

- listSessions reads only the threads live sessions serve, from a new
  in-memory ProviderSessionManager.listLive (fork-marked, optional).
- maybeRebalance checks for room elsewhere before any session read.
- A released Claude session also settles its process-bound subagents when a
  replacement session already serves the thread (the hard-limit path places
  the replacement before the old release finishes; 9 cards stayed running).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L labels Oct 21, 2026
Review follow-up: a session manager without listLive (a wrapper or a
Layer.mock) silently brought back the blocking full-shell path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bk-agent-01

Copy link
Copy Markdown
Collaborator Author

Fork changes compared with upstream (2026-10-21)

stage now holds upstream 163bdeaa11 (merged by #336). I compared every fork change from this work with that commit, and with the 81 newer commits on upstream main (to 680152e27b).

Fork change Upstream state Action
#328 SQLite reader worker (nodeSqliteClient.ts, Sqlite.ts) Upstream now has it. stage matches upstream byte for byte. Nothing to revert; the #336 merge took upstream's code.
#328 shell reads on the reader worker (ws.ts, http.ts) Upstream ShellStream.ts and the PR sweep use readOnly now. The #336 merge replaced my wraps. Nothing to revert. Only the fork compatibility read in ProjectionSnapshotQuery.ts keeps its wrap.
#328 ThreadLaunchService race fix (pingdotgg#18443) Upstream has it. Nothing to revert.
#325 PR sweep idle rotation Upstream still checks every unsettled thread each minute. Keep.
#325 turn-item run filter INDEXED BY Upstream still uses the OR run_id IS NULL form, which defeats the index. Keep.
#330 subagent archive cascade Upstream has none. Keep.
#334 / this PR: cancel open subagents on session release Upstream ProviderSessionManager has none. Keep.
This PR: listLive Upstream's session manager still has no list method (shutdown, open, get, isLive, close, closeInstance, release, detach). Keep.
This PR: ForkProviderSessions, ClaudeAccountsService Fork-only files. Keep.

None of the 81 newer upstream commits changes ThreadPullRequestService.ts, ProviderSessionManager.ts, ProjectionStore.ts or ProjectionSnapshotQuery.ts.

@bk-agent-01
bk-agent-01 merged commit 4d6aa74 into stage Oct 21, 2026
44 checks passed
@bk-agent-01

Copy link
Copy Markdown
Collaborator Author

Merged as bk-agent-01, pinned to the approved head 0c1f55d4f24748133373dad0b1c197bf0d6a4843. Merge commit: 4d6aa7488f.

  • Go: Tushar answered "I approved, merge now" in the session, and @tusharbhardwaj-bk approved 0c1f55d4f2 on GitHub (2026-10-21 15:35 UTC).
  • Checks on the head: all passed or were skipped. GitHub left the first "Test Server 2" check as "in progress" after its job finished green, so I reran that job, and it passed. "Transfer report artifact" never runs on this repository.

Next: I check stagebkt3. The bkmain promotion waits until after 2026-10-22 09:00 IST (PR #337 first).

@bk-agent-01

Copy link
Copy Markdown
Collaborator Author

On bkt3 production

PR #341 (full stage promotion) shipped this change to bkmain (0c8bfb3cfa). bkt3 has run it since 2026-10-21 18:31:20 UTC.

Measurement 24 h before (2026-10-20 13:41 → 2026-10-21 14:15 UTC) Since this deploy (2026-10-21 18:31 → 2026-10-22 03:42 UTC)
"event loop stalled" warnings 98 (12–22 per hour from 10:00 UTC) 7: one at startup (4.6 s), three of 2.1–2.6 s before 19:05, and three of 6.0–7.4 s close together at 23:40–23:41
list-sessions-full-shell-fallback — 0
release-subagents-cancelled / -failed 6 / 0 1 / 0
Subagent cards stuck "running" (60+ min) 9 0
3-minute reply probe p90 611 ms, p99 1,071 ms, 12.5% > 500 ms p50 2 ms, p90 325 ms, p99 801 ms, max 1,267 ms, 4.9% > 500 ms

The trace from 02:53 to 03:42 UTC has no SQL statement slower than 300 ms and no main-thread gap longer than about 1 s. The trace keeps too little history to explain the 23:40 burst. The probe endpoint has no span, so the cause of the slower probe replies is not known yet. Most of this window was night time with less load, so a 24-hour check runs at 18:40 UTC today.

@bk-agent-01

Copy link
Copy Markdown
Collaborator Author

24-hour check on bkt3 (bkmain 0c8bfb3cfa, 2026-10-21 18:31 → 2026-10-22 18:50 UTC, no restart)

Measurement 24 h before #338 24 h after #338
"event loop stalled" warnings 98 (12–22 per hour under full accounts) 16 (at most 3 per hour), with 65 hard-limit events today
Longest freezes up to 8.5 s 7.4 s, 7.0 s, 6.0 s (one burst at 23:40), then 4.6 s (startup), 4.3 s, 3.4 s
list-sessions-full-shell-fallback — 0
release-subagents-cancelled / -failed 6 / 0 5 / 0
restart-deferred for background work / wait expired — 2 / 1
Subagent cards stuck "running" (60+ min) 9 0 (3 open, all recent)
5-minute reply probe (18:40–18:45) p90 611 ms, p99 1,071 ms, 12.5% > 500 ms p50 2 ms, p90 47 ms, p99 1,510 ms, max 4,904 ms, 3.5% > 500 ms

New cause of the remaining bursts. The trace holds the 18:37–18:41 UTC burst (logged freezes of 3.4 s, 2.5 s and 3.3 s). There, the main-thread gaps match one-thread reads from the fork's agent MCP toolkit (apps/server/src/mcp/toolkits/agent/handlers.ts) on the main SQLite connection:

  • agentToolkit.loadPendingDetail → ProjectionStore.readCanonicalProjection: 2,504 ms at 18:40:51.
  • agentToolkit.goalOf → ProjectionStore.readTimelineIndex: 2,169 ms at 18:40:59, and 1.8–1.9 s at 18:41:08.

These reads serve agents' T3 MCP calls (each freeze is followed by a personal MCP request proxied line), for example the Manager's 15-minute runs. This code is fork-only, so upstream will not change it. The causes that #338 fixed (listSessions full-shell reads and the replacement-session gap) did not appear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants