Repository navigation
fix(server): account policies no longer freeze the server reading the shell - #338
Conversation
… shell On bkt3 (2026-10-21) the server logged 98 event-loop stalls in 24 h once the Claude accounts hit their limits. ForkProviderSessions.listSessions read the full shell (active and archived) on the event loop and then one record read per thread, about 2.2 s, and maybeRebalance called it at least twice at every turn end of a thread on a full account before checking for room. - listSessions reads only the threads live sessions serve, from a new in-memory ProviderSessionManager.listLive (fork-marked, optional). - maybeRebalance checks for room elsewhere before any session read. - A released Claude session also settles its process-bound subagents when a replacement session already serves the thread (the hard-limit path places the replacement before the old release finishes; 9 cards stayed running). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-up: a session manager without listLive (a wrapper or a Layer.mock) silently brought back the blocking full-shell path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fork changes compared with upstream (2026-10-21)
None of the 81 newer upstream commits changes |
|
Merged as bk-agent-01, pinned to the approved head
Next: I check stagebkt3. The |
On bkt3 productionPR #341 (full
The trace from 02:53 to 03:42 UTC has no SQL statement slower than 300 ms and no main-thread gap longer than about 1 s. The trace keeps too little history to explain the 23:40 burst. The probe endpoint has no span, so the cause of the slower probe replies is not known yet. Most of this window was night time with less load, so a 24-hour check runs at 18:40 UTC today. |
24-hour check on bkt3 (bkmain
|
| Measurement | 24 h before #338 | 24 h after #338 |
|---|---|---|
| "event loop stalled" warnings | 98 (12–22 per hour under full accounts) | 16 (at most 3 per hour), with 65 hard-limit events today |
| Longest freezes | up to 8.5 s | 7.4 s, 7.0 s, 6.0 s (one burst at 23:40), then 4.6 s (startup), 4.3 s, 3.4 s |
list-sessions-full-shell-fallback |
— | 0 |
release-subagents-cancelled / -failed |
6 / 0 | 5 / 0 |
restart-deferred for background work / wait expired |
— | 2 / 1 |
| Subagent cards stuck "running" (60+ min) | 9 | 0 (3 open, all recent) |
| 5-minute reply probe (18:40–18:45) | p90 611 ms, p99 1,071 ms, 12.5% > 500 ms | p50 2 ms, p90 47 ms, p99 1,510 ms, max 4,904 ms, 3.5% > 500 ms |
New cause of the remaining bursts. The trace holds the 18:37–18:41 UTC burst (logged freezes of 3.4 s, 2.5 s and 3.3 s). There, the main-thread gaps match one-thread reads from the fork's agent MCP toolkit (apps/server/src/mcp/toolkits/agent/handlers.ts) on the main SQLite connection:
agentToolkit.loadPendingDetail→ProjectionStore.readCanonicalProjection: 2,504 ms at 18:40:51.agentToolkit.goalOf→ProjectionStore.readTimelineIndex: 2,169 ms at 18:40:59, and 1.8–1.9 s at 18:41:08.
These reads serve agents' T3 MCP calls (each freeze is followed by a personal MCP request proxied line), for example the Manager's 15-minute runs. This code is fork-only, so upstream will not change it. The causes that #338 fixed (listSessions full-shell reads and the replacement-session gap) did not appear.
Problem
The 24-hour check after PR #335 failed (evidence).
hard-limitevents, 31place-none). The main-thread gaps in the trace match full shell reads of 2.1–2.2 s with no parent span (for example, at 14:04:57.7, 14:09:27.4 and 14:13:27.4 UTC). The 5-minute probe gave p90 611 ms, p99 1,071 ms and 12.5% of replies slower than 0.5 s.ForkProviderSessions.listSessionsread the full shell (active and archived, about 1,800 threads) on the event loop, and then it ran one record read per thread.ClaudeAccountsService.maybeRebalancecalled it at least twice at each turn end of a thread on a full or not-placeable account: once for the live check, and once throughthreadIsIdle. It did this before it checked whether another account had room.running. At 12:42:48 UTC, the hard-limit path stopped the Claude session, and at 12:42:49 it placed a new session for the same thread, before the old release finished at 12:42:53.settleReleasedSessionSubagents(fix(server): subagents no longer stay Running after an account restart stops their session #334) skips every thread that a live session serves, so it settled nothing.Fix
listSessionsreads only the threads that live sessions serve. A new fork-marked, optionalProviderSessionManager.listLivelists the live sessions from memory. Each attached thread then costs one small record read. The old full-shell path stays only as a fallback whenlistLiveis missing.maybeRebalanceruns the in-memory room checks (pruneExhausted,disallowedFor,pendingCounts,roomElsewhere) before any session read. When all accounts are full, a turn end costs no session read.providerThreadId = null, started at or before the release) when a replacement session already serves the thread. A Claude subagent lives only in the process that started it. A SendMessage resume re-stampsstartedAt, so the replacement's own work is not touched. Other drivers keep the old skip.No new event types, schema changes or migrations. The fork-marker check passes.
How I tested
This host runs no tests (the dev-server compute policy). CI runs these on the head:
ForkProviderSessions.expbkt3.test.ts:listSessionskeeps the same field mapping (including the legacy"claude"driver), lists archived threads, and skips deleted and missing threads. It never callsgetShellSnapshot.ClaudeAccountsService.test.ts: a turn end on a full, not-placeable account with no room elsewhere makes zerolistSessionscalls. When another account has room, the thread moves as before.ProviderSessionManager.releasedSubagents.expbkt3.test.ts:After the merge into
stage, I check stagebkt3. Thebkmainpromotion waits until after 2026-10-22 09:00 IST, because PR #337 is under test.Review
A fresh-context review of
c57071a1cefound no P0 or P1 defects. CI on that head was green for the typecheck, all server test shards and the fork-marker check. I fixed one P2 finding in the next commit:listSessionsnow logsfork-provider-sessions.list-sessions-full-shell-fallbackwhen it falls back to the full-shell read. These stay open as follow-ups:cancelleduntil the subagent's next event.Risks
listSessionsno longer callssessions.getfor every session. The old calls reset each session's idle timer, so sessions now idle-release on their normal schedule.cancelleduntil its next event.🤖 Generated with Claude Code