Repository navigation
fix(server): shell reads run on a SQLite reader worker and no longer freeze the server - #328
Merged
Merged
Conversation
…atabases (pingdotgg#19226) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Alex Southwell <saphid@gmail.com> (cherry picked from commit 8659fcf) Conflict resolution: ShellStream.ts keeps the fork side. This base has no loadShellSnapshotParts (upstream pingdotgg#17141, 1f31906, is not on stage), so the reader wrap for the shell snapshot routes lands in the follow-up fork commit at the routes' own transactions in http.ts and ws.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cherry-picked reader worker (upstream pingdotgg#19226) only moves reads marked NodeSqliteClient.readOnly. Upstream marks its shell snapshot routes inside loadShellSnapshotParts, which stage does not have, so this marks the fork's equivalents and the fork-only shell readers that froze bkt3: - http.orchestration.loadShellSnapshot (GET /api/orchestration/shell, the Linear bridge) and ws.orchestrationV2.loadShellSnapshot (shell subscription): the whole read transaction, enrichment after it. - ws getArchivedShellSnapshot: the whole read transaction. - ws.orchestrationV2.projectShellItems: the per-thread shell read. Events are published after their commit, so the reader already sees them. - Fork ProjectionSnapshotQuery shell() and threadShell(): only the ProjectionStore read, not the provider name lookup. Covers threadUsage.get, claudeAccounts.setThreadMode (access guard and owner lookup) and the other compatibility callers. Every marked read is a pure read of committed state. Inside an open transaction readOnly is ignored and the read stays on the writer. On the reader a deferred BEGIN does not wait on another process's write lock, where the writer's BEGIN IMMEDIATE blocks the event loop for up to busy_timeout (the 5 s setThreadMode stall). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fork marker check treats a line as upstream's only when it matches upstream main. Upstream changed the reader worker after pingdotgg#19226 (pingdotgg#19305, pingdotgg#19324, pingdotgg#19417, pingdotgg#19425, pingdotgg#19559), so the cherry-picked lines differ from upstream main even though they are upstream's code. - nodeSqliteClient.ts: BEGIN/END markers around the three cherry-picked regions, saying the next upstream merge takes upstream's side. The code stays byte-identical to 8659fcf. - ThreadPullRequestService.ts: drop the two comment lines that upstream itself removed in pingdotgg#19559, so the wrap matches upstream main exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A review of the reader-worker port found that the fork-only wraps of projectShellItems' per-event getThreadShell and ProjectionSnapshotQuery's threadShell() put frequent small reads on the single reader worker, where each waits behind multi-second full snapshots. After a restart that can fill the 1,000-event live buffers and make shell streams resubscribe. Upstream keeps these reads on the writer; only full snapshots move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ingdotgg#18443) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: SAPHID <saphid@users.noreply.github.com> (cherry picked from commit 62c74a9) Conflict resolution: ThreadLaunchService.ts keeps the fork's cleanup condition (`cancelled || !workspaceRecorded`, terminal close inside it). Stage has no `runReleased` flag, so only the rename fiber handle and its interrupt are ported. The test applies unchanged. Why now: "cancels tracked setup before provider work is released" failed on this branch because the background rename's thread.metadata.update (worktreePath: worktreeCwd) landed after the cancel's update to null. The race was already in the code; the reader-worker client adds a few effect steps per SQL statement, which moved the fiber interleaving. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bk-agent-01
marked this pull request as ready for review
October 19, 2026 19:38
tusharbhardwaj-bk
approved these changes
Oct 20, 2026
Collaborator
Author
|
Merged as bk-agent-01, pinned to the approved head
Next: I will confirm that stagebkt3 serves the merge and that its logs do not show "SQLite reader worker unavailable". This merge does not change |
This was referenced Oct 20, 2026
Merged
bk-agent-01
added a commit
that referenced
this pull request
Oct 20, 2026
#328's shell-read reader worker is upstream's (loadShellSnapshotParts, readerWorker) on this branch, so its stage-only copies are dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
bkt3 runs every SQLite statement on the Node event loop. The thread-list ("shell") query takes about 0.8 s for the 1,228 active threads with warm pages, and 4–8 s with cold pages or a held write lock. While it runs, the whole server freezes, and desktop clients show "Connecting".
Measured on bkt3 on 2026-10-19, after the sweep fix (#325/#326):
ThreadPullRequestServiceV2.synchronize, and 5.0 s in a one-thread read forclaudeAccounts.setThreadMode. The second matchesbusy_timeout = 5000: writer transactions useBEGIN IMMEDIATE, so a read on the writer waits on the event loop while another process holds the write lock.Fix
Cherry-pick of upstream
8659fcf145(fix(server): thread list reads no longer freeze the server on large databases pingdotgg/t3code#19226). It adds a reader worker topackages/shared/src/nodeSqliteClient.ts: a second connection in a worker thread. Transactions marked withNodeSqliteClient.readOnlyrun there with a deferredBEGIN, which does not wait on writers in WAL mode.Sqlite.tsturns it on, and the PR sweep's shell read uses it.ShellStream.tsconflicted, becausestagedoes not have upstream'sloadShellSnapshotParts. I kept the fork side and wrapped the fork's own read sites instead (next point).The fork's full shell reads now run on the reader:
http.ts)getArchivedShellSnapshot(ws.ts)ProjectionSnapshotQuery.shell(), which coversthreadUsage.get, the MCP control tools and other full-shell callers.One-thread reads (
getThreadShell,threadShell(), the per-eventprojectShellItemsread) stay on the writer, as upstream does. A review found that putting them on the one reader worker makes them wait behind multi-second full snapshots, which can fill the 1,000-event live buffers after a restart. Inside an open write transaction,readOnlyhas no effect and the read stays on the writer (upstream's test covers this). The reader sees every committed write.Cherry-pick of upstream
62c74a92f9(fix(server): stop the branch rename before cancelling worktree setup pingdotgg/t3code#18443, "stop the branch rename before cancelling worktree setup"). The ported client's extra Effect steps change fiber timing, and that exposed a race that the fork already had: a background branch rename could write the oldworktreePathback after a cancel cleared it. CI failed onThreadLaunchService.test.ts("cancels tracked setup before provider work is released") on both earlier heads. The cherry-pick interrupts the rename before cleanup and adds upstream's regression test.Fork markers on the cherry-picked regions, because upstream changed them again later (perf(shared): cached SQLite statements no longer hold large parameters pingdotgg/t3code#19305, perf(server): SQLite checkpoints no longer block the event loop pingdotgg/t3code#19324, fix(shared): SQLite checkpoints no longer fall back to the main thread silently pingdotgg/t3code#19417, feat(observability): SQL spans show lock wait and connection; stalls keep their own history pingdotgg/t3code#19425, refactor: prune narration and history comments pingdotgg/t3code#19559). On the next upstream merge, take upstream's side there.
If the worker cannot start, reads fall back to the writer and the server logs one warning. The worker code is an inline source string, so it needs no new bundle entry, and it works under the desktop app's
ELECTRON_RUN_AS_NODE=1.How I tested
nodeSqliteClient.test.tscomes with the cherry-pick. It covers the worker, nested transactions, andreadOnlyinside a write transaction.605efd6d1b. Low risk, upstream behaviour: if the worker does not start within 10 s, reads use the writer until the next restart (oneSQLite reader worker unavailablewarning).node scripts/check-fork-markers.tspasses (FORK_UPSTREAM_REF=upstream-ro/main).stage, I will check stagebkt3 for the reader-worker start log and errors. After the promotion to bkmain, I will check the bkt3 freeze log and the 250 ms probe.🤖 Generated with Claude Code — Claude Opus 5.5 in T3 Code.