Skip to content

fix(server): stop one wedged provider stop from stalling every queued prompt - #123

Merged
tusharbhardwaj-bk merged 1 commit into
bkmainfrom
fix/bkmain-stop-session-hol-blocking
Aug 20, 2026
Merged

tusharbhardwaj-bk merged 1 commit into
bkmainfrom
fix/bkmain-stop-session-hol-blocking

Conversation

@tusharbhardwaj-bk

@tusharbhardwaj-bk tusharbhardwaj-bk commented Aug 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

bkmain outage 2026-08-20 10:41–12:02 UTC: every new prompt stayed Queued. Root cause and fix:

  • An unanswered Claude AskUserQuestion outlived its turn. ClaudeAdapter.stopSessionInternal cancels pendingApprovals but not pendingUserInputs, then waits on Fiber.interrupt(streamFiber) with no bound; the stream fiber sits in the uninterruptible Stream.fromAsyncIterable finalizer behind the SDK's pending canUseTool promise.
  • The session reaper wedged first (holding the per-thread lifecycle lock). The owner's later Stop request wedged the ProviderCommandReactor's single sequential worker lane, and every turn-start-requested wake / session-set runDue queued behind it.
  • DurableExecutionCoordinator only learned about work through that lane → parked in Queue.take with runnable rows in the table.

Changes

  • ClaudeAdapter: resolve pending user inputs on stop; bound the stream-fiber interrupt (5s) — query.close() still ends the runtime.
  • ProviderCommandReactor: bound providerService.stopSession (15s) in processSessionStopRequested so one provider cannot hold the lane.
  • DurableExecutionCoordinator: every idle wait is bounded by a 10s safety poll; a lost wake costs one poll, not an outage.

Test plan

  • New regression test: work accepted with no wake is claimed by the safety poll (drives only the TestClock). Times out on the previous coordinator.
  • DurableExecutionCoordinator.test.ts 9/9 green; apps/server typecheck green.
  • CI (deploy-bkt3.yml) green → auto-deploy to bkmain → queue probe stays GREEN and Stop on a session with a pending question returns within seconds.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

… prompt

On 2026-08-20 every new prompt on bkmain sat in "Queued" for 80+ minutes.
A Claude AskUserQuestion outlived its turn; stopSessionInternal cancelled
pending approvals but not pending user inputs and then waited on
Fiber.interrupt(streamFiber) with no bound, while the stream fiber sat in
the uninterruptible Stream.fromAsyncIterable finalizer behind the SDK's
pending canUseTool promise. The session reaper wedged first (holding the
per-thread lifecycle lock); the owner's later Stop request then wedged the
provider command reactor's single sequential worker lane, and every
turn-start wake and session-set runDue queued behind it. The durable
coordinator only ever learned about work through that lane, so it parked
in Queue.take with runnable rows in the table.

- ClaudeAdapter.stopSessionInternal: resolve pending user inputs and bound
  the stream-fiber interrupt (5s); query.close() still ends the runtime.
- ProviderCommandReactor.processSessionStopRequested: bound
  providerService.stopSession (15s) so a stuck provider cannot hold the lane.
- DurableExecutionCoordinator: bound every idle wait with a safety poll
  (10s) so a lost wake costs one poll instead of an outage. Regression test
  accepts work with no wake and drives only the clock; it times out on the
  previous code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Aug 20, 2026
@tusharbhardwaj-bk
tusharbhardwaj-bk merged commit 3e155cd into bkmain Aug 20, 2026
13 checks passed
@tusharbhardwaj-bk
tusharbhardwaj-bk deleted the fix/bkmain-stop-session-hol-blocking branch August 20, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant