Skip to content

fix(claude-sdk): post-result watchdog for stuck SDK iterator - #218

Merged
dylanneve1 merged 2 commits into
mainfrom
fix/sdk-post-result-watchdog
May 21, 2026
Merged

fix(claude-sdk): post-result watchdog for stuck SDK iterator#218
dylanneve1 merged 2 commits into
mainfrom
fix/sdk-post-result-watchdog

Conversation

@claudiusthebot

Copy link
Copy Markdown
Collaborator

What

Adds a short watchdog that force-closes the SDK async iterator when it ghosts after emitting result. Targeted fix for the chat-handler wedge that hung Dylan's chat for 90 min on 2026-05-19.

The wedge (talon.log, PID 2109752, chat 352042062)

14:22:54.406  dispatcher [71488224] message chat=352042062 started
14:22:54.528  agent [352042062] <- (132 chars)
…             ~30 min: 50 API calls, lots of Read/Edit/Bash
14:52:23.878  gateway send_message chat=352042062 146ms      ← final reply landed
14:52:23.979  agent PostToolBatch: terminating SDK loop on mcp__telegram-tools__end_turn
14:52:23.999  agent SDK result: sdkModel=default, contextWindow=1000000,
              contextTokens=251464, numApiCalls=50
              ⏸️ NOTHING. (no `[chatId] -> …` summary, no `Context released`)
              watchdog: No messages processed for 37/38/…/126 minutes
16:22:54.652  shutdown: Respawn requested (telegram /restart)
              "Waiting for 1 in-flight queries to drain..."

The PostToolBatch hook fired, the SDK emitted result, the handler logged it — and the for-await loop never exited. Dispatcher kept the chat in active state for 90 minutes; typing-indicator pulse (which fires while context is held) ran the entire time.

The fix

Three small changes:

1. AbortController in chat-handler optionsbuildSdkOptions(chatId, abortController?) accepts an optional controller and threads it into Options.abortController. The SDK's canonical cancellation primitive.

2. Watchdog timer inside the for-await loop — when isResult(message) fires, arm a 5-second timer (env-tunable via TALON_SDK_POST_RESULT_GRACE_MS). On a clean exit the iterator closes in milliseconds and the timer never runs — finally clears it.

3. Force-close on grace expiry — call abortController.abort() (kills the SDK subprocess) AND qi.return(undefined) (resolves the async generator with {done: true}, exits the for-await without throwing). Set a postResultForceClosed flag so the catch block falls through to post-loop accounting instead of treating it as an error. The response is already delivered and state is already populated — the only outstanding work is the summary log + dispatcher release.

Why not the alternatives

  • qi.interrupt() directly: tried historically (commit d5ce30f per the comment in handler.ts), raced with in-flight MCP tool dispatches → MCP error -32001: AbortError. PostToolBatch replaced it. Same race isn't a concern here because we only arm after result — every tool in the batch has already resolved by definition.
  • Dispatcher-side hard timeout: right Layer 3 backstop, worth a follow-up PR (different change shape, needs chat-vs-heartbeat asymmetry tuning). This PR is scoped to the proximate cause: the SDK→handler shutdown handshake.
  • Orphan subprocess sweep (PR fix(heartbeat): evict wedged SDK subprocesses, never deadlock the lock #144 style): chat has dispatcher-level cleanup that PR fix(heartbeat): evict wedged SDK subprocesses, never deadlock the lock #144's background path doesn't — the SDK subprocess gets reaped when qi falls out of scope and pipes close. Not strictly needed here. Easy to add later if real wedges still leak subprocesses.

Tests (3 new, all green)

claude-sdk-handler-watchdog.test.ts:

  1. Hang after result — synthesizes an iterator that emits system_init + result then parks forever. Handler returns within ~50ms (grace stubbed via env), abortController.signal.aborted === true, qi.return() was called, sdk.iterator_force_close_after_result counter incremented.
  2. Happy path — iterator closes naturally. No abort, no force-close, no counter increment.
  3. No result emitted — defensive: watchdog stays disarmed when the iterator closes without ever seeing result.

Full suite: 2664/2678 (2 failures are pre-existing OpenCode integration timeouts unrelated to this change).

Knobs

  • TALON_SDK_POST_RESULT_GRACE_MS (default 5000): how long to wait for the SDK to close after result before force-closing. 5s is generous — every clean turn closes within ~50ms.

What this doesn't fix

  • Pre-result hangs (model in an infinite tool loop, API request stuck mid-stream). Different failure shape, would need either a wall-clock turn budget or a no-progress watchdog on stream deltas.
  • Handler-side post-loop hangs (e.g. metrics write blocking). Dispatcher-level deadline is the right answer for that — follow-up.

@dylanneve1
dylanneve1 enabled auto-merge (squash) May 21, 2026 12:19
@dylanneve1
dylanneve1 force-pushed the fix/sdk-post-result-watchdog branch from e9093f8 to 49c5400 Compare May 21, 2026 12:19
claude and others added 2 commits May 21, 2026 15:59
The PostToolBatch hook returns `{continue: false}` after `end_turn`/`send`,
and the SDK is supposed to emit `result` and close the async iterator
immediately. In production, the SDK can emit `result` and then ghost — the
for-await loop in `handleMessage` stays parked forever, the dispatcher
context stays held, and the typing-indicator pulse keeps firing.

Observed wedge: 2026-05-19 14:52Z, chat 352042062, contextTokens=251464,
numApiCalls=50. SDK logged `SDK result: …` then never closed the iterator.
Dispatcher held the lock for 90 minutes until Dylan ran `/restart`. Talon
log confirms the post-loop accounting block (`[chatId] -> (…)` summary
and `Context released for chat …`) never ran.

Fix — minimal and targeted:

1. Thread an `AbortController` from `handleMessage` into the SDK options.
   The SDK exposes `options.abortController` as the canonical
   cancellation signal — when aborted, it tears down the spawned
   subprocess and stops streaming.

2. Arm a short watchdog timer (default 5s, env-tunable via
   `TALON_SDK_POST_RESULT_GRACE_MS`) inside the for-await loop the moment
   `isResult(message)` fires. On a clean SDK exit the iterator closes
   within milliseconds and the timer never runs — the `finally` block
   clears it.

3. On grace expiry: call `abortController.abort()` (kills the SDK
   subprocess + closes pipes) AND `qi.return(undefined)` (resolves the
   async generator without throwing, exits the for-await cleanly). Set
   a `postResultForceClosed` flag so the catch block recognises the
   abort as our own deliberate close and falls through to the post-loop
   accounting code — the response is already delivered, the result
   message is already processed, the only thing left is to log the
   summary and release the dispatcher context.

Why not just call `qi.interrupt()`: historical note in handler.ts records
that approach was tried and raced with in-flight MCP tool dispatches
(`MCP error -32001: AbortError`). PostToolBatch replaced it. Same race
isn't a concern here because we only arm after `result` — every tool in
the batch has already resolved.

Why not a dispatcher-side hard timeout: that's the right Layer 3
backstop and worth doing, but it's a different change (dispatcher.ts +
its own tests + tuning for chat-vs-heartbeat asymmetry). This PR is
scoped to the SDK→handler shutdown handshake, which is the proximate
cause of every reproducible "typing indicator stuck" wedge so far.

Tests (3 new, all passing):

- `claude-sdk-handler-watchdog.test.ts`:
  * Hang after result: handler returns within ~50ms (grace stubbed via
    env), abort signal fired, `qi.return()` called, counter incremented.
  * Happy path: iterator closes naturally, no abort, no force-close.
  * No result emitted (defensive): iterator closes naturally, no
    watchdog armed at all.

Existing test suite: 2664/2678 pass (2 pre-existing OpenCode integration
timeouts unrelated; same as main).
@dylanneve1
dylanneve1 force-pushed the fix/sdk-post-result-watchdog branch from 49c5400 to c88f924 Compare May 21, 2026 14:59
@dylanneve1
dylanneve1 merged commit a6796d1 into main May 21, 2026
37 checks passed
@dylanneve1
dylanneve1 deleted the fix/sdk-post-result-watchdog branch May 21, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants