fix(claude-sdk): post-result watchdog for stuck SDK iterator - #218
Merged
Conversation
dylanneve1
enabled auto-merge (squash)
May 21, 2026 12:19
dylanneve1
force-pushed
the
fix/sdk-post-result-watchdog
branch
from
May 21, 2026 12:19
e9093f8 to
49c5400
Compare
The PostToolBatch hook returns `{continue: false}` after `end_turn`/`send`,
and the SDK is supposed to emit `result` and close the async iterator
immediately. In production, the SDK can emit `result` and then ghost — the
for-await loop in `handleMessage` stays parked forever, the dispatcher
context stays held, and the typing-indicator pulse keeps firing.
Observed wedge: 2026-05-19 14:52Z, chat 352042062, contextTokens=251464,
numApiCalls=50. SDK logged `SDK result: …` then never closed the iterator.
Dispatcher held the lock for 90 minutes until Dylan ran `/restart`. Talon
log confirms the post-loop accounting block (`[chatId] -> (…)` summary
and `Context released for chat …`) never ran.
Fix — minimal and targeted:
1. Thread an `AbortController` from `handleMessage` into the SDK options.
The SDK exposes `options.abortController` as the canonical
cancellation signal — when aborted, it tears down the spawned
subprocess and stops streaming.
2. Arm a short watchdog timer (default 5s, env-tunable via
`TALON_SDK_POST_RESULT_GRACE_MS`) inside the for-await loop the moment
`isResult(message)` fires. On a clean SDK exit the iterator closes
within milliseconds and the timer never runs — the `finally` block
clears it.
3. On grace expiry: call `abortController.abort()` (kills the SDK
subprocess + closes pipes) AND `qi.return(undefined)` (resolves the
async generator without throwing, exits the for-await cleanly). Set
a `postResultForceClosed` flag so the catch block recognises the
abort as our own deliberate close and falls through to the post-loop
accounting code — the response is already delivered, the result
message is already processed, the only thing left is to log the
summary and release the dispatcher context.
Why not just call `qi.interrupt()`: historical note in handler.ts records
that approach was tried and raced with in-flight MCP tool dispatches
(`MCP error -32001: AbortError`). PostToolBatch replaced it. Same race
isn't a concern here because we only arm after `result` — every tool in
the batch has already resolved.
Why not a dispatcher-side hard timeout: that's the right Layer 3
backstop and worth doing, but it's a different change (dispatcher.ts +
its own tests + tuning for chat-vs-heartbeat asymmetry). This PR is
scoped to the SDK→handler shutdown handshake, which is the proximate
cause of every reproducible "typing indicator stuck" wedge so far.
Tests (3 new, all passing):
- `claude-sdk-handler-watchdog.test.ts`:
* Hang after result: handler returns within ~50ms (grace stubbed via
env), abort signal fired, `qi.return()` called, counter incremented.
* Happy path: iterator closes naturally, no abort, no force-close.
* No result emitted (defensive): iterator closes naturally, no
watchdog armed at all.
Existing test suite: 2664/2678 pass (2 pre-existing OpenCode integration
timeouts unrelated; same as main).
dylanneve1
force-pushed
the
fix/sdk-post-result-watchdog
branch
from
May 21, 2026 14:59
49c5400 to
c88f924
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a short watchdog that force-closes the SDK async iterator when it ghosts after emitting
result. Targeted fix for the chat-handler wedge that hung Dylan's chat for 90 min on 2026-05-19.The wedge (talon.log, PID 2109752, chat 352042062)
The PostToolBatch hook fired, the SDK emitted
result, the handler logged it — and the for-await loop never exited. Dispatcher kept the chat inactivestate for 90 minutes; typing-indicator pulse (which fires while context is held) ran the entire time.The fix
Three small changes:
1.
AbortControllerin chat-handler options —buildSdkOptions(chatId, abortController?)accepts an optional controller and threads it intoOptions.abortController. The SDK's canonical cancellation primitive.2. Watchdog timer inside the for-await loop — when
isResult(message)fires, arm a 5-second timer (env-tunable viaTALON_SDK_POST_RESULT_GRACE_MS). On a clean exit the iterator closes in milliseconds and the timer never runs —finallyclears it.3. Force-close on grace expiry — call
abortController.abort()(kills the SDK subprocess) ANDqi.return(undefined)(resolves the async generator with{done: true}, exits the for-await without throwing). Set apostResultForceClosedflag so the catch block falls through to post-loop accounting instead of treating it as an error. The response is already delivered andstateis already populated — the only outstanding work is the summary log + dispatcher release.Why not the alternatives
qi.interrupt()directly: tried historically (commitd5ce30fper the comment in handler.ts), raced with in-flight MCP tool dispatches →MCP error -32001: AbortError. PostToolBatch replaced it. Same race isn't a concern here because we only arm afterresult— every tool in the batch has already resolved by definition.qifalls out of scope and pipes close. Not strictly needed here. Easy to add later if real wedges still leak subprocesses.Tests (3 new, all green)
claude-sdk-handler-watchdog.test.ts:result— synthesizes an iterator that emitssystem_init+resultthen parks forever. Handler returns within ~50ms (grace stubbed via env),abortController.signal.aborted === true,qi.return()was called,sdk.iterator_force_close_after_resultcounter incremented.resultemitted — defensive: watchdog stays disarmed when the iterator closes without ever seeingresult.Full suite: 2664/2678 (2 failures are pre-existing OpenCode integration timeouts unrelated to this change).
Knobs
TALON_SDK_POST_RESULT_GRACE_MS(default5000): how long to wait for the SDK to close afterresultbefore force-closing. 5s is generous — every clean turn closes within ~50ms.What this doesn't fix