fix(api,org): stop auto-wake cutting short long tool-call sessions - #2584
Merged
Conversation
The 60s threshold false-positives on common synchronous tool calls (git push, npm install, find /, gh pr view) because ACP session/update events fire around tool calls, not during them. A single long tool produces no streamed events for its entire duration, so the streaming-context gate decays past 60s and the worker re-prompts a healthy in-flight session. 180s covers the empirically observed gaps with a 3x margin on common slow-tool durations. The load-bearing fix is at the org layer (spawner no longer releases its serialisation lane on a stale 5-minute ActivationTimeout); this is defence in depth at the session layer. Refs: helix-specs design/tasks/002091_the-worker-sessions-are/ Spec-Ref: helix-specs@3eefc4b9b:002091_the-worker-sessions-are
The 5-minute ActivationTimeout was applied to both ensureSession and pollUntilDone via a single shared context. When the deadline fired on a long-running but healthy session (docs-writing, large bundles, slow remote pushes), the spawner returned context.DeadlineExceeded, the per-Worker Queue lane released, and the next pending trigger spawned a decoy interaction on top of the still-running session. The session-layer auto-wake worker then mistook that decoy for a stuck row and fired an unnecessary re-prompt, interrupting the agent mid-flight. The fix splits the deadline by phase: - SessionStartupTimeout (5min default) bounds ensureSession and the pre-session work (project apply, MCP attach, secret injection). Five minutes is generous; a hang here is a real failure. - ActivationRunawayGuard (24h default, not operator-tunable) bounds pollUntilDone as a pure resource-safety backstop. The Queue lane stays held until the session API reports terminal status, which is the correct serialisation behaviour. Stuck-session detection lives at the session layer (auto-wake worker), not the org layer. The org layer's job is to serialise per-Worker and trust the session API. Production wiring (helix_org.go) does not set ActivationTimeout — inherits the new defaults transparently. Tests that set ActivationTimeout=N (test fixture safety blanket) are remapped to ActivationRunawayGuard=N since they were exercising the poll-loop deadline. The startErr-path test maps to SessionStartupTimeout because ensureSession returns immediately via error. New tests TestSpawnerSessionStartupTimeoutBoundsStartup and TestSpawnerPollPhaseNotBoundedBySessionStartupTimeout pin the split. Refs: helix-specs design/tasks/002091_the-worker-sessions-are/ Spec-Ref: helix-specs@46785c552:002091_the-worker-sessions-are
Captures the root cause (decoy interaction + sub-tool-duration threshold), the layered fix (org-layer split, session-layer bump), the rejected LivenessProbe approach, and the test coverage. Per CLAUDE.md convention so the notes survive context compaction. Spec-Ref: helix-specs@6840a3a7b:002091_the-worker-sessions-are
…uto-wake-from Spec-Ref: helix-specs@793ab1e40:002091_the-worker-sessions-are
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Worker sessions running any tool call longer than ~60s (docs-writing,
git push,npm install, large file writes) were being interrupted bya "↻ Retried" / "Incomplete interaction" banner and re-prompted. The
agent was healthy — the chat UI was showing the rendered transcript of
events that already arrived — but the API saw no streamed ACP events
during the tool's execution and concluded the session was stuck.
Two compounding bugs, fixed at the right layer each:
Spawner.ActivationTimeout = 5 minwas applied toboth
ensureSessionandpollUntilDone. When the timer fired on along-running healthy session, the per-Worker
activation.Queuelane released and the next pending trigger spawned a fresh "decoy"
interaction on top of the still-running session. That decoy
(
state=waiting, emptyresponse_message, NULLresponse_entries)matched the auto-wake worker's SQL filter perfectly.
defaultAutoWakeStuckThreshold = 60sis shorterthan typical synchronous tool durations. The streaming-context
gate's
lastPublishdecayed past 60s during a normal 90s tool call,and the gate failed by ~1 second.
Changes
api/pkg/org/infrastructure/runtime/helix/spawner.goSpawnerConfig.ActivationTimeout→SessionStartupTimeout(default 5 min). Applied only to
ensureSessionand pre-sessionwork (project apply, MCP attach, secret injection).
SpawnerConfig.ActivationRunawayGuard(default 24h, notoperator-tunable). Applied only to
pollUntilDone. Pureresource-safety backstop, not a liveness threshold.
Spawnerbody: sharedparentCtxcarries thebearer token;
startupCtxbounds startup;pollCtxbounds thepoll loop. Lane stays held until the session API reports terminal
status — correct serialisation behaviour.
responsibility lives at the session layer.
api/pkg/server/auto_wake_stuck_interactions.godefaultAutoWakeStuckThreshold: 60s → 180s. Covers typicalsynchronous tool durations with 3× margin on the observed ~61s gap.
was picked and what the empirical false-positive mode was.
maybeAutoWake's streaming-contextgate to retract the claim that "tool-call cascades touch lastPublish
on every event" — true for cascades of short tools, false for a
single long tool.
Tests
Spawnerpin the startup/poll split:TestSpawnerSessionStartupTimeoutBoundsStartup— hangingStartSessionfiresSessionStartupTimeoutbefore the muchlarger
ActivationRunawayGuardwould.TestSpawnerPollPhaseNotBoundedBySessionStartupTimeout— pollloop runs past
SessionStartupTimeoutboundary and terminatesonly at
ActivationRunawayGuard. Direct regression test forthe decoy-spawning bug.
TestSpawnerTimeoutEmitsExitErrorretargeted atActivationRunawayGuard.autoWakeStuckThreshold()pin the 180sdefault and the env-var override path.
Createdfixtures bumped from-90sto-4 * time.Minuteso they still clear the new threshold.Docs
design/2026-06-11-auto-wake-tool-call-fix.mdcaptures root-causeanalysis, why the layering matters, the rejected
LivenessProbeapproach, and test coverage.
Verification
go build ./api/pkg/org/... ./api/pkg/server/...— clean.TestSpawner*tests pass (including the two new ones).AutoWake*tests pass.pick up the changes):
[AUTO_WAKE] Started auto-wake worker ... stuck_threshold=180000.sleep 200 && echo "tool finished"tool-call payload. Agent ran the full 200-second sleep.
row,
auto_wake_count=0, transitioned cleanly fromstate=waiting→
state=complete. No decoy interaction ever spawned, includingwell past the old 5-minute
ActivationTimeoutboundary.AUTO_WAKElog entries in the 12-minute test window."It printed: tool finished")with no "↻ Retried" or "Incomplete interaction" banner.
design/tasks/002091_the-worker-sessions-are/screenshots/.Operator notes
HELIX_AUTO_WAKE_STUCK_THRESHOLD_SECONDS=600(or any value above180) as a manual mitigation can drop the override after merge —
180s is the new safe default.
ActivationTimeoutwas a code-level config field with no documentedenv-var binding (verified via
grep -rn HELIX_ACTIVATION_TIMEOUT .). The rename is invisible to operators.ActivationRunawayGuardis intentionally NOT operator-tunable.24h is generous-but-finite; tuning it shorter re-introduces the
decoy-spawning failure mode.
Spec task
002091 —
design.md, requirements.md, tasks.md and the reviewer-flagged
relayering of the fix.
🔗 Open in Helix
📋 Spec:
🚀 Built with Helix