feat(helix-org): session-layer transcript mirror + fire-and-forget worker sessions - #2566
Merged
Merged
Conversation
…prompts Fixes #2557 follow-up: the activation stream was still empty for inline-chat turns, and spawner activations on churned sessions were silently orphaned. ## The problem - Transcripts came only from per-activation spawner bridges, missing all inline-chat turns (no spawner = no mirror). - Even for activations, bridges subscribed too late (after the turn streamed). - Worker sessions churn (stale resume → fresh session), but the old mirror design pinned a fixed session ID, so the stream went silent on churn. ## The fix: session-layer Mirror that *follows* the worker - One persistent per-worker tracker (not per-activation bridge). - Polls the worker's current session (project's most-recent exploratory session — exactly what the inline chat follows). - Re-points the subscription when the session changes, so the stream never goes silent on churn. - **Captures both sides**: user prompts (PromptMessage from WebsocketEvent frames) + agent replies (EntryStream), recorded as `user:` + `assistant:` lines on s-activations-<worker>, deduped once per interaction. ## Architecture - Mirror.Ensure(org, worker) starts a long-lived per-worker tracker; idempotent, persists across activations. - Spawner calls Ensure on each activation; inline chat needs nothing. - ensureBootstrap calls EnsureAll per org (once at startup) so pre-existing workers are mirrored before any activation. - lifecycle.Fire calls Stop to avoid leaking the subscription on delete. - ExploratorySession resolver (wired to store.GetProjectExploratorySession) gives the mirror the stable "current session" to follow; no fixed session IDs. - Poll interval: 5s; stream can lag up to one interval on real session change, then catches up. Proportionate to the churn: no firehose, stays per-worker. ## Testing New tests: TestMirrorCapturesTurnWithoutSpawner (inline chat without spawner), TestMirrorRepointsOnSessionChurn (core fix — mirror follows session change), TestMirrorCapturesUserPrompt (dedup user lines), TestMirrorEnsureIsIdempotent, TestMirrorStop. All helix/lifecycle/server suites green. ## Live verification Inline chat to w-owner landed on s-activations-w-owner (no activation). Inline chat to aaa (on a churned session) landed correctly; mirror followed the session change via re-point polling. ## Known gaps - Multi-part prompts (images) produce no `user:` line; text only (the common case) is covered. - First fresh-session turn (hire) streams before the mirror attaches; every subsequent turn is captured. - aaa's session churn itself (`exit: error: … external agent timeout`) is separate and pre-existing — worth a separate look. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… end churn
helix-org worker activations went through the blocking OpenAI-compat chat
path (POST /sessions/chat → RunExternalAgent), which waits up to 180s for
the whole turn. Real worker turns (git pull specs, read role/identity,
commit, push) routinely exceed that, so they were killed mid-turn; the
spawner misread the timeout as a "stale session" and opened a fresh one,
which also timed out → endless session churn and lost conversation
continuity.
Fix: use the same canonical, non-blocking primitives every other
autonomous flow uses — the cron trigger, spec tasks, the frontend:
- StartSession → StartExternalAgentSession (creates session + starts
desktop + queues the prompt). For a worker's first activation.
- SendMessage → POST /sessions/{id}/messages (fire-and-forget). For every
subsequent turn.
Neither blocks on the turn, so neither hits the response timeout. The
spawner observes completion via pollUntilDone + the transcript mirror.
Stale-session detection is deleted, not preserved: a worker keeps ONE
durable session, and Helix already recovers a downed desktop transparently
(sendCommandToExternalAgent → autoStartDevContainerForSession +
pickupWaitingInteraction) on the SAME session — preserving the Zed thread,
strictly better than the old "open a fresh session" behaviour.
Deleted: in-proc StartChatWithStatus + sseCapture/parseSSE machinery;
runtimehelix.StartChatRequest/SessionChatMessage/MessageContent/
NewTextMessage; EnsureAndSend's resume-vs-fresh branching, hadStreamErr,
cold-start retry, sendToSession.
The shared RunExternalAgent 180s cap is untouched (correct for genuine
OpenAI-compat callers). Tests: StartSession (no session) + SendMessage
(follow-up) + no-churn-on-down-desktop; removed the cold-start/stale
tests for behaviour that no longer exists. Full suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
philwinder
changed the title
Condense the doc/inline comments added in the two prior commits — keep the essential "why", drop the restated mechanics. No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related fixes to how helix-org drives worker (Zed external-agent) sessions, found in one investigation:
1. Session-layer transcript mirror (churn-proof, two-sided)
Worker transcripts now reach
s-activations-<worker>for every turn — spawner activation or human inline chat — via one persistent per-workerMirrorthat follows the worker's current session (polls the project's exploratory session, re-points on change). Captures bothuser:prompts andassistant:/tool_*replies. Replaces the per-activation bridge that missed inline-chat turns and went silent on session changes.2. Fire-and-forget worker sessions (ends session churn)
The spawner drove turns through the blocking OpenAI-compat path (
/sessions/chat→RunExternalAgent, 180s cap). Long worker turns exceeded it → misread as "stale session" → fresh session opened → churn. Now it uses the same canonical, non-blocking primitives every other autonomous flow uses:StartSession→StartExternalAgentSession(the cron trigger's interface) — first activation.SendMessage→POST /sessions/{id}/messages(frontend + spec tasks) — every subsequent turn.Stale detection deleted: a worker keeps one durable session; Helix already auto-resumes a downed desktop on the same session (
autoStartDevContainerForSession+pickupWaitingInteraction), preserving conversation continuity. The sharedRunExternalAgent180s cap is untouched (correct for real OpenAI-compat callers).Verification
user:+assistant:ons-activations-w-owner/-aaa, captured across a session change (mirror re-pointed).external agent response timeout/open fresh helix sessionerrors, downed desktops auto-resume on the same session.Design docs
design/2026-06-09-activation-stream-transcript-still-empty.mddesign/2026-06-09-helix-org-session-churn-fix.md🤖 Generated with Claude Code