fix(helix-org): record full agent transcript on activation stream - #2557
Merged
Conversation
The activation stream only recorded lifecycle markers ("=== activation
... ===", "=== exit ... ==="), never the agent's actual transcript
(assistant text, tool_use, tool_result). The org redesign (#2516)
replaced the bridge's client subscribe with a topic-based
SubscribeSessionUpdates call that hardcoded an empty ownerID:
SubscribeSessionUpdates(ctx, cfg.PubSub, cfg.Snapshotter, "", sessionID)
But helix publishes every session update to
GetSessionQueue(session.Owner, sessionID) — the owning user's topic
(observability.go, websocket_external_agent_sync.go, controller/
sessions.go all use the real owner). Subscribing with "" lands the
bridge on session-updates..<id>, a topic nobody publishes to, so it
receives zero frames. Only the markers the spawner publishes directly
survived — exactly the reported symptom.
Fix: resolve the session owner (via a new SpawnerClient.SessionOwner,
backed by Store.GetSession) and subscribe to the owner's topic, mirroring
the browser WS handler in websocket_server_user.go which already does
this and documents why. The transcript test now publishes under a real
owner and would fail if the empty-owner regression returns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
philwinder
added a commit
that referenced
this pull request
Jun 9, 2026
…rker sessions (#2566) * feat(helix-org): session-layer transcript mirror, churn-proof + user prompts Fixes #2557 follow-up: the activation stream was still empty for inline-chat turns, and spawner activations on churned sessions were silently orphaned. ## The problem - Transcripts came only from per-activation spawner bridges, missing all inline-chat turns (no spawner = no mirror). - Even for activations, bridges subscribed too late (after the turn streamed). - Worker sessions churn (stale resume → fresh session), but the old mirror design pinned a fixed session ID, so the stream went silent on churn. ## The fix: session-layer Mirror that *follows* the worker - One persistent per-worker tracker (not per-activation bridge). - Polls the worker's current session (project's most-recent exploratory session — exactly what the inline chat follows). - Re-points the subscription when the session changes, so the stream never goes silent on churn. - **Captures both sides**: user prompts (PromptMessage from WebsocketEvent frames) + agent replies (EntryStream), recorded as `user:` + `assistant:` lines on s-activations-<worker>, deduped once per interaction. ## Architecture - Mirror.Ensure(org, worker) starts a long-lived per-worker tracker; idempotent, persists across activations. - Spawner calls Ensure on each activation; inline chat needs nothing. - ensureBootstrap calls EnsureAll per org (once at startup) so pre-existing workers are mirrored before any activation. - lifecycle.Fire calls Stop to avoid leaking the subscription on delete. - ExploratorySession resolver (wired to store.GetProjectExploratorySession) gives the mirror the stable "current session" to follow; no fixed session IDs. - Poll interval: 5s; stream can lag up to one interval on real session change, then catches up. Proportionate to the churn: no firehose, stays per-worker. ## Testing New tests: TestMirrorCapturesTurnWithoutSpawner (inline chat without spawner), TestMirrorRepointsOnSessionChurn (core fix — mirror follows session change), TestMirrorCapturesUserPrompt (dedup user lines), TestMirrorEnsureIsIdempotent, TestMirrorStop. All helix/lifecycle/server suites green. ## Live verification Inline chat to w-owner landed on s-activations-w-owner (no activation). Inline chat to aaa (on a churned session) landed correctly; mirror followed the session change via re-point polling. ## Known gaps - Multi-part prompts (images) produce no `user:` line; text only (the common case) is covered. - First fresh-session turn (hire) streams before the mirror attaches; every subsequent turn is captured. - aaa's session churn itself (`exit: error: … external agent timeout`) is separate and pre-existing — worth a separate look. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(helix-org): drive worker sessions via shared fire-and-forget API, end churn helix-org worker activations went through the blocking OpenAI-compat chat path (POST /sessions/chat → RunExternalAgent), which waits up to 180s for the whole turn. Real worker turns (git pull specs, read role/identity, commit, push) routinely exceed that, so they were killed mid-turn; the spawner misread the timeout as a "stale session" and opened a fresh one, which also timed out → endless session churn and lost conversation continuity. Fix: use the same canonical, non-blocking primitives every other autonomous flow uses — the cron trigger, spec tasks, the frontend: - StartSession → StartExternalAgentSession (creates session + starts desktop + queues the prompt). For a worker's first activation. - SendMessage → POST /sessions/{id}/messages (fire-and-forget). For every subsequent turn. Neither blocks on the turn, so neither hits the response timeout. The spawner observes completion via pollUntilDone + the transcript mirror. Stale-session detection is deleted, not preserved: a worker keeps ONE durable session, and Helix already recovers a downed desktop transparently (sendCommandToExternalAgent → autoStartDevContainerForSession + pickupWaitingInteraction) on the SAME session — preserving the Zed thread, strictly better than the old "open a fresh session" behaviour. Deleted: in-proc StartChatWithStatus + sseCapture/parseSSE machinery; runtimehelix.StartChatRequest/SessionChatMessage/MessageContent/ NewTextMessage; EnsureAndSend's resume-vs-fresh branching, hadStreamErr, cold-start retry, sendToSession. The shared RunExternalAgent 180s cap is untouched (correct for genuine OpenAI-compat callers). Tests: StartSession (no session) + SendMessage (follow-up) + no-churn-on-down-desktop; removed the cold-start/stale tests for behaviour that no longer exists. Full suites green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(helix-org): trim verbose comments on the mirror + session changes Condense the doc/inline comments added in the two prior commits — keep the essential "why", drop the restated mechanics. No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The per-Worker activation stream (
s-activations-<workerID>, surfaced by theworker_logtool and the live UI) only recorded high-level lifecycle markers:The actual agent transcript — assistant text,
tool_use,tool_result, errors — never appeared, making it impossible to drill into what a Worker's Claude/Zed agent actually did.Root cause
Each activation opens a "live transcript bridge" that mirrors the agent's session WebSocket onto the activation stream. The org redesign (#2516) replaced the old client-method subscribe with a topic-based call that hardcoded an empty
ownerID:But the pub/sub topic embeds the owner:
Every publisher of agent traffic uses the real session owner (
agent/observability.go,websocket_external_agent_sync.go,controller/sessions.go). So the bridge subscribed tosession-updates..<id>— a topic nobody publishes to — and received zero frames. Only the markers the spawner publishes directly survived.The browser WS handler (
websocket_server_user.go) already gets this right and documents it: "The API always publishes to the session owner's queue ... must subscribe using the owner's ID, not their own."Fix
SessionOwner(ctx, sessionID)toSpawnerClient, implemented oninProcHelixClientviaStore.GetSession(mirrors the browser handler).u-owner) on both sides, turning it into a genuine regression guard — it fails if the empty-owner bug returns.Testing
go build ./pkg/org/infrastructure/runtime/helix/ ./pkg/server/— cleango test ./pkg/org/infrastructure/runtime/helix/— all pass🤖 Generated with Claude Code