Skip to content

fix(helix-org): record full agent transcript on activation stream - #2557

Merged
philwinder merged 1 commit into
mainfrom
fix/helix-org-activation-transcript
Jun 9, 2026
Merged

fix(helix-org): record full agent transcript on activation stream#2557
philwinder merged 1 commit into
mainfrom
fix/helix-org-activation-transcript

Conversation

@philwinder

Copy link
Copy Markdown
Member

Problem

The per-Worker activation stream (s-activations-<workerID>, surfaced by the worker_log tool and the live UI) only recorded high-level lifecycle markers:

=== activation: hire ===
=== exit: ok ===

The actual agent transcript — assistant text, tool_use, tool_result, errors — never appeared, making it impossible to drill into what a Worker's Claude/Zed agent actually did.

Root cause

Each activation opens a "live transcript bridge" that mirrors the agent's session WebSocket onto the activation stream. The org redesign (#2516) replaced the old client-method subscribe with a topic-based call that hardcoded an empty ownerID:

// spawner.go (before)
ch, err := SubscribeSessionUpdates(ctx, cfg.PubSub, cfg.Snapshotter, "", sessionID)

But the pub/sub topic embeds the owner:

func GetSessionQueue(ownerID, sessionID string) string {
    return "session-updates." + ownerID + "." + sessionID
}

Every publisher of agent traffic uses the real session owner (agent/observability.go, websocket_external_agent_sync.go, controller/sessions.go). So the bridge subscribed to session-updates..<id> — a topic nobody publishes to — and received zero frames. Only the markers the spawner publishes directly survived.

The browser WS handler (websocket_server_user.go) already gets this right and documents it: "The API always publishes to the session owner's queue ... must subscribe using the owner's ID, not their own."

Fix

  • Add SessionOwner(ctx, sessionID) to SpawnerClient, implemented on inProcHelixClient via Store.GetSession (mirrors the browser handler).
  • The transcript bridge resolves the owner once and subscribes to the owner's topic.
  • The transcript test now publishes under a real owner (u-owner) on both sides, turning it into a genuine regression guard — it fails if the empty-owner bug returns.

Testing

  • go build ./pkg/org/infrastructure/runtime/helix/ ./pkg/server/ — clean
  • go test ./pkg/org/infrastructure/runtime/helix/ — all pass
  • Verified the guard fails when the empty-owner subscribe is reintroduced.

🤖 Generated with Claude Code

The activation stream only recorded lifecycle markers ("=== activation
... ===", "=== exit ... ==="), never the agent's actual transcript
(assistant text, tool_use, tool_result). The org redesign (#2516)
replaced the bridge's client subscribe with a topic-based
SubscribeSessionUpdates call that hardcoded an empty ownerID:

    SubscribeSessionUpdates(ctx, cfg.PubSub, cfg.Snapshotter, "", sessionID)

But helix publishes every session update to
GetSessionQueue(session.Owner, sessionID) — the owning user's topic
(observability.go, websocket_external_agent_sync.go, controller/
sessions.go all use the real owner). Subscribing with "" lands the
bridge on session-updates..<id>, a topic nobody publishes to, so it
receives zero frames. Only the markers the spawner publishes directly
survived — exactly the reported symptom.

Fix: resolve the session owner (via a new SpawnerClient.SessionOwner,
backed by Store.GetSession) and subscribe to the owner's topic, mirroring
the browser WS handler in websocket_server_user.go which already does
this and documents why. The transcript test now publishes under a real
owner and would fail if the empty-owner regression returns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@philwinder
philwinder merged commit de4ec65 into main Jun 9, 2026
1 check passed
@philwinder
philwinder deleted the fix/helix-org-activation-transcript branch June 9, 2026 07:20
philwinder added a commit that referenced this pull request Jun 9, 2026
…rker sessions (#2566)

* feat(helix-org): session-layer transcript mirror, churn-proof + user prompts

Fixes #2557 follow-up: the activation stream was still empty for inline-chat
turns, and spawner activations on churned sessions were silently orphaned.

## The problem
- Transcripts came only from per-activation spawner bridges, missing all
  inline-chat turns (no spawner = no mirror).
- Even for activations, bridges subscribed too late (after the turn streamed).
- Worker sessions churn (stale resume → fresh session), but the old mirror
  design pinned a fixed session ID, so the stream went silent on churn.

## The fix: session-layer Mirror that *follows* the worker
- One persistent per-worker tracker (not per-activation bridge).
- Polls the worker's current session (project's most-recent exploratory
  session — exactly what the inline chat follows).
- Re-points the subscription when the session changes, so the stream never
  goes silent on churn.
- **Captures both sides**: user prompts (PromptMessage from WebsocketEvent
  frames) + agent replies (EntryStream), recorded as `user:` + `assistant:`
  lines on s-activations-<worker>, deduped once per interaction.

## Architecture
- Mirror.Ensure(org, worker) starts a long-lived per-worker tracker; idempotent,
  persists across activations.
- Spawner calls Ensure on each activation; inline chat needs nothing.
- ensureBootstrap calls EnsureAll per org (once at startup) so pre-existing
  workers are mirrored before any activation.
- lifecycle.Fire calls Stop to avoid leaking the subscription on delete.
- ExploratorySession resolver (wired to store.GetProjectExploratorySession)
  gives the mirror the stable "current session" to follow; no fixed session IDs.
- Poll interval: 5s; stream can lag up to one interval on real session change,
  then catches up. Proportionate to the churn: no firehose, stays per-worker.

## Testing
New tests: TestMirrorCapturesTurnWithoutSpawner (inline chat without spawner),
TestMirrorRepointsOnSessionChurn (core fix — mirror follows session change),
TestMirrorCapturesUserPrompt (dedup user lines), TestMirrorEnsureIsIdempotent,
TestMirrorStop. All helix/lifecycle/server suites green.

## Live verification
Inline chat to w-owner landed on s-activations-w-owner (no activation).
Inline chat to aaa (on a churned session) landed correctly; mirror followed
the session change via re-point polling.

## Known gaps
- Multi-part prompts (images) produce no `user:` line; text only (the common
  case) is covered.
- First fresh-session turn (hire) streams before the mirror attaches; every
  subsequent turn is captured.
- aaa's session churn itself (`exit: error: … external agent timeout`) is
  separate and pre-existing — worth a separate look.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(helix-org): drive worker sessions via shared fire-and-forget API, end churn

helix-org worker activations went through the blocking OpenAI-compat chat
path (POST /sessions/chat → RunExternalAgent), which waits up to 180s for
the whole turn. Real worker turns (git pull specs, read role/identity,
commit, push) routinely exceed that, so they were killed mid-turn; the
spawner misread the timeout as a "stale session" and opened a fresh one,
which also timed out → endless session churn and lost conversation
continuity.

Fix: use the same canonical, non-blocking primitives every other
autonomous flow uses — the cron trigger, spec tasks, the frontend:

- StartSession → StartExternalAgentSession (creates session + starts
  desktop + queues the prompt). For a worker's first activation.
- SendMessage → POST /sessions/{id}/messages (fire-and-forget). For every
  subsequent turn.

Neither blocks on the turn, so neither hits the response timeout. The
spawner observes completion via pollUntilDone + the transcript mirror.

Stale-session detection is deleted, not preserved: a worker keeps ONE
durable session, and Helix already recovers a downed desktop transparently
(sendCommandToExternalAgent → autoStartDevContainerForSession +
pickupWaitingInteraction) on the SAME session — preserving the Zed thread,
strictly better than the old "open a fresh session" behaviour.

Deleted: in-proc StartChatWithStatus + sseCapture/parseSSE machinery;
runtimehelix.StartChatRequest/SessionChatMessage/MessageContent/
NewTextMessage; EnsureAndSend's resume-vs-fresh branching, hadStreamErr,
cold-start retry, sendToSession.

The shared RunExternalAgent 180s cap is untouched (correct for genuine
OpenAI-compat callers). Tests: StartSession (no session) + SendMessage
(follow-up) + no-churn-on-down-desktop; removed the cold-start/stale
tests for behaviour that no longer exists. Full suites green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(helix-org): trim verbose comments on the mirror + session changes

Condense the doc/inline comments added in the two prior commits — keep the
essential "why", drop the restated mechanics. No behaviour change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant