Skip to content

Fix forked workers exiting immediately and false memory-persistence failures - #634

Merged
jamiepine merged 2 commits into
mainfrom
jamiepine/worker-fork-fixes
Aug 10, 2026
Merged

Fix forked workers exiting immediately and false memory-persistence failures#634
jamiepine merged 2 commits into
mainfrom
jamiepine/worker-fork-fixes

Conversation

@jamiepine

Copy link
Copy Markdown
Member

Two bugs from the worker-reliability work, both found by testing #633 against a live instance.

Forked workers never ran

Forking channel history by default populated prior_history for every new worker. run_inner read that field as "this worker already ran and relayed its result", so every forked worker skipped its task loop entirely and returned an empty result in about a millisecond:

05:11:42.605543  worker starting      task=Clone the GitHub repository...
05:11:42.605854  resuming interactive worker with prior history  prior_messages=2
05:11:42.606735  worker completed

No LLM call, no tools, nothing. The agent got empty results back and re-spawned the same worker four times trying to verify work that never happened.

prior_history was carrying two meanings: a resumed idle worker, and a fresh worker seeded with channel context. Only the first has already relayed a result, and the state machine already distinguishes them — Worker::resume is the only pre-run site that sets WaitingForInput:

let resuming = self.state == WorkerState::WaitingForInput;

Second, worker_runs.transcript was read as Vec<u8>. A NULL blob decodes to an empty vec rather than erroring, so NULL became Some([]), failed the gzip header read with unexpected end of file, and suppressed the live-transcript fallback — which is why worker transcripts came back empty in the UI and via worker_inspect.

Memory persistence reported failure after succeeding

A persistence run called memory_persistence_complete, then took one more LLM turn with nothing left to say. The provider answers that with an empty message, the Responses parser rejects it, and the branch reported:

Branch failed: CompletionError: ResponseError: empty or unsupported response from
OpenAI ChatGPT Responses API ... received output types: message

Every memory had already been persisted. Channel turns already terminate after reply/skip for exactly this reason; the branch's terminal tool never got the same treatment. The hook now terminates once a valid terminal outcome is recorded, branches conclude from that outcome, and ingestion treats the signal as a completed chunk. Tool errors still continue, so the contract-retry path is unchanged.

This also drops one LLM call per persistence run and per ingestion chunk.

Testing

Two hook tests cover the new termination: a valid terminal outcome stops the loop with the right reason (both saved and no_memories), and a tool error does not. Full lib suite passes at 1100.

The worker fix has no test — nothing constructs a Worker outside channel_dispatch with live AgentDeps, so covering resume-vs-fork needs a test fixture that doesn't exist yet. Worth adding separately, since this is the second bug in that path.

Forking channel history by default populated `prior_history` for every
new worker, and `run_inner` read that field as "this worker already ran
and relayed its result". Every forked worker skipped its task loop and
returned empty in about a millisecond. Resumption now keys off the state
the worker was built with — only `Worker::resume` yields WaitingForInput.

Also read `worker_runs.transcript` as nullable. A NULL blob decodes to an
empty vec rather than erroring, so it became `Some([])`, failed the gzip
header read, and suppressed the live-transcript fallback.
A memory-persistence run called `memory_persistence_complete` and then
took one more LLM turn with nothing left to say. Providers answer that
with an empty message, the response parser rejects it, and the branch
reported failure after every memory had already been persisted.

Terminate the loop once the terminal outcome is recorded, matching what
channel turns already do after reply/skip. Branches conclude from the
recorded outcome; ingestion treats the same signal as a completed chunk
instead of a failure.
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c2b1cd97-8f5d-410b-9904-6ba302ecc7e4

📥 Commits

Reviewing files that changed from the base of the PR and between a95b32e and c79fcc4.

📒 Files selected for processing (5)
  • src/agent/branch.rs
  • src/agent/ingestion.rs
  • src/agent/worker.rs
  • src/conversation/history.rs
  • src/hooks/spacebot.rs

Walkthrough

Memory-persistence completion now records terminal outcomes, stops hook processing, and propagates completion through ingestion and branch execution. Worker resumption and empty transcript handling also receive state-specific updates.

Changes

Memory persistence completion

Layer / File(s) Summary
Hook completion contract
src/hooks/spacebot.rs
SpacebotHook adds a completion reason and predicate. Valid memory_persistence_complete results record terminal outcomes and stop the hook loop. Tests cover saved, no-memory, and error outcomes.
Agent completion propagation
src/agent/branch.rs, src/agent/ingestion.rs
Branch execution clears the active contract and formats the terminal outcome. Ingestion treats the completion cancellation as successful chunk processing.
Worker and transcript state handling
src/agent/worker.rs, src/conversation/history.rs
Worker resumption detection uses WaitingForInput. Fresh workers log seeded forked messages. Empty transcript blobs use the missing-transcript fallback.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jamiepine/worker-fork-fixes

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jamiepine
jamiepine marked this pull request as ready for review August 10, 2026 06:40
@jamiepine
jamiepine merged commit c972487 into main Aug 10, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant