isolate daemon sessions in worker processes

[kevinjosethomas]

• isolates each root session tree in a recoverable worker while preserving detached and headless workflows.
• adds compact snapshots, attachment-local backpressure, session leases, and transparent supervisor recovery.
• moves schedules into per-session artifacts so heartbeats run concurrently and survive supervisor replacement.

---

Note

High Risk
Major process and protocol change to core session execution, persistence, and scheduling; recovery/idempotency bugs could lose work or leave stray processes, though v1 legacy paths remain for older daemons.

Overview
Replaces the monolithic daemon with a supervisor that routes clients and spawns one resident worker per root session tree (plus a catalog subprocess for saved-session I/O). Print, JSON, RPC, and --no-session interactive runs use a separate owned worker frontend in cli.ts while keeping the same public I/O contracts.

Protocol v2 adds command envelopes (clientId, commandId), generation-aware event cursors, chunked attach snapshots, compact assistant streaming on the private channel, and attachment-local backpressure with catch-up/resync. Clients gain transparent reconnect (recoverDaemon), session_resynced, and daemon retry / daemon restart / shutdown --force.

Concurrency and durability: process-safe session leases block double-writes to the same JSONL; cron/heartbeats move from a global file to per-session scheduled-jobs.json with durable claim/dispatch and concurrent per-worker schedulers. Command and worker recovery journals plus an orphan-process journal avoid replaying uncertain work after crashes; autonomous quality gates now honor AbortSignal and kill detached process trees on abort.

^{Reviewed by Cursor Bugbot for commit 29ee005. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Isolate daemon sessions in dedicated worker processes with supervisor lifecycle management

• Introduces a supervisor/worker process topology where each headless session runs in an isolated worker process, with a new daemon supervisor (daemon-supervisor.ts) managing worker lifecycle, authentication, recovery, and restart.
• Adds session leases (session-lease.ts) so only one process can own a session at a time; a second attempt throws SessionAlreadyActiveError and exits with a user-facing error.
• Introduces a private binary framing protocol (private-framing.ts) between supervisor and workers, with backpressure-aware chunked snapshot streaming and compact assistant-delta encoding to reduce payload size.
• Adds WorkerRecoveryJournal, CommandRecoveryJournal, and orphan-process journal for crash recovery: interrupted dispatches are detected on restart, in-flight commands are replayed after reconnect, and orphaned subprocesses are tracked and cleaned up.
• Migrates cron job state to per-session artifact files with cross-process file locking, atomic claim/record semantics, and a migration path from the legacy global store.
• Extends the daemon protocol to v2 with event generation cursors, session_resynced events, chunked snapshot delivery, ack_result/retry_worker/restart commands, and auto-reconnect with resync in DaemonAgentConnection.
• Risk: existing daemon clients connecting to a v2 supervisor receive a different wire format (envelopes, cursors, resynced events); legacy v1 daemons are detected via hello negotiation and served via a legacy request path.

^{Macroscope summarized 29ee005.}

[macroscopeapp]

Macroscope has since reviewed this pull request. An earlier review was skipped by a cost limit; a review has now completed, so that notice no longer applies.

[kevinjosethomas]

@macroscope-app please review!

[macroscopeapp]

Manual reviews triggered for commit 642ca78:

• Approvability Check
• Correctness Check

^{All prior checks · these links stay valid even if you push more commits.}

[macroscopeapp]

Review in progress! I'll provide feedback through check runs and inline comments once complete.

[macroscopeapp]

Approvability

Verdict: Needs human review

17 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

[linear]

ENG-4576 Prime Agent multi-processing

• Each session should run its own process so process crashes and errors are isolated
  • Daemon should be a supervisor process that can be replaced by any child process if it does crash

ENG-4527 Dispatch heartbeats concurrently across isolated session workers

Summary

The global daemon scheduler serializes heartbeat execution across every isolated root worker. Although each root has its own worker process, the supervisor waits for one heartbeat's complete LLM/tool turn before dispatching the next.

This defeats the throughput benefit of worker isolation for scheduled work and causes later jobs to starve behind earlier or slow jobs.

Current behavior

The supervisor-owned scheduler:

• iterates over due jobs sequentially;
• awaits worker_run_cron for each job;
• receives the worker response only after the full heartbeat LLM/tool turn finishes;
• waits up to 30 seconds for a slow worker before advancing to the next job.

Live stress-test evidence:

• 68 ready worker processes with distinct PIDs;
• 61 active RLM heartbeats;
• 59 overdue heartbeats;
• strong creation-order bias in run counts;
• repeated 30-second worker_run_cron timeouts from one root blocking unrelated jobs.

RLM heartbeat mutations also write the shared cron store from workers and call a local scheduler wake method, even though resident-worker schedulers are not started.

Required behavior

• Atomically claim and advance due jobs before dispatch.
• Dispatch due jobs concurrently across different root workers with no global session or worker cap.
• Preserve per-root prompt serialization.
• Coalesce duplicate missed ticks instead of creating an unbounded backlog.
• Acknowledge worker dispatch when the heartbeat is accepted, queued, or skipped.
• Report heartbeat completion asynchronously for bookkeeping.
• A slow or timed-out worker must not delay unrelated roots.
• Route RLM heartbeat create/update/delete and scheduler wakeups through the supervisor-owned cron store.

Acceptance criteria

• 50 idle root workers with simultaneous 10-second heartbeats begin their turns concurrently across workers.
• One worker blocked for more than 30 seconds does not delay any unrelated heartbeat.
• No creation-order starvation occurs.
• Missed ticks are coalesced per job.
• Heartbeat state remains correct across supervisor replacement.
• Add real-process integration coverage for 50-worker heartbeat fanout and one indefinitely blocked worker.

ENG-4526 Reconnect daemon clients transparently after supervisor replacement

Summary

Attached clients surface a fatal Daemon socket closed error when the global supervisor is replaced, even though resident workers survive and elect a healthy replacement supervisor.

Expected supervisor failover should be represented as temporary connection state, not as a fatal session error.

Observed incident

During a local stress test, one session reached approximately 40 GB of memory. Killing it from Agents View with Ctrl+X produced this sequence:

• The selected worker shut down cleanly.
• The supervisor exited immediately afterward.
• Both attached clients surfaced Error: Daemon socket closed.
• Another resident worker elected a replacement supervisor within a few seconds.
• The replacement adopted the surviving workers.
• Because the intentional-stop descriptor had not been removed before the supervisor exited, the replacement recovered the intentionally killed worker as if it had crashed.

The daemon remained functional, but client continuity failed and the killed session was resurrected.

Required behavior

• Attached clients retain their stable client ID and event cursor across transient supervisor loss.
• Show a non-fatal yellow warning such as “Daemon reconnecting…” while replacement/adoption is in progress.
• Clear the warning after reconnect and cursor replay or snapshot catch-up.
• Only surface a fatal error after bounded reconnection attempts fail.
• Persist an intentional-stop tombstone before asking a worker to exit so a replacement supervisor cannot resurrect a killed root.
• Record supervisor generation and exit reason so future failures are diagnosable.
• Supervisor replacement must not restart or interrupt unrelated resident workers.

Acceptance criteria

• Killing one worker does not disconnect attached clients or restart unrelated workers.
• Killing or restarting the supervisor during active streams produces a temporary warning and reconnects both clients automatically.
• Attachments, stable client IDs, and event cursors survive replacement.
• An intentionally killed root is never recovered by a replacement supervisor.
• A failed reconnect eventually surfaces a clear fatal error without hiding permanent failure.
• Add real-process integration coverage for supervisor loss during worker kill and during an active stream.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 29ee005. Configure here.}