isolate daemon sessions in worker processes

packages/coding-agent/CHANGELOG.md

@@ -12,18 +12,23 @@
 - Added `/btw` and `/side` for one-turn inline side questions that use the current context without changing the main session ([ENG-4509](https://linear.app/primeintellect/issue/ENG-4509/add-btw-and-side-side-question-flows)).
 - Changed scheduled heartbeat prompts to steer (interrupt the current turn) by default, with a `steer`/`follow_up` delivery mode selectable via `/heartbeat --steer|--follow-up` and the `rlm_heartbeat` skill's `delivery_mode` argument.
 - Changed the new-chat splash to show only version, model, and cwd metadata and rotate among five example prompts.
+- Changed daemon and headless execution to isolate each root session tree in a recoverable worker process, with protocol-v2 chunked snapshots, compact streaming, attachment-local backpressure, session leases, and unchanged print, JSON, and RPC interfaces.
 - Fixed self-updates losing restored daemon sessions to a socket cleanup race and leaving open session or agents-view windows disconnected.
+- Fixed supervisor replacement surfacing fatal socket errors or recovering roots that were intentionally stopped ([ENG-4526](https://linear.app/primeintellect/issue/ENG-4526/reconnect-daemon-clients-transparently-after-supervisor-replacement)).
 - Changed daemon connection errors to report the failed operation, session identity, recovery steps, socket, and diagnostic log instead of raw protocol reasons.
 - Changed the Agents View and new-chat splashes to keep one blank row above the butterfly.
 - Fixed Agents View retrying after an intentional daemon shutdown instead of stopping with restart guidance.
-- Fixed stale heartbeat jobs reopening archived, deleted, or concurrently terminated sessions ([ENG-4519](https://linear.app/primeintellect/issue/ENG-4519/heartbeats-rebirth-sessions-that-were-previously-killed)).
+- Fixed daemon catch-up snapshots being disposed mid-transfer or triggering resets that cleared drafts, local queues, dialogs, active UI state, or in-flight reasoning traces.
+- Fixed compact daemon streams occasionally duplicating the first token of an assistant response.
+- Fixed subagent prompts and usage counters flickering or disappearing during daemon resyncs and large parallel runs, and added compact fixed-width recap rows.
+- Fixed stale heartbeat jobs reopening sessions after they were archived, deleted, explicitly shut down, concurrently terminated, or lost resident worker ownership ([ENG-4519](https://linear.app/primeintellect/issue/ENG-4519/heartbeats-rebirth-sessions-that-were-previously-killed)).
+- Fixed heartbeat starvation by moving durable schedules into per-session artifacts and running them concurrently in their owning resident workers, independent of supervisor replacement ([ENG-4527](https://linear.app/primeintellect/issue/ENG-4527/dispatch-heartbeats-concurrently-across-isolated-session-workers)).
 - Fixed onboarding blocking normal TUI use by reopening login or model selection after startup ([ENG-4537](https://linear.app/primeintellect/issue/ENG-4537/stop-onboarding-from-gating-normal-tui-use)).
 - Fixed IPython Bash cells with leading blank lines being labeled and previewed as Python ([ENG-4529](https://linear.app/primeintellect/issue/ENG-4529/leading-newline-before-percentpercentbash-names-tool-call-as-python)).
 - Fixed recap layout shifts by keeping the previous recap visible until its replacement arrives ([ENG-4533](https://linear.app/primeintellect/issue/ENG-4533/reserve-space-for-recap-to-prevent-layout-shift)).
 - Changed the new-chat tray to hide shortcut guidance while typing and keep the `agents` link visible.
 
 ## [0.2.8] - 2026-07-09
-
 - Added built-in Herdr integration that reports agent lifecycle state to Herdr panes automatically, without requiring `herdr integration install pi`.
 - Changed Escape to interrupt active work with a visible abort notice, double Escape to open the session tree from an empty prompt or clear an idle draft, and `?` to show shortcuts ([ENG-4489](https://linear.app/primeintellect/issue/ENG-4489/rewire-prime-agent-shortcuts-to-match-claude-code-flow)).
 - Changed new-chat guidance to show concise shell, command, file, and shortcut hints, with Agents View first and `? for shortcuts` after the model and effort ([ENG-4489](https://linear.app/primeintellect/issue/ENG-4489/rewire-prime-agent-shortcuts-to-match-claude-code-flow)).

packages/coding-agent/docs/daemon-implementation-summary.md

@@ -1,167 +1,159 @@
-# Daemon Implementation Summary
+# Daemon and Session Worker Architecture
 
-This branch adds a tmux-like daemon mode for Prime Agent. The daemon is a background process that listens on a Unix socket and owns active agent sessions. CLI clients connect to that socket to create, attach, detach, prompt, rename, kill, and list sessions.
+Prime Agent isolates each persistent root session tree in its own process. Process isolation is internal: interactive, print, JSON, RPC, piped-stdin, and `--no-session` invocations keep their existing commands and public I/O contracts.
 
-## User Flow
+## Process Topology
 
-The main entrypoint is:
-
-```sh
-prime-agent daemon
+```text
+interactive clients
+        |
+        v
+detached supervisor
+  |- catalog subprocess
+  |- resident worker: root A + RLM descendants + kernels
+  `- resident worker: root B + RLM descendants + kernels
+
+print / JSON / RPC / interactive --no-session
+        |
+        v
+frontend process --private owner channel--> owned worker + root tree
 ```
 
-That command:
+The supervisor owns public sockets, client attachments, routing, global agent-message routing, worker health, and update coordination. Each resident worker owns scheduling and execution for its root tree. The supervisor does not execute providers, tools, compaction, bash, kernels, session schedules, or session-history scans.
 
-1. Starts the background daemon if it is not already running.
-2. Creates a new active session.
-3. Attaches the terminal to that new session.
+Each worker owns one root `AgentSessionRuntime` and every RLM descendant below it. New, switch, fork, and import operations replace the root runtime inside that worker while preserving the root active-session ID.
 
-Explicit daemon commands are also available:
+The catalog subprocess owns saved-session scans and inactive-session file operations. A catalog failure can make a catalog request fail, but cannot interrupt active workers.
 
-```sh
-prime-agent daemon start
-prime-agent daemon list
-prime-agent daemon list -a
-prime-agent daemon create scratch
-prime-agent daemon attach <session>
-prime-agent daemon prompt <session> "Say hello"
-prime-agent daemon rename <session> scratch
-prime-agent daemon kill <session>
-prime-agent daemon shutdown
-```
+## Resident Workers
 
-`prime-agent daemon start` launches a detached background daemon and returns. `prime-agent daemon start --foreground` keeps the daemon attached to the current terminal for debugging.
+Normal interactive sessions use resident workers:
 
-## Session Identity
+- The supervisor dynamically starts one detached process group per root tree.
+- Closing the TUI detaches the client; it does not stop the worker.
+- Worker descriptors, authentication tokens, active-session IDs, session paths, and recovery journals are stored with mode `0600` below the agent directory.
+- Workers monitor the public supervisor socket. If it disappears, one worker acquires an atomic launch lease and starts a replacement supervisor.
+- A replacement supervisor adopts the existing worker PIDs and active-session IDs.
+- A worker crash affects only its root tree. Recovery retries after 250 ms, 1 second, and 5 seconds; three failed attempts mark that root failed.
+- `prime-agent daemon retry <session>` retries a failed root.
+- `prime-agent daemon restart` replaces only the supervisor and adopts workers.
+- `prime-agent daemon shutdown` stops workers before the supervisor; `--force` sends `SIGKILL` to workers that do not stop gracefully.
 
-There are two session identifiers:
+There are no session, worker, client, or workload caps in this layer.
 
-- `activeSessionId`: daemon-local live handle for a currently active session.
-- `sessionId`: persisted JSONL session id.
+## Owned Workers
 
-The `activeSessionId` exists because one persisted session can be attached to a running daemon process and needs a socket-level live handle. The `sessionId` is the durable id stored in the JSONL header.
+Headless and ephemeral modes use the same process isolation without joining the resident daemon registry:
 
-The list table displays 12-character suffix ids. Commands accept suffix matches and report an error if the suffix is ambiguous.
+- `prime-agent -p`, piped stdin, and JSON mode proxy stdin/stdout/stderr through an owned worker and return its existing exit status.
+- RPC keeps stdin/stdout in the frontend. It forwards strict LF-delimited JSONL to the worker and preserves every public command, response, event, and extension-UI shape.
+- If a persisted RPC worker crashes, the frontend reports an ordinary failure for commands whose result is uncertain, reopens the exact persisted session, and continues accepting commands. It never replays an uncertain command.
+- RPC EOF and frontend `SIGTERM`/`SIGHUP` stop only the owned worker and its subprocesses.
+- Interactive `--no-session` uses an owned in-memory worker and creates no recovery descriptor.
+- Owned workers never appear in the agents view and never require the global daemon.
+- Direct SDK calls to `runPrintMode(runtime, ...)` and `runRpcMode(runtime)` remain in-process and source-compatible. Non-serializable extension factories therefore continue to work for embedders.
 
-## Per-Session Config
+An inherited, unreferenced IPC owner channel links the worker to its frontend. Frontend loss disconnects the channel and terminates the worker, which reaps its kernels and detached children; the unreferenced channel does not prevent clean RPC EOF or print completion.
 
-The daemon has default session config from the command that started it. Each created session can override that config.
+## Session Ownership
 
-Examples of per-session config:
+Persisted sessions use process-safe leases keyed by canonical JSONL path. Resident and owned workers both acquire leases.
 
-- `cwd`
-- model/provider
-- tools
-- thinking level
-- extensions, skills, prompts, themes
-- context/settings flags
+- Resident workers acquire a target lease before opening an existing session. Owned CLI workers acquire it before runtime writes, and all workers acquire the replacement lease before switching runtimes.
+- The previous lease is released only after replacement succeeds.
+- Concurrent opens return `session_already_active` with the owning active-session ID.
+- Concurrent daemon creates for the same path share one worker launch.
 
-This means one daemon can own sessions with different models, working directories, and tool sets.
+This prevents a script and resident daemon worker from writing the same session concurrently.
 
-## Persistence Layout
+## Scheduled Jobs
 
-Session JSONL files are stored flat:
+Each resident worker runs one scheduler for the root and RLM descendant sessions it owns. Schedule state is persisted per session at `session-artifacts/<session-id>/scheduled-jobs.json`; workers never scan or write a shared global cron file.
 
-```text
-~/.prime/agent/sessions/<sessionId>.jsonl
-```
+- Creating or changing a heartbeat writes its session store and wakes the same worker's scheduler.
+- Due ticks are claimed and advanced durably before prompt delivery, so a crash never replays an uncertain prompt.
+- The timer loop dispatches different target sessions independently and does not wait for complete model or tool turns before scheduling its next pass.
+- A still-active claim coalesces later missed ticks instead of accumulating a backlog.
+- Supervisor replacement does not pause timers or scheduled work because resident workers remain alive.
+- Worker recovery marks uncertain claims interrupted, preserves the advanced schedule, and resumes only future ticks.
+- The supervisor routes cron commands to workers and merges their job summaries for global listing.
 
-The session root can be overridden with:
+The first worker-owned release migrates the legacy global `cron-jobs.json` into the corresponding session artifact directories before adopting workers. Archived, deleted, or descriptorless sessions are cancelled rather than revived.
 
-```sh
-PRIME_AGENT_SESSION_DIR=/path/to/sessions
-```
+## Public Protocol v2
 
-Session artifacts are stored separately:
+The public daemon socket remains JSONL-framed. Version 2 adds:
 
-```text
-~/.prime/agent/session-artifacts/<sessionId>/...
-```
+- stable command envelopes carrying protocol version, client ID, and command ID;
+- generation-aware event cursors `{ generation, sequence }`;
+- worker states `starting`, `ready`, `recovering`, and `failed` in session summaries;
+- reconnectable clients that retain their stable identity and cursor;
+- attach acknowledgement followed by `session_snapshot_begin`, `session_snapshot_chunk`, and `session_snapshot_end`;
+- 512 KiB target transcript chunks;
+- file-backed transcript caches above 4 MiB.
 
-The current concrete use of `session-artifacts` is RLM/subagent child session storage. A parent session can have child session directories under its artifact directory.
+The one-release v1 update handoff uses raw v1 requests only to prepare and stop the old daemon. Busy older daemons that cannot produce a recovery manifest are left running.
 
-## Session Metadata
+JSON mode and RPC never expose daemon greetings, envelopes, lifecycle messages, snapshot records, or connection metadata.
 
-Metadata that is not inferable from messages is stored as typed append-only JSONL entries in the session file.
+## Private Worker Transport
 
-Examples:
+Supervisor-worker traffic uses a binary frame:
 
-```json
-{"type":"session_info","name":"scratch"}
-{"type":"session_state","state":{"status":"sleep"}}
-{"type":"model_change","provider":"openai","modelId":"gpt-4o-mini"}
-{"type":"thinking_level_change","thinkingLevel":"off"}
+```text
+4-byte JSON header length
+4-byte payload length
+small JSON routing header
+opaque payload bytes
 ```
 
-Current rules:
-
-- Immutable session facts live in the JSONL header: `id`, `timestamp`, `cwd`, `parentSession`.
-- Mutable metadata is appended as typed entries.
-- The latest matching metadata entry wins during reconstruction.
-- Metadata/control entries are ignored by LLM context unless they are explicitly context-bearing, such as `custom_message`.
-
-Daemon inactive state is persisted with `session_state`. It is not guessed from file paths.
+Workers serialize an outbound public event once. The supervisor reads the routing header and writes the same payload buffer to every eligible client.
 
-## List Behavior
+Assistant streaming uses compact start/delta/end payloads on the private channel. The supervisor reconstructs the existing full public `message_update` once per delta. A growing assistant message is therefore not serialized or transferred in full for every token.
 
-`prime-agent daemon list` shows active and idle sessions.
+Transcript snapshots are encoded in the worker, not the supervisor. The worker streams opaque chunks to a bounded supervisor cache. The supervisor forwards chunks while they arrive and never parses a history-sized JSON object.
 
-`prime-agent daemon list -a` includes inactive saved sessions as well.
+## Backpressure
 
-Table columns:
+Backpressure is attachment-local:
 
-```text
-name  id  status  age  model  messages  clients
-```
+- A blocked client stops receiving incremental events.
+- Other clients and workers continue normally.
+- The supervisor retains no unbounded per-client event queue.
+- On drain, that attachment receives a cursor catch-up or a new chunked snapshot.
+- Finalized transcript caching is separate from live partial-message reconstruction.
 
-The `cwd` column is intentionally omitted. `age` is based on the JSONL modified time.
+## Recovery and Idempotency
 
-Statuses are sorted in this order:
+Mutating commands are keyed by `clientId + commandId` and recorded before dispatch in an append-only journal.
 
-1. `user`
-2. `idle`
-3. `tool`
-4. `model`
-5. `sleep`
-6. `crash`
+- A repeated completed command returns its stored result.
+- A received command with no durable result is reported as uncertain and is not replayed.
+- Read and mutation envelopes survive transient supervisor reconnects with the same command ID.
+- Clients durably acknowledge completed mutations, allowing acknowledged journal entries to be compacted away.
 
-Only the status text is colored.
+Workers journal operation transitions and detached subprocess identities. After a crash, the supervisor reaps the old process group plus tracked detached bash trees, appends a visible recovery marker to the persisted transcript, restores the root under the same active-session ID, and holds uncertain side effects rather than replaying them.
 
-Status meaning:
+## Coordinated Updates
 
-- `user`: active session has an attached client and is waiting for user input.
-- `idle`: active session has no attached client and is not currently streaming.
-- `tool`: active session is streaming and currently has pending tool calls.
-- `model`: active session is streaming model output.
-- `sleep`: inactive persisted session that was cleanly killed or shut down.
-- `crash`: inactive persisted session marked as crashed.
+Update preparation is two-phase:
 
-## Protocol Shape
+1. Every resident worker creates a non-destructive checkpoint in parallel.
+2. The supervisor validates and atomically persists the aggregate manifest.
+3. Only after all prepares and validation succeed does it commit and stop workers.
 
-CLI clients send JSON commands over the daemon socket. Commands include an optional request `id`, which is only used to correlate command responses.
+If any prepare or manifest validation fails, prepared workers are released and all roots keep running.
 
-Example command shape:
+## Performance Benchmark
 
-```json
-{"id":"request-1","type":"create","name":"scratch"}
-```
+Run from `packages/coding-agent`:
 
-Example response shape:
-
-```json
-{"id":"request-1","type":"response","command":"create","success":true,"data":{"session":{}}}
+```sh
+npx tsx test/daemon-multiclient-bench.ts
+npx tsx test/daemon-multiclient-bench.ts --generated-session-mib 100
+npx tsx test/daemon-multiclient-bench.ts --generated-session-mib 500
+npx tsx test/daemon-multiclient-bench.ts --session-file /path/to/session.jsonl
+PRIME_AGENT_STRESS_WORKERS=50 npx tsx ../../node_modules/vitest/dist/cli.js --run test/daemon-supervisor-process.test.ts -t "hosts resident roots"
 ```
 
-Attach is event-oriented. The daemon sends an initial `session_attached` event with state and historical messages, then streams subsequent `session_event` messages for new activity.
-
-## Tests
-
-Tests were added or updated around:
-
-- daemon list table output
-- daemon id suffix matching and ambiguity handling
-- persisted session state
-- flat session storage
-- `PRIME_AGENT_SESSION_DIR`
-- session artifact directory behavior
-- RLM/subagent child session placement
+The benchmark reports legacy and v2 fanout/attach paths side by side, including serialization count, elapsed time, throughput, and sampled RSS growth. The process stress case starts the requested number of resident roots, gives every root a simultaneous heartbeat while its own session is busy, and verifies that all schedules advance independently.