Skip to content

acpmux: a runner for background tasks that outlive the agent - #26446

Merged
teamleaderleo merged 7 commits into
feat-cmux-nextfrom
hotswap-opus/session-bg-runner
Oct 28, 2026
Merged

teamleaderleo merged 7 commits into
feat-cmux-nextfrom
hotswap-opus/session-bg-runner

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Leo: "also when it comes to hibernation...can't you hibernate the agent but not hibernate the bg tasks?"

Part 1 of 4: the runner. Part 2 (#26463) adds the daemon's methods, the idle-gate rule, the hibernation test and docs/hibernation.md. Part 3 (#26659) ends a purged chat's tasks. Part 4 (#26447) adds the agent's MCP tools and the Claude Code hook.

Why a runner

A harness cannot keep its own background shells alive. We checked both harnesses directly:

  • Claude Code 2.1.310. A run_in_background shell runs in its own session.
    • After SIGTERM to the harness, Claude Code's exit cleanup kills it within 5 s.
    • After SIGKILL it survives, but claude --resume reports it "stopped ... left no completion record" and cannot stop it.
    • The agent host stops a harness with SIGTERM first (agent_host/host.rs), so hibernating would kill every one of these shells.
  • Codex (codex-rs). Unified exec processes do run in their own session (utils/pty pty.rs:324, process_group.rs:49), but they still die with the harness:
    • On Linux they also get PDEATHSIG SIGTERM (process_group.rs:27).
    • On any shutdown the harness calls terminate_all_processes (core/src/session/handlers.rs:584), and dropping a process handle kills its group (unified_exec/process.rs:638).
    • Their output exists only in memory (HeadTailBuffer).

What

acpmux __bg-run (src/bg_run.rs) runs one task. The daemon starts it in a session of its own (part 2), so it outlives the harness and the daemon.

  • Before the task starts, the runner:
    • holds <id>.lock for the task's whole life (the lock, not a pid, says the task runs);
    • creates its stop FIFO <id>.ctl;
    • creates the first output segment.
  • Start. The runner starts bash -c COMMAND (/bin/sh where there is no /bin/bash, matching the agent's Bash tool) as the leader of a new session, with no signals blocked and SIGPIPE back to its default. It then writes <id>.json (written to a temp file and renamed, always under one lock) and answers one line. If the record or the answer fails, it stops the task, so no task runs unrecorded. The runner ignores SIGPIPE, so a closed answer pipe is an error it handles, not a death. A start that fails removes its FIFO and first segment.
  • Output, bounded per task.
    • Segments are <id>.out.<start>, named by where each starts in the whole output. The files alone define what is kept, so a crash cannot leave them disagreeing with a record.
    • At 4 MiB a new segment starts and the one before the previous is deleted. At most two segments are kept: the newest 8 MiB.
    • A reader opens the newest two segments before measuring them. A rotation deletes a name, never an open file, and only the newest segment grows, so a read is always a consistent prefix.
  • Stop.
    • A stop is one byte written to the FIFO. Opening the FIFO for writing fails unless the runner holds it open, so a stop reaches only this runner, and nothing signals a pid.
    • The runner then sends SIGTERM to the task's own setsid group, and SIGKILL to the group 5 s later, so a stop takes 5 s.
    • It learns of the leader's exit with waitid(WNOWAIT) and reaps it only once the pipe ends. The zombie keeps the group id the task's, so a stop still reaches children the leader left behind, and it marks the leader reaped under the same lock it signals under, so it never signals a freed group id.
    • SIGTERM or SIGINT to the runner does the same. If the task has not started yet, the runner just exits. SIGHUP is ignored.
  • Exit. The runner reads the pipe until EOF, so output a child writes after the leader exited is kept. It stops waiting early only when a stop's SIGKILL and grace leave the pipe open (a process that left the group holds it). Once a stop began, the reap also waits for the stop's SIGKILL, so a member that ignored SIGTERM and sent its output elsewhere (it holds no pipe, so the pipe ends without it) is killed too. Then it reaps the leader, writes the exit, removes the FIFO and releases the lock.
  • Spool errors (a full disk) are recorded as outputError, and the pipe keeps being drained so the task never blocks. A read that loses every retry against a rotation is an error ("read again"), never an empty output.
  • Environment. The task gets exactly the env in its spec (the daemon builds it), never the runner's own underneath.
  • Safety of the group signal. SIGCHLD is reset to its default at start, so nothing reaps the leader behind the runner's back. If the leader cannot be waited for, the runner never signals its group. A stop after the task ended changes nothing, its record included. A signal before the task starts removes the FIFO and the first segment.
  • Rotation. A segment that will not delete stops the spool (outputError) rather than keep a third. Records carry their folder's generation (acpmux: background tasks the daemon owns outlive a hibernated agent #26463).
  • No deadline. A task runs while anything it started still holds its output, including a process that left its group (setsid cmd &). That output is still the task's, and a dev server may run for days. A stop ends the wait.

New timers, loops and watchers

Each one, with when it stops and what cleans it up:

  • Signal thread (sigwait loop). It lives as long as the runner process, which exits right after the leader is reaped.
  • FIFO control thread (blocking read). It ends with the runner process; the FIFO file is removed at exit.
  • Output thread (pipe read loop). It ends at EOF (the last holder of the pipe is gone), or with the runner process. The spool files are bounded at two segments.
  • Stop timer (5 s between SIGTERM and SIGKILL, then 5 s more). It runs at most once per task and returns if the leader was reaped (only when no stop began). After the SIGKILL it lets the reap go on; after the last grace it ends the runner's wait for the pipe.

There is no polling anywhere.

Tests

bg_run::tests::the_spool_keeps_two_segments_and_reads_across_them checks:

  • rotation keeps exactly the newest two segments, named by their starts;
  • a tail read gives the right start, total and kept-from;
  • a read from byte 0 starts at the oldest kept byte and crosses the segment boundary.

End to end, the runner is exercised in #26463's hub integration tests: hibernation, the cap, FIFO stop, SIGKILL after an ignored SIGTERM, a child that ignores SIGTERM and holds no pipe, late output after the leader exits, binary output, purge.

Fleet Linux at the exact head 7c08a5f (rebased onto feat-cmux-next with #26127 and #26174): cargo clippy -p acpmux --all-targets -D warnings clean; cargo test -p acpmux --lib -- bg_run 2 passed.

Crash ratchet: the runner's two process::exit calls (a stop before the task starts, and the end of the task) are reviewed // crash-allow waivers. The runner is its own process, and its stop and spool threads may still block in reads. The baseline's waived acpmux exit row goes from 4 to 6. The ratchet's other row, cmux-tui-core unwrap at 1150, is red the same way on base.

Review

Review (codex gpt-6-astra high, fast): SHIP at 7c08a5f (round 8 reviewed the crash-allow waivers; round 7, a fresh review after the rebase at 4e87527 with the patches unchanged from round 6's SHIP at 36d228a, it found no overlap with #26174 and confirmed that the idle census does not see the runner, which runs in its own session).

Fleet, whole stack at 6e2602f (run 40638654802): clippy clean for acpmux and cmux-tui; hub_integration bg_tasks, session_hibernate and processes: 58 passed; lib bg_run, bg_hook, acpmux_mcp, protocol, idle and the config test: 27 passed; tests/agent_tools.rs: 2 passed; cmux-tui bg_steer and instructions: 2 passed.

Risk

Low on its own: nothing starts the runner until part 2.

Agent session: agent session

🤖 Generated with Claude Code

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@cursor

cursor Bot commented Oct 27, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo teamleaderleo changed the title acpmux: background tasks the daemon owns outlive a hibernated agent acpmux: a runner for background tasks that outlive the agent Oct 27, 2026
teamleaderleo and others added 6 commits October 28, 2026 06:28
Leo: "also when it comes to hibernation...can't you hibernate the agent but
not hibernate the bg tasks?"

acpmux __bg-run holds a lock for its task's life, opens a stop FIFO, spools
the task's output into two segments named by their start (newest 8 MiB),
starts /bin/sh -c COMMAND as a new session's leader, records it, and stops
it (SIGTERM, then SIGKILL after 5 s, to the task's own group, only before
reaping the leader) on a FIFO byte or SIGTERM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3891b16636fa80ffc495
… dies of SIGPIPE

Review fixes. The runner ignores SIGPIPE (the task gets it back), so a
closed answer pipe is an error that stops the task. The leader stays
unreaped until the pipe ends, so output a child it left writes later is
kept and a stop still reaches that child through the group id; a stop
whose SIGKILL leaves the pipe open ends the wait after its grace. A spool
write failure is recorded (outputError) and the pipe is still drained. A
start that fails removes its FIFO and first segment. Tasks run under bash
where /bin/bash exists, like the agent's Bash tool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3891b16636fa80ffc495
…nd holds no pipe

The runner reaped the leader at the pipe's EOF, even while a stop waited
out its grace, so a member that ignored SIGTERM with its output sent
elsewhere outlived the stop. The reap now waits for the stop's SIGKILL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3891b16636fa80ffc495
… the task gets exactly its env

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3891b16636fa80ffc495
…el before any stop, and says a spool it cannot list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3891b16636fa80ffc495
…p it cannot wait for, cleans a start a signal cut short, and stops spooling when a segment will not go

A stop after the reap changes nothing. SIGCHLD is reset to default and
a failed waitid marks the group unknown, so nothing signals it. Every
way out of a stop lets the reap go on. Records carry the folder's
generation from the spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3891b16636fa80ffc495
@teamleaderleo
teamleaderleo force-pushed the hotswap-opus/session-bg-runner branch from 36d228a to 4e87527 Compare October 28, 2026 13:28
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 7c08a5f: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr26446 -f ref=hotswap-opus/session-bg-runner -f platform=macos -f backend_mode=local

web and size checks: green

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 28, 2026
The runner (`acpmux __bg-run`) is its own process: a stop before its task starts and the end of
its task both end it with process::exit, since its stop and spool threads may still block in
reads. The crash ratchet's acpmux exit.waived row goes from 4 to 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-3891b16636fa80ffc495
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-payvm5 created for this PR by the cmux coordinator.

@teamleaderleo
teamleaderleo merged commit b3d8997 into feat-cmux-next Oct 28, 2026
199 checks passed
@teamleaderleo
teamleaderleo deleted the hotswap-opus/session-bg-runner branch October 28, 2026 14:46
@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

This branch was previously deployed

3 inactive deployments
artifacts — 7c08a5f0 Deployed Oct 28, 2026 by teamleaderleo via add the Windows daemon to the tree #71973
sentry-dsyms — 7c08a5f0 Deployed Oct 28, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #71973
ci-routing-agent — 7c08a5f0 Deployed Oct 28, 2026 by teamleaderleo via wait for same-key tree publisher #71973
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant