Skip to content

acpmux: a turn a daemon restart cut off resumes by itself, before the queue behind it - #23921

Merged
teamleaderleo merged 8 commits into
feat-cmux-nextfrom
cmux/acpmux-restart-resumes
Oct 26, 2026
Merged

teamleaderleo merged 8 commits into
feat-cmux-nextfrom
cmux/acpmux-restart-resumes

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Leo's exact words: "ACPMUX daemons, peer routing and messaging across machines."

A daemon restart can interrupt a turn after its agent host has exited. Before this change, acpmux marked that turn outcome_unknown and immediately ran prompts queued behind it, leaving the interrupted work idle until a person typed “continue”.

This PR makes the restart recovery resume the interrupted turn first:

  • During log recovery, append one idempotent continuation before the outcome_unknown result. The continuation uses the cut prompt's next attempt id (<prompt>~resume-<n>), quotes the original prompt and steers, and records _meta.provenance.kind=auto_resume with reason=daemon_restart.
  • Restore it through the ordinary durable queue and move it to the head, including the crash window before queue_order is written. Prompts queued behind the cut turn remain behind it and each runs once.
  • Preserve the cut turn's control, trust gate, and kept prompt images. A Web steer keeps the resumed turn in Web control and re-enters the folder trust gate.
  • Respect autoResumeDelaysMs: [] and stop a restart chain after the existing four-attempt bound.

Leo's call is limited to daemon restart recovery; the daemon update schedule and trigger remain in #23199.

Changelog

Fixed: acpmux resumes a turn interrupted by a daemon restart before running prompts queued behind it.

Verification

  • Hosted focused verification run 40017332381 on exact head 598b2a5b4bc5b229b3fdf56f0ca3882e642ce178: selected durable-agent test, Rust MSRV, Go pane, and daemon artifact jobs passed.
  • Hosted Linux/macOS clippy failures are base-red: both report the unrelated base file cmux-tui/crates/cmux-tui-core/src/mux/resource_topology/creation_settlement.rs:35, which this PR does not touch.
  • Local rustfmt --edition 2024 --check and git diff --check pass. Native app builds were not run locally.

Risk: risk:normal.

Agent session: huge-blue

@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 22, 2026
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 22, 2026 00:28 — with GitHub Actions Inactive
@teamleaderleo

teamleaderleo commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Owned by cx4-au23921 on huge-blue

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 4302410: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23921 -f ref=cmux/acpmux-restart-resumes -f platform=macos -f backend_mode=local

web and size checks: green

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-04l0b created for this PR by the cmux coordinator.

@blacksmith-sh

blacksmith-sh Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Found 13 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmpgz3okmn2/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpmps5btbc/summary-fail.json (live):/summary-fail.json (live): View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test_base_passes (main.NoticesPairTests.test_base_passes)/
test_base_passes (main.NoticesPairTests.test_base_passes)
View Logs
test_notices_for_another_ghostty_next_commit_fail (main.NoticesPairTests.test_notic
es_for_another_ghostty_next_commit_fail)/
test_notices_for_another_ghostty_next_commit_fail (main.NoticesPairTests.test_notic
es_for_another_ghostty_next_commit_fail)
View Logs
test_targets_outside_the_app_go_to_the_mac_canary (main.SwiftCanaryFleetCoverage.te
st_targets_outside_the_app_go_to_the_mac_canary)/
test_targets_outside_the_app_go_to_the_mac_canary (main.SwiftCanaryFleetCoverage.te
st_targets_outside_the_app_go_to_the_mac_canary)
View Logs

...and 3 more test failures. View all on Blacksmith: run 40020181944, run 40020181945, run 40020181951

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: no activity for 13 h, and no lane holds a lease on it. Adopted by the orphan sweep: merged feat-cmux-next in at 931e627. A later pass queues it for the merge hub if it is reviewed, small and red only on base reds.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / tooling-tests is red on 17 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

rust workspaces / plan (changed Cargo workspaces) is red on 16 of 16 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: no activity for 54 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in cmux-tui/crates/acpmux/src/hub/prompt_images.rs. Listed for a lane to resolve.

teamleaderleo and others added 7 commits October 26, 2026 08:37
…red queue

When the agent ended with the daemon, the new daemon settles the open turn
outcome_unknown and queues one continuation at the head of the chat's queue:
it quotes the cut turn's prompt and its steers, runs once across further
crashes (prompt id <P>~resume-<n>, logged before the turn_result), and is
recorded as auto_resume_scheduled reason daemon_restart.

Agent-Session: 01a15ca9-5ac5-7692-adfc-a3ae828256ca

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15ca9-5ac5-7692-adfc-a3ae828256ca&host=huge-blue
…trust gate

Codex review of #23921:
- A daemon that stopped between the continuation's queued record and its
  queue_order restored it behind the queue; restore_queue now puts a
  daemon_restart continuation at the head (a later reorder still holds).
- The continuation hard-coded trustGate false; turn_started now records
  the cut turn's trust gate and the continuation carries it (a log from
  before that checks a web turn again).
- Images in the cut prompt or its steers were dropped; they are read back
  from attachments and sent again.

prompt_supersede's restart test (from #23733, newer than this PR) turns
restart continuation off: its until-cancel turn would otherwise run again
ahead of the queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a167e2-7b7b-7aa3-8fcb-5b845bd68fdb&host=huge-blue
The supersede-after-restart test waited for the replacement's answer
before resume_queues, so the restored continuation of the cut turn (ahead
of it in the queue, as Leo asked: it resumes what it interrupted) never
ran and the test deadlocked. Supersede itself was intact (replaced: 1).
Resume first, then wait; also check the cut turn went on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a167e2-7b7b-7aa3-8fcb-5b845bd68fdb&host=huge-blue
…st gate

Codex re-review of #23921: the continuation took control and trust gate
from turn_started alone, so a local turn a Web steer had raised
(turn_control) went on as local with no folder-trust check. It now
applies turn_control, and a Web continuation is always gated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a167e2-7b7b-7aa3-8fcb-5b845bd68fdb&host=huge-blue
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next fast gates / post cmux-next/fast-gates is red on 14 of 20 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next fast gates / post cmux-next/fast-gates is red on 15 of 20 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

high-signal review link / link is red on 21 of 22 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo
teamleaderleo merged commit 7246ee3 into feat-cmux-next Oct 26, 2026
176 of 181 checks passed
@teamleaderleo
teamleaderleo deleted the cmux/acpmux-restart-resumes branch October 26, 2026 09:53
@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

This branch had an error being deployed

2 failed and 2 inactive (1 outdated) deployments
artifacts — 4302410d Deployed Oct 26, 2026 by teamleaderleo via publish the Windows daemon to R2 #56290
sentry-dsyms — 4302410d Deployed Oct 26, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #56290
cmux-tui-merge-tree — 4302410d Deployed Oct 26, 2026 by teamleaderleo via publish the merge tree #6390
ci-routing-agent — 5edd475b Deployed Oct 23, 2026 by github-actions[bot] via cmux-next macOS placement #31907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant