Skip to content

acpmux: resume dead agents and recreate their linked worktree (cx-fbt7f) - #23120

Merged
teamleaderleo merged 33 commits into
feat-cmux-nextfrom
acpmux-revive-dead-agent
Oct 24, 2026
Merged

teamleaderleo merged 33 commits into
feat-cmux-nextfrom
acpmux-revive-dead-agent

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

A session whose agent dies mid-turn can continue on the same conversation, using the existing bounded retry and Stop rules. Before restarting, acpmux recreates a recorded linked worktree from its local branch, origin branch, or last commit. If recovery fails, cwd stays unchanged and the prompt receives the existing retryable session.folder_missing refusal.

Leo's exact words: "ACPMUX daemons, peer routing and messaging across machines."

The checkout uses the existing configuration-isolated path, with hooks and repository filters disabled. Remote-origin sessions do not recreate worktrees. Usage-limit and sign-in failures are never automatically continued; stderr is drained before classifying a process exit.

This is only the dead-agent/worktree item. Incoming prompts retain the existing transport-continuation policy; this PR adds no idle-status or parent-folder-routing change. No UI surface changed, so there are no gallery variants or screenshot URLs.

Validation

Changelog

Fixed: Agent chats resume after an agent process exits mid-turn and can recreate their deleted linked worktree before restarting.

Bead: cx-fbt7f. Owner stop condition: merged.

Agent session: huge-blue

… missing worktree)

A turn whose agent process died is continued like one a connection error
cut off (cause agent_exited); an agent that exits between turns after an
answered turn leaves the chat idle; a prompt during the backoff fires the
continuation now; a start whose linked worktree is gone adds it back from
its branch; Claude's "You've hit your limit" is a limit error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d2f-72e7-70c0-a474-50025052907f&host=cosmic-gray
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 07:19 — with GitHub Actions Inactive
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by projects-a4 on cosmic-gray · session

@teamleaderleo

teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Owned by cx4-au23120 on huge-blue

@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-fbt7f created for this PR by the cmux coordinator.

teamleaderleo and others added 2 commits October 21, 2026 00:23
The worktree record type lives in git_state.rs, the worktree revive under
worktree_care/, and TurnUsage's methods with the rest of a turn's cost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d2f-72e7-70c0-a474-50025052907f&host=cosmic-gray
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 07:29 — with GitHub Actions Inactive
…is told

Coordinator scope addition: a missing cwd at respawn or at turn start never
kills the session. Its worktree comes back on its branch, else from
origin/<branch>, else detached at its last commit or origin/HEAD; otherwise
(no record, or the recreate fails) the session moves to the nearest folder
above it that exists. The next prompt starts with an [acpmux] line saying
which. Remote chains are left as they were (their sandbox), and git runs
with no repository hooks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d2f-72e7-70c0-a474-50025052907f&host=cosmic-gray
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 08:31 — with GitHub Actions Inactive
teamleaderleo and others added 3 commits October 21, 2026 01:32
The old test asserted the continuation was dropped; requirement 3 makes the
prompt fire it and queue behind it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d2f-72e7-70c0-a474-50025052907f&host=cosmic-gray
…ve-dead-agent

# Conflicts:
#	cmux-tui/crates/acpmux/src/hub/turns.rs
@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 14 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp20wcsegk/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpp_iukoua/summary-missing.json (live):/summary-missing.json (live): View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test_a_pull_request_that_changes_the_shortcut_inputs_still_rewrites_them (main.Sett
ingsOnlyArtifactLeavesTheActionFollowers.test_a_pull_request_that_changes_the_shortcut_
inputs_still_rewrites_them) (job='repair')/
test_a_pull_request_that_changes_the_shortcut_inputs_still_rewrites_them (main.Sett
ingsOnlyArtifactLeavesTheActionFollowers.test_a_pull_request_that_changes_the_shortcut_
inputs_still_rewrites_them) (job='repair')
View Logs
test_base_passes (main.NoticesPairTests.test_base_passes)/
test_base_passes (main.NoticesPairTests.test_base_passes)
View Logs
test_every_routed_input_triggers_the_workflow (main.PullRequestTiers.test_every_rou
ted_input_triggers_the_workflow) (path='.git/config')
View Logs

...and 4 more test failures. View all on Blacksmith: run 40043018826, run 40043018876

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 21, 2026 09:17 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 86055a7: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23120 -f ref=acpmux-revive-dead-agent -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 80beafb to compare; do not merge until green; see lane-rules

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

From the persistent-machines lane: I had this same item (turns surviving a harness restart) on my plan and dropped my version, so this PR is the one. Three cases my brief called for that I couldn't confirm from the description. They may already be covered:

  1. Fallback profile. When turns.rs (~398) re-sends to a fallback profile after "agent process closed", does auto-resume stay off? A test that exactly one of the two fires would pin this down.
  2. Planned stops. A planned restart or update (chat Idle or Closed) should never count as an agent death and trigger a continuation.
  3. Retry cap. A test where the harness dies on every respawn and the retries stop after the cap, leaving the chat disconnected with the queue held.

No reply needed.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Fleet test results at head 146d2b628fe5e5ab3621983c4f92877e81ddcffd, on cmux16 (Darwin arm64):

  • cargo fmt -p acpmux --check: clean
  • cargo clippy -p acpmux --all-targets: no new warnings (the 4 it prints are in files this PR doesn't touch: cmux-worktree-care/store.rs, steer_now_tests.rs, self_command.rs, tests/prompt_images.rs)
  • cargo test -p acpmux --lib: 370 passed, 2 failed. Both fail the same way on base 6b19b5d9180: protocol::tests::protocol_registry_names_every_served_method and worktree_sessions::care_tests::care_a_stopped_daemon_s_sessions_are_use_not_open.
  • cargo test -p acpmux --test hub_integration (all of it): 319 passed, 2 failed, 1 ignored. Both fail the same way on base 6b19b5d9180: session_offers::an_offered_chat_is_a_chip_only_the_person_starts_and_it_reports_its_parent and transfer_guard::a_chat_with_no_asking_mode_moves_under_peer_control_but_not_under_web.
    • All 7 agent_revive::* tests pass, as do all 12 auto_resume::*, all 8 status_indicator::* and all of fork_through::*.
  • scripts/cmux-next/check-no-godfiles.sh --only rust --base origin/feat-cmux-next: exit 0
  • scripts/cmux-next/crash_ratchet.py: acpmux expect count back at 10, the same as base

CI reds at this head, none in this diff (acpmux Rust only):

  • cmux-next checks: check-scrollbars (Swift/CSS), FIRST-TURN-COST (Chief), conventions ERROR in HomeThoughtHeadline.swift, and the package graph CmuxNextHomeEngineTests. All are base reds.
  • Earlier heads also showed publish the merge tree, merge freeze and high-signal review link failing with a GitHub API 403 (rate limit), not a test.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

codex-review: SHIP at 146d2b6

  • Reviewed the current diff, its focused tests, and the affected call paths; no correctness bug found.

@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 23, 2026 20:32 — with GitHub Actions Inactive
@github-actions github-actions Bot added risk:normal Risk tier (docs/ci/risk-tiers.md) and removed risk:high Risk tier (docs/ci/risk-tiers.md) labels Oct 23, 2026
@teamleaderleo teamleaderleo changed the title acpmux: a session whose agent died resumes by itself (and recreates a missing worktree) acpmux: resume dead agents and recreate their linked worktree (cx-fbt7f) Oct 23, 2026
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 23, 2026 20:36 — with GitHub Actions Inactive
@cursor

cursor Bot commented Oct 23, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cursor

cursor Bot commented Oct 23, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex gpt-6.1-sol xhigh): SHIP at 86055a7

Independent final-head review covered dead-agent continuation, linked-worktree recreation and identity checks, archive detached restoration, branch-tip races before exposure, and detached replacement safety. The final focused fleet run passed Linux ACPMUX tests, Rust MSRV, artifact build, and Go checks; remaining Linux lint reds are unchanged base findings, and the macOS job failed during fleet runner acquisition.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

This branch had an error being deployed

1 failed and 3 inactive (1 outdated) deployments
artifacts — 86055a73 Deployed Oct 24, 2026 by teamleaderleo via publish tree-addressed cmux-tui #47611
sentry-dsyms — 86055a73 Deployed Oct 24, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #47611
cmux-tui-merge-tree — 86055a73 Deployed Oct 24, 2026 by teamleaderleo via publish the merge tree #5377
ci-routing-agent — 4941c183 Deployed Oct 23, 2026 by github-actions[bot] via cmux-next macOS placement #32587
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant