Skip to content

cmux NEXT daemon update: acpmux ssh peers as update targets - #25388

Merged
teamleaderleo merged 5 commits into
feat-cmux-nextfrom
lane/hb-acpmux-peer-update
Oct 26, 2026
Merged

teamleaderleo merged 5 commits into
feat-cmux-nextfrom
lane/hb-acpmux-peer-update

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Part 5 of the rebuild of #23199 and #24070 (adopted by hb-acpmux). On feat-cmux-next, after #25387 (the queue) and #25425 (acpmux host update --keep-newer, which every run here passes) landed. +860, a little over the 800-line guide: about half is the stand-in CLI tests, and the target, its CLI reply and its command only work together.

acpmux ssh peers as daemon update targets. Nothing offers them to the queue yet: the launch wiring and the reconnect sweep are the next PR.

Leo: "BRO CANT YOU JUST LIKE SUSPEND SHIT AFTER A TOOL CALL/TURN THEN JUST GO AND AUTOUPDATE OUR TEAMMATES NEED ACPMUX UPDATING TOO..."

Leo: "OUR SHIT IS HARDLY EVER GOING TO GO IDLE. IT JUST WONT. YOU'LL JUST HAVE TO ACCEPT IT BREAKING AT SOME POINT BUT IT RESUMES WHATEVER IT INTERRUPTED. END OF STORY FOR THE DAEMON UPDATE."

Leo-ask: ask-48bbaec1d4
Leo-ask: ask-061e3574a6

Change

From #24070 at bd4892c, checked against base's acpmux host update (cli/host_update.rs: the row's outcome, target, build, reason, error, migratedFrom, wrapped in {"updated": [...], "peers": [...]}) and _acpmux/status (peer.rs: remoteBuild):

  • AcpmuxPeerUpdateTarget: one ssh peer of the app's daemon. update(to:whileBusy:) runs acpmux --json host update NAME --binary-dir DIR --keep-newer, with --if-idle unless the queue asks while busy. On needsBinary it downloads the bundled build's published cmux-tui-acpmux-<target>, stages it (mode 755, only when the published commit is the build's) and runs once more. unreachable, and failed with reason updateRunning (another update, such as the peer's own self-update timer, holds the host's update lock), are asked again later, never a failure; cloudMachine is left to the Cloud update.
  • AcpmuxPeerUpdateSource: the local daemon's status, the bundled CLI, the staged binaries (other builds' are removed). targets() waits for ssh peers to connect after a launch, then offers the connected ssh peers, and only while the local daemon runs the clean bundled build.
  • AcpmuxPeerUpdateReply / AcpmuxPeerUpdateStatus / AcpmuxPeerUpdateCommand: the only readers of that JSON, and one CLI run through AsyncProcess with a deadline.
  • No wait for idle: with cmux NEXT: one queue that brings other machines' daemons to the app's build #25387's default the queue asks without --if-idle from the first ask, so a peer where an agent works is updated mid-turn and its daemon resumes the turn (as cmux NEXT: update acpmux ssh peers and Cloud machines to the app's build, also mid-turn #24070 did). A peer whose own timer (acpmux: Linux hosts keep their daemon on the newest feat-cmux-next build by themselves #24232) already moved it to a newer build is never moved back: the app skips a peer it already sees on a newer build, and --keep-newer (acpmux: host update --keep-newer never moves a peer back past its own self-update #25425) makes the CLI check again under the host's update lock, so a peer that moved on while the app downloaded the binary is left alone too.

Timers and their stop conditions

  • The settle reads in targets() (15 s, 30 s, 60 s apart): they end once every ssh peer connected and named its build, after 3 minutes, or when the local daemon stops answering.
  • The host update deadline (ProcessRequest.timeout, 15 minutes): ends when the CLI exits; at the deadline AsyncProcess sends SIGTERM, then SIGKILL after a 10 s grace.
  • The exit wait (AcpmuxChildExit, one DemandTimer, 10 s): a stopped run returns once the kernel's exit event for the child arrives (armed at the launch), or at the bound, reported as a child that outlived SIGKILL (a failed run, not retried in this app run). Its exit source ends at the exit or when the run returns.

Tests

AcpmuxPeerUpdateTargetTests (a stand-in acpmux script that logs its arguments and prints canned replies; no real peer or daemon):

  • only connected ssh peers with plain names are targets; a dirty local daemon, or one not on the bundled build, pushes nothing; status and reply parsing (every outcome, the CLI's row for the peer, lines logged before it); ssh host from the peer URL;
  • needsBinary downloads, stages and runs again; a binary that can't be downloaded is needsBinary; rollback and failure carry the CLI's reason;
  • through the queue: theFirstHostUpdateDropsIfIdle (the default); withACapHostUpdatesCarryIfIdleUntilIt (the optional cap: idle, idle, idle, then once without --if-idle, never two runs at once); aPeerOnANewerBuildIsNotMovedBack; anUnreachablePeerIsAskedAgainNotRecordedFailed; aPeerWhoseUpdateLockIsHeldIsAskedAgain (held lock, then success on the next ask);
  • updateArgumentsKeepNewerAndDropIfIdleOnlyWhileBusy; aRunPastItsDeadlineEndsItsChild (a child that ignores SIGTERM is gone when the call returns, and the reply says it exited); anExitWatchWaitsForItsChildUpToItsBound.

Review fixes

Codex (gpt-6-astra, xhigh) BLOCKed the first commit (ebae2c1, now a391c8f) on three findings, each fixed test-first (red 1f891dd, fix 453a627): a download-and-retry could push an older build over a peer that moved on meanwhile (now --keep-newer, checked by the CLI under the lock, #25425); a held update lock was recorded as a failure (now busy); the deadline left a TERM-ignoring child running (now AsyncProcess with SIGKILL escalation). The second round found the SIGKILL path still returned before the exit: red 3e3b887, fix f23da3f (AcpmuxChildExit). The same review found #25425's own-build check skipped a pushed binary with no build id, fixed there.

Red on base: every test here names types base doesn't have, so none compiles there. Red first: 1f891dd has the three review-fix tests on a391c8f's code, without the fix; 453a627 is the fix.

Observed / Not run

  • Fleet Mac (cmux-ci run, package-test-lane suite, step 7731c771 on aws-m4pro-4) at f23da3f: AcpmuxPeerUpdateTargetTests and DaemonUpdatePolicyTests passed.
  • This PR's CI at f23da3f (run 40308174348): the Release compile (Xcode 26) passes; in swift test AcpmuxPeerUpdateTargetTests, DaemonUpdatePolicyTests and DaemonUpdateTargetTests passed, and the red tests (browser tab mirror, ActionRunCreated, ActionBatch deadline) fail on feat-cmux-next too.
  • Not run: no real ssh peer was updated (a stand-in acpmux script answers).

No capture: no visible UI.

Agent session: big-red
🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review f23da3f: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25388 -f ref=lane/hb-acpmux-peer-update -f platform=macos -f backend_mode=local

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by hb-acpmux on big-red · session

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 26, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-g3tgv created for this PR by the cmux coordinator.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next fast gates / post cmux-next/fast-gates is red on 20 of 21 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

high-signal review link / link is red on 19 of 21 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo
teamleaderleo force-pushed the lane/hb-acpmux-peer-update branch 3 times, most recently from 15ac190 to ebae2c1 Compare October 26, 2026 09:47
teamleaderleo added a commit that referenced this pull request Oct 26, 2026
…nder the update lock

With --keep-newer, a peer whose daemon runs a later day's build than the one pushed (its
acpmux-update.timer moved it past it), or a build with local changes, is left alone:
current, reason newer or ownBuild. It is read under the host's update lock, so a
self-update that ran after the caller looked is never undone. The app passes it for its
automatic peer update (#25388).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Oct 26, 2026
… self-update (#25425)

* acpmux: host update --keep-newer, tests first (the option is read, not yet applied)

keeps_over decides it; the next commit applies it under the host's update lock, so
keep_newer_leaves_a_later_or_own_build_and_updates_an_older_one fails here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* acpmux: host update --keep-newer leaves a later or own build alone, under the update lock

With --keep-newer, a peer whose daemon runs a later day's build than the one pushed (its
acpmux-update.timer moved it past it), or a build with local changes, is left alone:
current, reason newer or ownBuild. It is read under the host's update lock, so a
self-update that ran after the caller looked is never undone. The app passes it for its
automatic peer update (#25388).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* acpmux host update --keep-newer: test an own build against a binary with no build id

Red: the check needs both builds, so a source-archive binary stamped
nogit is pushed over someone's dirty build. The next commit fixes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* acpmux host update --keep-newer: an own build is kept whatever binary is pushed

The dirty check no longer needs the pushed binary to name its build: a
source-archive binary stamped nogit is never pushed over someone's own
build. A pushed binary with no date is still not older than anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo and others added 5 commits October 26, 2026 10:31
An acpmux ssh peer of the app's daemon (`acpmux host add NAME ssh://host`)
becomes a DaemonUpdateTarget: `acpmux --json host update NAME --binary-dir
DIR [--if-idle]` on the bundled CLI, with the published
cmux-tui-acpmux-<target> staged when the peer asks for its platform's
binary. The CLI's reply (acpmux cli/host_update.rs) and the daemon's
`_acpmux/status` peers are read in one place each. Nothing offers these
targets yet: the launch wiring and the reconnect sweep come next.

From #24070 (daemon-peer-update bd4892c), on the queue of #25387, whose
default asks idle first and mid-turn after 10 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd a run past its deadline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ck is busy, a stopped run is gone

Every host update passes --keep-newer (#25425), so a peer its own self-update moved past the
app's build after the app read it is left alone under the host's lock. failed/updateRunning
(another update holds the lock) reads as busy and is asked again, never recorded. The CLI runs
through AsyncProcess: past its deadline it gets SIGTERM, then SIGKILL, and the run returns only
once the child is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s child exited

Red: the run returns at the SIGKILL without waiting for the exit, and
there is no exit watch yet. The next commit adds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… exited

The kernel's exit event is armed at the launch (AcpmuxChildExit), and a run
past its deadline waits for it, at most 10 s after the SIGKILL, before the
queue moves on. A child that outlives even SIGKILL is reported as such.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo
teamleaderleo force-pushed the lane/hb-acpmux-peer-update branch from a172688 to f23da3f Compare October 26, 2026 10:32
@teamleaderleo
teamleaderleo marked this pull request as ready for review October 26, 2026 11:42
@teamleaderleo
teamleaderleo merged commit aad2cb4 into feat-cmux-next Oct 26, 2026
145 of 150 checks passed
@teamleaderleo
teamleaderleo deleted the lane/hb-acpmux-peer-update branch October 26, 2026 11:42
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

1 inactive deployment
ci-routing-agent — f23da3fc Deployed Oct 26, 2026 by teamleaderleo via namespace autofix #3345
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant