Skip to content

cmux NEXT daemon update: Cloud machines follow the app's build - #25396

Merged
teamleaderleo merged 3 commits into
feat-cmux-nextfrom
lane/hb-cloud-update-follow
Oct 26, 2026
Merged

teamleaderleo merged 3 commits into
feat-cmux-nextfrom
lane/hb-cloud-update-follow

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Part 10 of the rebuild of #23199 and #24070, the last (adopted by hb-acpmux). On feat-cmux-next. #25393, #25394 and #25395 below it landed (c743a34, 30f84dd, 3660113), and this branch is rebased onto them: head 4381e75, the same patch-ids as the reviewed e9e2fa2 (+513 −6).

Cloud machines now follow the app's build by themselves, also while an agent works there (no wait for idle), and the header shows the update.

Leo: "BRO CANT YOU JUST LIKE SUSPEND SHIT AFTER A TOOL CALL/TURN THEN JUST GO AND AUTOUPDATE OUR TEAMMATES NEED ACPMUX UPDATING TOO..."

Leo: "OUR SHIT IS HARDLY EVER GOING TO GO IDLE. IT JUST WONT. YOU'LL JUST HAVE TO ACCEPT IT BREAKING AT SOME POINT BUT IT RESUMES WHATEVER IT INTERRUPTED. END OF STORY FOR THE DAEMON UPDATE."

Leo-ask: ask-48bbaec1d4
Leo-ask: ask-061e3574a6

Change

From #24070 at bd4892c:

  • CloudMachineUpdates.start (from AppServices.startCloud): every new handshake of a Cloud machine's daemon (first connect, reconnect, or one the app refused) offers the machine to the shared queue, behind cloud.updateMachines and agent.updatePeers. HandshakeFollower spends a handshake only on an offer that was made. One seen while the toggles are off, or before the app has a cmux client, is offered once they allow it; the follower observes the toggles too. A machine with no handshake forgets its last, so a second refusal after a drop is new.
  • CloudMachineSession.reconnectAfterUpdate(restarted:): after an update that may have restarted the daemon, the machine connects afresh. A legacy link's waiting connect is woken. An app link drops the current connect's ticket and connects again, after any connect already in flight. This holds also while the app still holds the old daemon's connection, whose drop can arrive after the CLI answered: that late drop is then an old ticket's and changes nothing. An update that restarted nothing leaves a healthy connection alone.
  • CloudLinkSession.currentSocket: the app link's socket without handing it to a connection, so a refused daemon on an app-server link can be updated (old review fix a187752).
  • CloudCreationRows.withUpdate: the header shows the update's line while it runs or retries; a failure stays until the machine runs the build it was to (old review fix e6ff99d).
  • The release-notes highlight now names Cloud machines too.

Timers and their stop conditions

  • The handshake follower (one ObservationStream task): app lifetime; it does no work between handshakes.
  • Each offer (one task): ends once the queue took the machine.

Tests

  • aRefusedDaemonOnAnAppLinkIsUpdatedAndConnectedAgain (CloudMachineUpdateAppLinkTests: a real CloudMachineSession on an app link, a stand-in daemon that answers an old protocol, a stand-in cmux): the refused daemon is updated through the link's socket with --to-bundled, and the machine connects again with a fresh link.

  • theHeaderShowsTheUpdateState: the line while it runs or retries; a failure stays while the machine runs another build, also one serving every capability, and goes once it runs the update's build.

  • CloudMachineUpdateFollowTests: a refusal while off is offered once the toggles turn on; an offer not made stays due; a handshake after a drop is new; an offer is made again after the toggles go off and on; and the reconnect decision after an update, including the CLI answering before the old connection's drop.

Review (Codex gpt-6-astra xhigh): the first round blocked on 2 P2s (a missed app-link reconnect when the CLI finishes before the old drop; refusals deduplicated for good). Both are fixed in 1c7615c. The second round found that turning the toggles off and on again could leave an offered machine unoffered; fixed in e9e2fa2 (turning them off forgets every offer). SHIP at e9e2fa2.

Red on base: these name CloudMachineUpdates and withUpdate, which base doesn't have.

Observed / Not run

  • Release compile at 4381e75 on the Mac fleet (cmux-ci compile-check --release --mac): arm64 passed (step 7e90c0f3cbda4e2334da4c6e, cmux7s-Mac-mini, 1042 s) and x86_64 passed (step 16473528fa5aa094a41c498e, aws-m4pro-7, 460 s). PR CI's app compile (build fleet) was green at 1201f4d: https://github.com/manaflow-ai/cmux/actions/runs/40348120303/job/121661437686
  • Fleet: CloudMachineUpdateTargetTests, CloudMachineUpdateAppLinkTests and CloudMachineUpdateFollowTests pass.
  • PR CI cmux-next swift test at 1201f4d: 26 reds. 24 are on the known base and flaky list. The other 2 pass when their suites (SharedIndicatorSurfacesTests, ChiefGroupChatActionsTests) run again on the fleet; this PR touches neither suite.
  • Codex gpt-6-astra xhigh: SHIP at e9e2fa2.

No capture: the header line appears only during a real Cloud update; its text is DaemonUpdateBoard.line (#25393).

Agent session: big-red
🤖 Generated with Claude Code

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by hb-acpmux on big-red · session

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 26, 2026
@blacksmith-sh

This comment has been minimized.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-ovtxh created for this PR by the cmux coordinator.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next fast gates / post cmux-next/fast-gates is red on 12 of 14 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo
teamleaderleo force-pushed the lane/hb-cloud-update-follow branch from ce50273 to a586eeb Compare October 26, 2026 08:04
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

high-signal review link / link is red on 12 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo
teamleaderleo force-pushed the lane/hb-cloud-update-follow branch from a586eeb to 748a486 Compare October 26, 2026 08:34
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 4381e75: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25396 -f ref=lane/hb-cloud-update-follow -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 3660113 to compare; do not merge until green; see lane-rules

Each handshake of a Cloud machine's daemon (the first connect, every
reconnect, and one the app refused) offers the machine to the shared
daemon update queue (CloudMachineUpdates). After `cmux link machine
update` the machine connects again (CloudMachineSession.reconnectAfterUpdate);
an app-server link's socket is read without handing it to a connection
(CloudLinkSession.currentSocket), so a refused daemon can be updated. The
machine header shows the update's line while it runs or retries, and a
failure until the machine runs the build it was to.

Timers: the handshake follower is one observation task for the app's
lifetime; each offer is one task that ends once the queue took the machine.

From #24070 (daemon-peer-update bd4892c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo and others added 2 commits October 26, 2026 16:12
…also before the old drop, and a skipped handshake stays due

Review fixes on the Cloud follower:
- After an update that may have restarted the daemon, an app link drops
  the current connect's ticket and connects afresh (after a hop in
  flight): the old connection's drop can reach the app after the CLI
  answered, and its endpoint call is then an old ticket's and changes
  nothing. An update that restarted nothing leaves a healthy connection.
- A handshake is spent only by an offer that was made: one seen while the
  toggles are off, or with no cmux client, is offered once they allow it
  (the follower also observes the toggles). A machine with no handshake
  forgets its last, so a second refusal after a drop is new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The policy may drop a machine it queued while the toggles are off, so a
handshake offered before stayed spent and was never offered again once
they turned on. Turning them off now forgets every offer: turned on
again, each live handshake is offered once more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo
teamleaderleo force-pushed the lane/hb-cloud-update-follow branch from f6c07ef to 4381e75 Compare October 26, 2026 16:12
@blacksmith-sh

blacksmith-sh Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Found 8 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp950tomxb/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpezysx7n_/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test_bot_attempt_three_drops_stale_owned_pool_placement (main.PathRoutingStructure.
test_bot_attempt_three_drops_stale_owned_pool_placement)/
test_bot_attempt_three_drops_stale_owned_pool_placement (main.PathRoutingStructure.
test_bot_attempt_three_drops_stale_owned_pool_placement)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Checks job structure is red on 10 of 14 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo
teamleaderleo marked this pull request as ready for review October 26, 2026 16:31
@teamleaderleo
teamleaderleo merged commit 727ab70 into feat-cmux-next Oct 26, 2026
129 of 133 checks passed
@teamleaderleo
teamleaderleo deleted the lane/hb-cloud-update-follow branch October 26, 2026 16:31
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

This branch had an error being deployed

1 failed (outdated) and 2 inactive (1 outdated) deployments
ci-routing-agent — 4381e75d Deployed Oct 26, 2026 by teamleaderleo via namespace autofix #3464
artifacts — 18c208d0 Deployed Oct 26, 2026 by teamleaderleo via publish the Windows daemon to R2 #56726
sentry-dsyms — 18c208d0 Deployed Oct 26, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #56726
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant