Skip to content

acpmux: _acpmux/peer_state_changed pushes each peer link state change to watchers (peerStateEvents) - #26823

Merged
teamleaderleo merged 8 commits into
feat-cmux-nextfrom
lane/hb-peer-state-events
Oct 28, 2026
Merged

teamleaderleo merged 8 commits into
feat-cmux-nextfrom
lane/hb-peer-state-events

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Leo's ask (manaflow-ai/cmuxterm-hq#4770, plan row 3, one machine-state feed for the composer strip and the sidebar):

"you can just not choose a project ... you just get a little green dot ... as if you were working on your own computer, but now it's remote."

Today the composer and the sidebar can only read a peer's state by polling _acpmux/peers. This PR adds a push. It follows #26756 (merged, 85748f3), which made the state honest: connected only after the handshake.

Contract

Notification _acpmux/peer_state_changed, sent to every watching connection (_acpmux/watch):

{peer: string, state: "connecting"|"connected"|"degraded"|"reconnecting"|"offline",
 lastSeen: number|null (ms), error: string|null, at: number (ms)}

When it is sent:

  • Only on a change of PeerState (peer_link.rs). It never repeats while the state stays the same, so a peer whose retries keep failing sends one reconnecting, not one per attempt.
  • It is sent from the code that changes the state. Every Link change goes through Peer::with_link, which sends the note under the link's lock, so notes go out in order. No timer, no polling.
  • A peer that is added (peer_add, config, lease, reconnect) sends its first state.
  • A peer that is removed (peer_remove, an invalid saved ssh URL, a lease that ends) sends a last offline with error: "removed".
  • A stopped instance sends nothing more. The instance that replaces it speaks for the name.

What each field means:

How a client finds it:

  • Feature peerStateEvents is in the daemon's initialize feature list, next to harnessUpdatePlan.
  • Capability peer-state-events-v1 is in initialize/_acpmux/status/acpmux capabilities.

Registration:

  • Typed in protocol/peer_state.rs (PeerStateChanged, PeerLinkState) and registered in protocol/mod.rs.
  • The IR and the generated web types are regenerated.
  • Documented in docs/acpmux-schema.json.

Ordering across instances: src/peer_states.rs keeps, per name, the instance that speaks for it and its last note, under one lock. Because of that:

  • a replaced or removed instance never speaks after its successor;
  • a reconnect that is still reconnecting sends no repeat;
  • a removal's offline/removed goes out only while the removed instance still speaks.

The notes have their own 256-slot feed, kept apart from the harness feed so a burst of notes never pushes a harness reload out of a slow watcher's buffer. A watcher that falls behind that feed (RecvError::Lagged) gets every name's last note again, so a client that sees the same state twice may ignore the second one. The table keeps one entry per running peer and the last note of at most 64 removed names; the oldest removal is dropped first.

Fan-out uses the same delivery path as _acpmux/harnesses_changed: server/fanout.rs reads the feed and passes each note to server/harness_admin.rs watch_note, which sends a payload marked peerStateChanged as _acpmux/peer_state_changed to connections that watch.

Test: fails before, passes after

tests/peer_state_events.rs a_watcher_hears_each_peer_state_change_once sets up two hubs, with A's link to B going through a TCP relay that the test drops. A watching connection on A must hear:

  1. connecting, degraded (the handshake), then connected, each once, with lastSeen/at set and error null;
  2. after the relay drops, reconnecting once with an error, and nothing more over 6 s while at least one more retry fails;
  3. after remove_peer, offline with error: "removed", and then nothing;
  4. peerStateEvents in initialize's features.

Before (#26756's head, 6fb3ca5 in that run, plus only this test), cmux-ci run https://github.com/manaflow-ai/cmux/actions/runs/40655109455:

test a_watcher_hears_each_peer_state_change_once ... FAILED
assertion `left == right` failed: []
  left: []
 right: ["connecting", "degraded", "connected"]

After (this branch, 969ac56 in that run; the next commit is rustfmt only), cmux-ci run https://github.com/manaflow-ai/cmux/actions/runs/40666930100:

peer_state_events: test result: ok. 1 passed (run 3 times, 3 passes)
acpmux --lib:      test result: ok. 1623 passed; 0 failed   (protocol_ir_is_current included)
peer_integration:  test result: ok. 50 passed
harnesses_changed: test result: ok. 1 passed
cargo clippy -p acpmux --all-targets -- -D warnings: clean

At the head (907be1d, after the review fixes): cmux-ci run https://github.com/manaflow-ai/cmux/actions/runs/40671765409 passed fmt, peer_state_events 3 out of 3, peer_integration 50 out of 50 and clippy. In that run, lib had one failure in an unrelated test, cli::codex_auth_broker. Run https://github.com/manaflow-ai/cmux/actions/runs/40672480655 then passed lib 1624 out of 1624, including the new unit test removed_names_are_kept_up_to_a_bound.

Review

An independent Codex review (gpt-6-astra, high effort, fast mode) ran in four rounds, and each blocking finding was fixed:

  • lease-end ordering;
  • a stopped instance emitting late;
  • a repeated reconnect note;
  • lost notes after watcher lag;
  • the harness feed shared with the notes;
  • unbounded tombstones.

The last round, at 907be1d, found nothing and returned VERDICT: SHIP.

Capture

No capture: no UI changes. The only webview files are the regenerated protocol types (generated/methods.ts, generated/types.ts), which add a type for the new notification. Nothing renders it yet; the composer lane consumes it.

Overlap

pr-overlap names #26747, #26700 and #24797, all from the same account. They share only the registry, spec, capability and feature-list lines, plus Peer's field list. The second one to land rebases and re-blesses the IR. None of them adds a peer-state push.

Risk

Medium-low. This adds one notification to watchers and changes no existing message.

Agent session: agent session

🤖 Generated with Claude Code

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review e18a690: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr26823 -f ref=lane/hb-peer-state-events -f platform=macos -f backend_mode=local

Unclear: web / react-apps-check failed: no known signature; failed step: Check webview test shards. This PR merged before its CI finished: fix forward on main.

CI failed on e18a690c2e (run 40675938063 attempt 1): 2 code, 1 unknown.

Job Verdict Why
web / react-apps-base-check (4) code a web test failed
web / react-apps-base-check (1) code a web test failed
web / react-apps-check unknown no known signature; failed step: Check webview test shards

Failures:

  • seen on other PRs accent-glass-invariants.test.ts no orange or system accent and no opaque glass on the gallery stages: failed (job)
  • seen on other PRs agent-pane-peer-chat.test.ts a chat on an SSH machine > counts its running turn from the prompt and offers no machine switch mid-...: failed (job)
  • seen on other PRs agent-pane-queue-row-click.test.ts queued row with a chip's menu open (chromium) > one press on more closes the model menu and does wha...: failed (job)
  • seen on other PRs agent-pane-queue-row-click.test.ts queued row with a chip's menu open (webkit) > one press on more closes the model menu and does what ...: failed (job)
  • seen on other PRs composer-alignment-centers.test.ts at 720 px (light): + , placeholder and mic share one row center; the columns and the quiet row line ...: failed (job)
  • seen on other PRs composer-alignment-centers.test.ts at 720 px (dark): + , placeholder and mic share one row center; the columns and the quiet row line u...: failed (job)
  • seen on other PRs composer-alignment-centers.test.ts at 420 px (light): + , placeholder and mic share one row center; the columns and the quiet row line ...: failed (job)
  • seen on other PRs composer-alignment-centers.test.ts at 420 px (dark): + , placeholder and mic share one row center; the columns and the quiet row line u...: failed (job)
  • seen on other PRs layout.test.ts only page.css imports the page's sheets: failed (job)
  • seen on other PRs agent-pane-queue-scroll-bottom.test.ts chromium > the queue opened, the view scrolled to its end shows the last row whole, and it stays: failed (job)
  • ... 6 more in the job logs
Matched log lines
web / react-apps-base-check (4): (fail) no orange or system accent and no opaque glass on the gallery stages [586672.65ms]
web / react-apps-base-check (1): (fail) only page.css imports the page's sheets [23.06ms]

Not re-run automatically: web / react-apps-base-check (4), web / react-apps-base-check (1), web / react-apps-check are not machine failures.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: web / Web status; base red: the same failure on the base 3e1acf6; does not block
red: web / react-apps-base-check (1); no finished base run on 3e1acf6 to compare; do not merge until green; see lane-rules
red: web / react-apps-base-check (4); no finished base run on 3e1acf6 to compare; do not merge until green; see lane-rules
red: web / react-apps-check; base red: the same failure on the base 3e1acf6; does not block

Gallery before/after for e18a690: 16 states rendered, 10 images differ, 2153 more touched states not rendered. Images are in the run's artifact, not on this public PR.

gh run download 40675937759 -R manaflow-ai/cmux -n gallery-advisory-26823-e18a690c2e0a8548044eec9a2bce3228d0f89a25 (then read gallery-review.json)

States
State Theme Status Diff Picked by
agent-pane.auth-health/signed-in Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/signed-in Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/signed-out Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/signed-out Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/expired Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/expired Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/unknown Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/unknown Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-inspector/subagents-done Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.chat-inspector/subagents-done Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.chat-inspector/full Apple System Colors Light changed 0.0859% imports types.ts (2 hops)
agent-pane.chat-inspector/full Apple System Colors changed 0.0813% imports types.ts (2 hops)
agent-pane.chat-inspector/workflow-running Apple System Colors Light changed 0.0016% imports types.ts (2 hops)
agent-pane.chat-inspector/workflow-running Apple System Colors changed 0.0013% imports types.ts (2 hops)
agent-pane.chat-inspector/workflow-finished Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.chat-inspector/workflow-finished Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-glance Apple System Colors changed 0.0775% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-glance Apple System Colors Light changed 0.0668% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-glance-open Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-glance-open Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-rules Apple System Colors changed 0.0551% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-rules Apple System Colors Light changed 0.0216% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-groups Apple System Colors changed 0.0606% imports types.ts (2 hops)
agent-pane.chat-inspector/layout-groups Apple System Colors Light changed 0.0422% imports types.ts (2 hops)
agent-pane.chat-width/standard Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/standard Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide-blocks Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide-blocks Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors unchanged 0% imports methods.ts (2 hops)

Pick other states with a Gallery: entry.id, entry.id/variant line in the PR body.

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 28, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-8y5nnl created for this PR by the cmux coordinator.

@blacksmith-sh

blacksmith-sh Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Found 39 test failures on Blacksmith runners:

Failures

Test View Logs
build_only must be a conjunctive exclusion on the helper, signing, and publication jobs
, and must not gate the unsigned app build/
build_only must be a conjunctive exclusion on the helper, signing, and publication jobs
, and must not gate the unsigned app build
View Logs
nightly.yml must derive the embedded key and sign every appcast through the helper/
nightly.yml must derive the embedded key and sign every appcast through the helper
View Logs
src/pages/settings/layout.test.ts/layout.test View Logs
src/pages/settings/layout.test.ts/layout.test View Logs
test/agent-pane-queue-scroll-bottom.test.ts/agent-pane-queue-scroll-bottom.test View Logs
test/agent-pane-queue-scroll-bottom.test.ts/agent-pane-queue-scroll-bottom.test View Logs
test/agent-pane-stream-jitter.test.ts/agent-pane-stream-jitter.test View Logs
test/agent-pane-stream-jitter.test.ts/agent-pane-stream-jitter.test View Logs
test/chat-open-perf.test.ts/chat-open-perf.test View Logs
test/chat-open-perf.test.ts/chat-open-perf.test View Logs

...and 29 more test failures. View all on Blacksmith: run 40675938063, run 40675938010

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

teamleaderleo and others added 8 commits October 28, 2026 03:49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
…val and a lagging watcher

Codex review: a replaced or removed instance could speak after its successor,
a reconnect repeated the state, and a watcher that lagged the feed lost a
transition. PeerStates keeps each name's speaker and last note under one lock,
and a lagging watcher gets every name's last note again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
A burst of peer state notes shared the 64-slot harness feed and could push a
harness reload out of a lagged watcher's buffer. The notes now have their own
256-slot feed, still delivered through watch_note, and a watcher that falls
behind it gets each peer's last note again. Removed names keep their last note
up to 64, the oldest removal going first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
@teamleaderleo
teamleaderleo force-pushed the lane/hb-peer-state-events branch from 907be1d to e18a690 Compare October 28, 2026 10:55
@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo
teamleaderleo merged commit 7f18564 into feat-cmux-next Oct 28, 2026
159 of 171 checks passed
@teamleaderleo
teamleaderleo deleted the lane/hb-peer-state-events branch October 28, 2026 11:31
@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40680723578 at 627df29 failed: cmux-next generated files.
Those jobs last passed at af982bd. This pull request is one of 1 merged since: #26823.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40682236069 at 8114288 failed: cmux-next generated files.
Those jobs last passed at ed7d5da. This pull request is one of 3 merged since: #26823, #26850, #26817.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40681979402 at d6c40eb failed: cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next generated files.
Those jobs last passed at ed7d5da. This pull request is one of 3 merged since: #26823, #26850, #26817.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40681192502 at 3da6f43 failed: cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next generated files.
Those jobs last passed at 52314b8. This pull request is one of 2 merged since: #26823, #26850.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40680828380 at dcba69e failed: cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next generated files.
Those jobs last passed at 52314b8. This pull request is one of 1 merged since: #26823.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40680946738 at 0339b44 failed: cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next generated files.
Those jobs last passed at 52314b8. This pull request is one of 2 merged since: #26823, #26850.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40683834325 at 2645001 failed: cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next generated files.
Those jobs last passed at 52314b8. This pull request is one of 3 merged since: #26823, #26850, #26817.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40683016589 at 9fbc12e failed: cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next generated files.
Those jobs last passed at 52314b8. This pull request is one of 3 merged since: #26823, #26850, #26817.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40680929292 at 9a6044d failed: cmux-next generated files.
Those jobs last passed at 52314b8. This pull request is one of 1 merged since: #26823.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

cmux-bot Bot added a commit that referenced this pull request Oct 28, 2026
…s peers (#26860)

* test(acpmux): a peer connection hears and lists none of this host's peers (failing)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a

* acpmux: a peer daemon's connection hears and lists none of this host's peers

A peer handshake turns _acpmux/watch on, so every daemon peered with this one
heard _acpmux/peer_state_changed for all of this host's peers: their names,
link states and connection errors (#26823). The notes, and the snapshot a
lagging watcher gets, now skip Origin::Peer connections. _acpmux/peers,
_acpmux/status and the peer_* replies give a peer an empty peers list, as
_acpmux/peer_status already gave it this machine's row only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>

This branch had an error being deployed

1 failed and 2 inactive deployments
sentry-dsyms — e18a690c Deployed Oct 28, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #71452
artifacts — e18a690c Deployed Oct 28, 2026 by teamleaderleo via publish to R2 #71452
ci-routing-agent — e18a690c Deployed Oct 28, 2026 by teamleaderleo via plan the macOS fallback build #71452
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant