Skip to content

acpmux: a peer chat opens at once on its copy while its peer reconnects (read peer chats ahead) - #28626

Merged
teamleaderleo merged 10 commits into
feat-cmux-nextfrom
lane/ao-cached-open
Nov 1, 2026
Merged

teamleaderleo merged 10 commits into
feat-cmux-nextfrom
lane/ao-cached-open

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Leo, 2026-10-30, opening a chat on a remote machine while the link to it was reconnecting: "AND IT DID IN FACT LOAD BUT TOOK 30+ F***ING SECONDS". The pane showed peer_unreachable: <machine> (reconnecting, last seen 10s ago: …) until the link came back.

The Mac's acpmux already keeps a durable copy of each peer chat (hub/peer_snapshot.rs, #25323..#25506), and the pane draws it at once under the " is offline. This is the copy this Mac kept… Last updated {time}" banner while the machine does not answer (peerCopy.ts, #25474). But the copy holds only what a client on this Mac read. A chat nobody opened here, such as a lane another agent started on that machine, has no copy, so it opens on the error until the link returns. A chat read hours ago has a copy that is hours old.

This PR changes the daemon so it reads peer chats ahead:

  • A daemon whose clients open peer chats (an attach forwarded this run) reads each peer's 24 most recently changed open chats (changed in the last 24 h). Closed, stale and empty chats are skipped.
  • When: a round starts 2 s after a peer's first row change, list or reconnect. One round runs per peer at a time, with one read at a time inside it. A round stops at the first read the link refuses; a chat the peer refuses on its own is skipped and read again after the interval.
  • How much: one page per chat per round, at most the pane's open page (400 records, 128 KB, the attach's kinds). Once a copy exists, a read asks only for what comes after it (afterSeq). It asks for the newest page when there is no copy, the copy is of another session, the copy is more than a page behind, or the copy is past the row (a peer back on an older disk, which the newest page then replaces). New records that don't fit one page are read on after the interval.
  • How often: each chat is read (or tried) at most once every 30 s per session generation, and its last change is always read when that interval ends.
  • Shutdown: the daemon's stop cancels every round before its last flush to disk, and no page is kept after the stop.
  • The page is kept the way any forwarded page is kept (capture_peer_page), with the same generation, link and ask-stripping rules. A newest page (beforeSeq past every seq, and no afterSeq) now counts as the chat's own run, like an attach's page. A page asked with both cursors is not kept, since it could skip records. As a side effect, the pane's hover read-ahead (acpmux: a chat opened on screen reads ahead of agent work #28239) also fills the durable copy.

Cost and staleness

bound
Disk unchanged: the copies' own bounds (2 MB + 256 KB per copy, 64 MB in total, least recently used first)
Memory unchanged: 16 MB of records loaded plus 4 MB of heads; the read-ahead state is one Instant per planned chat (at most 24 per peer)
Network, first round at most 24 × 128 KB per peer, once
Network, after that only records past each copy, at most 128 KB per chat per 30 s, and only for chats whose row changed
Staleness while the link is up a copy trails its chat by at most about 32 s (30 s interval + 2 s settle); a chat a client has attached is continued live, as before

Why the daemon and not the webview store: the webview's tail cache (tailCache.ts) lives only in memory and only for chats one page read, so it is gone after an app restart and empty for chats never opened. The daemon copy is durable, already bounded, already drawn by the pane, and already merged with the live page without a flash when the link returns (agent-pane-peer-copy.test.ts: "the machine coming back continues the copy where its page joins it, each record once").

What Leo's Mac showed

Coordinator's diagnostics from Leo's Mac, read-only:

  • ~/.acpmux/peer-snapshots is 63 MB, at the 64 MB bound.
  • The copy of the chat Leo opened is 2.1 MB with seqs 156247–157096. It was captured at 22:13Z, which is when Leo's own late open read the chat.
  • The live chat is at seq 166356, so that copy is now 9,260 records behind.

With this PR, the chat would have had a copy before the open, and it stays within about 30 s of live while the link is up.

Which part ships where

All changes are in the daemon (cmux-tui/crates/acpmux). No app or webview code changes: the pane's copy path and banner already exist. Leo's Mac runs daemon build c230bb5, so this reaches it with the next NEXT nightly's daemon update.

Not in this PR:

  • The handshake failure (ao-rpc-size-audit, ao-peer-hb).
  • The error card's wording (queued-messages-strip).
  • Pane routing (ux-tabs-panes).

Performance

metric before after delta harness or run link
open-to-content, peer chat never opened on this daemon, peer down no content until the link returns (Leo: 30+ s) 3 ms (5 records, from the copy) 30+ s → 3 ms cargo test -p acpmux --test peer_warm_copy -- --nocapture (two hubs, relay cut), local Linux run

Testing

  • cargo test -p acpmux --lib warm: 9 passed (adds the cut-short continuation, the stop, link-vs-chat refusals, the ahead copy and the per-generation throttle). Covers what a round plans (count, freshness, closed/stale/empty skipped, afterSeq vs newest, another generation), the 30 s interval, settle and one round per peer, a newest page being kept while an older beforeSeq page is not, and no read-ahead on a daemon whose clients open no peer chat.
  • cargo test -p acpmux --lib peer_snapshot: 62 passed. cargo clippy -p acpmux --lib --tests -- -D warnings: clean.
  • cargo test -p acpmux --test peer_warm_copy -- --nocapture: passed, open-to-content 1–3 ms across runs. Setup: B runs two chats and A mirrors B through a relay. A's client opens one chat, the other is read ahead, then the relay is cut. Opening the other chat then refuses the attach as peer_unreachable, and the copy holds its last reply.

Review

Astra (gpt-6-astra, high, fast), five rounds: BLOCK at a2cc1e2 (mixed cursors, cut-short catch-up, failures bypassing the throttle, rounds ignoring shutdown), BLOCK at 599fcb1 (stop/capture race, first-page failure never retried, page bursts), BLOCK at 112a7bf (a copy past its row, throttle not per generation), then SHIP at 5559a91. CI's panic ratchet then found this PR's own clippy::indexing_slicing in warm.rs (params["afterSeq"] = ...); 099eca7 inserts through as_object_mut and drops warm.rs's raw arithmetic, and round 5 is SHIP at 099eca7 (no behaviour change). Its one P3 from round 4: no test drives the ahead-copy replacement through Store::behind end to end (the planner side is tested).

Observed / Not run

  • Observed, on the lane VM at 099eca7: the three runs above and clippy. The ratchet's clippy (rust_panic_ratchet.py --flags) finds no hit in warm.rs and none on another changed line.
  • Observed, through cmux-ci at 5559a91 (run 41154152305): lib 2457 passed, 0 failed; peer_integration 51/51; peer_degraded 1/1; peer_warm_copy 1/1; clippy all-targets clean.
  • Observed, base red: cargo test -p acpmux --test peer_warm_copy with the PR's src reverted fails with no copy of a chat never opened here (the copy reply holds no events).
  • No capture: this PR changes no UI code. The "after" view is the existing peer-copy banner (PeerCopyBanner.gallery.ts, agent pane peer snapshots (7): the offline banner and the reply's reading #25474). Not measured on Leo's Mac: that needs this daemon build there.

Changelog

Fixed: A chat on another machine opens at once on the Mac's copy while that machine reconnects, even if it was never opened on this Mac.

Agent session: agent session

🤖 Generated with Claude Code

teamleaderleo and others added 2 commits October 31, 2026 00:26
…le its peer is down

Red on base: the daemon keeps a copy only of what its clients read, so
the test finds no copy of a chat it never opened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-299db44aaee03f80b442
… reconnects

The daemon's copy of a peer chat (hub/peer_snapshot.rs) held only what a
client here read, so a chat nobody opened on this Mac (a lane another agent
started on a remote machine) had none: while its machine reconnected it opened
on an error until the link came back (Leo, 2026-10-30: "AND IT DID IN FACT
LOAD BUT TOOK 30+ FUCKING SECONDS").

A daemon whose clients open peer chats now reads each peer's 24 most recently
changed open chats a little after their rows change (2 s settle, at most once
per chat each 30 s, its last change always read), the pane's open page at most
(400 records, 128 KB), only past what the copy covers once it covers some, and
keeps it as a forwarded page. A newest page (`beforeSeq` past every seq, which
the pane's read-ahead also asks for) is now kept like an attach's page.

Tests: warm_tests.rs (plan, settle, newest page kept); tests/peer_warm_copy.rs
(two hubs, relay cut: open-to-content 3 ms on the head, no copy on the base).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-299db44aaee03f80b442
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 966cda9: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28626 -f ref=lane/ao-cached-open -f platform=macos -f backend_mode=local

Shipped in:

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Found 3 test failures on Blacksmith runners:

Failures

Test View Logs
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)/
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)
View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)/
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

teamleaderleo and others added 2 commits October 31, 2026 00:39
…ge, pass over a refused chat, stop with the daemon

Review of #28626 at a2cc1e2 (Astra):
- a page asked with both afterSeq and a newest beforeSeq is the newest records after the seq, which
  need not start right after it: it is not kept (it would claim the seqs it skipped);
- new records past a copy that do not fit one page are read on from the newest one the page held,
  at most 4 pages a round, and the chat is read again after the interval if they do not get there;
- a chat is marked read when it is tried, so a refusal waits its interval too; a refusal of the
  chat alone passes over to the next chat, and only a link that is down ends the round;
- the daemon's stop (stop_idle_reaper) aborts every round, starts none, and no round keeps a page
  after it, before the last flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-299db44aaee03f80b442
…d chat, read one page a chat a round

Review of #28626 at 599fcb1 (Astra):
- the stopped check and the capture happen under the lock Warm::stop takes, and the daemon stops
  the reads (stop_idle_reaper) before its last flush, so no page is queued after that flush;
- a chat the peer refuses (not the link) is read again after the interval;
- one page a chat a round: new records past a copy that do not fit it are read after the
  interval (the plan continues from what the copy covers then), never in a burst.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-299db44aaee03f80b442
…e same session's read

Review of #28626 at 112a7bf (Astra):
- a copy past the row's lastSeq (a peer back on an older disk with the same session) is read
  again from the newest page, which replaces it (Store::behind), instead of passing as current;
- the 30 s interval is kept per session generation, so a read of the session before a recreation
  never holds back the first read of the new one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-299db44aaee03f80b442
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (gpt-6-astra, high, fast; independent of this PR's Claude author): SHIP at 5559a91

Rounds: BLOCK at a2cc1e2 → BLOCK at 599fcb1 → BLOCK at 112a7bf → SHIP at 5559a91. All blockers were fixed in the commits after each round (see the PR body's Review section). Remaining P3, non-blocking: no test drives the ahead-copy replacement through Store::behind end to end; the planner side is tested.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Non-green checks at 5559a91, classified:

  • cmux-tui spec inventory / clippy (changed cmux-tui packages), panic ratchet step: not this PR's code. The job says "red for acpmux, cmux-tui", and its annotations list only TRACKED_LINTS increases, which the script warns on and passes. Re-running the ratchet script on acpmux alone at this head (cargo clippy -p acpmux --lib --bins --locked with rust_panic_ratchet.py --flags, then rust_panic_ratchet.py --clippy-json, run 41155746946) reports no DENY_LINTS gain for acpmux: only arithmetic_side_effects 1498 > 1430 and cast_possible_truncation 222 > 215 as warnings, plus "counts went down in acpmux". This PR changes no cmux-tui crate source, and its new acpmux code is not feature-gated.
  • nightly-guards / workflow-guard-tests / release-notary and nightly-guards / Guard status: known base reds; neither job reads acpmux.
  • PR checks / link, "Update the PR's review line": a PR-metadata step with a bare exit 1, not a test. The same run shows GitHub 403s on the shared quota, so it most likely needs a re-run.

Full acpmux suite at this head (run 41154152305): lib 2457 passed, 0 failed; peer_integration 51/51; peer_degraded 1/1; peer_warm_copy 1/1; clippy --all-targets -D warnings clean.

teamleaderleo added a commit that referenced this pull request Oct 31, 2026
…28626 and base 976656f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The read-ahead's params were set by Value IndexMut (clippy::indexing_slicing,
a deny lint of rust_panic_ratchet.py) and its timing used raw Instant and u64
arithmetic (arithmetic_side_effects). Insert the cursor through as_object_mut,
add durations with checked_add, and write the constants without arithmetic.
warm.rs now has no ratchet hit; no hit falls on another changed line.

Ran: cargo clippy -p acpmux --lib --tests -D warnings (clean); cargo test -p
acpmux --lib peer_snapshot (62 passed); --test peer_warm_copy (1 passed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-299db44aaee03f80b442
@cursor

cursor Bot commented Oct 31, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Correction to the reds classification above: the Panic ratchet red at 5559a91 was this PR's own, not base. A per-file ratchet scan found two clippy::indexing_slicing hits (a deny lint) at cmux-tui/crates/acpmux/src/hub/peer_snapshot/warm.rs:187-188, from params["afterSeq"] = json!(..) (serde_json Value IndexMut).

099eca7 fixes it: the cursor goes in through as_object_mut().insert, and the Instant and u64 arithmetic in warm.rs is now checked_add / saturating_sub. Running rust_panic_ratchet.py --flags clippy on acpmux at this head finds no hit in warm.rs and none on any other line this PR changes; the 34 arithmetic_side_effects hits left in peer_snapshot.rs are all on lines it does not touch (base code). cargo clippy -p acpmux --lib --tests -- -D warnings is clean, cargo test -p acpmux --lib peer_snapshot passes 62/62, and --test peer_warm_copy passes 1/1.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (gpt-6-astra, high, fast; independent of this PR's Claude author): SHIP at 099eca7

Round 5 covers 5559a91..099eca7, the panic-ratchet fix: no behaviour change. The params are the same (NEWEST = 9007199254740991, limit 400, maxBytes 131072, the same seven kinds), the guarded saturating_sub gives the old result, and checked_add keeps the retry and settle deadlines. Rounds: BLOCK at a2cc1e2 → BLOCK at 599fcb1 → BLOCK at 112a7bf → SHIP at 5559a91 → SHIP at 099eca7. Remaining P3, non-blocking: no test drives the ahead-copy replacement through Store::behind end to end.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Awareness note from an agent lane that does not own this PR; it is not a review or a request to take the PR over.


Reds at 099eca7, by name:

  • Panic ratchet (job 124414122441, run on the merge commit 8a9950d with the base tip): every RED entry is in a crate this PR does not change.

    • cmux-tui-core: clippy::indexing_slicing 856 > baseline 853 (agent_hooks.rs, apps/*, …)
    • cmux-tui-core: clippy::string_slice 57 > baseline 52
    • cmux-chats-mirror/src/index.rs:143, chat.rs:78, feed.rs:54, feed.rs:66: allow(cast_possible_truncation) needs reason

    The PR changes only cmux-tui/crates/acpmux (8 files). The RED list has no acpmux entry: the PR's own indexing_slicing at warm.rs:187-188 is gone. acpmux shows up only as warnings for the tracked lints (arithmetic_side_effects, cast_possible_truncation), which never fail the job, and none of those hits is on a line this PR changes (local ratchet scan). "red for acpmux, cmux-tui" is the job's package scope (changed packages plus their dependents), not a list of red crates. So this is a base red.

  • release-notary (cmux-next guards): base red, as before.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at 5cc98ca

SHIP at 5cc98ca

Visual verdict: no images; Leo tries the build

  1. [P2] [follow-up] warm.rs:294 — Pruning read_at to the current top 24 bypasses the 30-second throttle. With 25 active chats, a recently read chat can leave the selection for one round, lose its timestamp, then change again and immediately re-enter. It is fetched again before its interval expires, potentially repeating every few seconds. Retain unexpired timestamps independently of the current selection and test this rotation.

Source review completed. Tests could not run because Cargo is unavailable in the box.

Follow-ups:

  • Preserve the per-generation throttle when chats leave and re-enter the selected 24.

cmux-bot[bot]
cmux-bot Bot previously approved these changes Nov 1, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 5cc98ca: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (gpt-6-astra, high, fast; carried): SHIP at 966cda9

Carried from the round-5 SHIP at 099eca7. 966cda9 only merges feat-cmux-next, and the PR's own diff has the same patch-id (git patch-id --stable d418d1bbc155 at both heads). The base merge picks up #29799, which fixes the cmux-tui crate ratchet hits that #29582 put on base (indexing_slicing 832 > 831, string_slice 83 > 82). The panic ratchet named those hits at 5cc98ca, and they are not in this PR's files.

@cmux-bot
cmux-bot Bot dismissed their stale review November 1, 2026 01:57

A newer verdict blocks 966cda9 (no independent SHIP at the head).

@teamleaderleo
teamleaderleo merged commit 853f0e0 into feat-cmux-next Nov 1, 2026
210 of 214 checks passed
@teamleaderleo
teamleaderleo deleted the lane/ao-cached-open branch November 1, 2026 02:36

This branch was previously deployed

3 inactive deployments
artifacts — 966cda9c Deployed Nov 1, 2026 by teamleaderleo via add the Windows daemon to the tree #92814
ci-routing-agent — 966cda9c Deployed Nov 1, 2026 by github-actions[bot] via cmux-next macOS placement #56876
sentry-dsyms — 966cda9c Deployed Nov 1, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #92814
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant