Skip to content

acpmux: ACP session/list goes in pages when the whole list is too large to send - #28161

Merged
teamleaderleo merged 10 commits into
feat-cmux-nextfrom
lane/acpmux-acp-list-pages
Oct 30, 2026
Merged

teamleaderleo merged 10 commits into
feat-cmux-nextfrom
lane/acpmux-acp-list-pages

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

ACP's session/list answered every session, local and mirrored, with tools, in one reply, and took no parameters. On a daemon that mirrors a fleet that reply passes the 15 MiB message limit: one daemon's 5241 rows were about 17 MB with tools on 2026-10-30 (the same list that broke peer handshakes, #28105), so session/list there came back as message.too_large, with no way to read less.

session/list now reads ACP's own cursor and answers with nextCursor (both in ACP's session/list schema). A list that fits in 14 MiB (a megabyte under the limit, for the frame and masking) is one reply exactly as before, newest first, cached per list build, with no nextCursor. A larger one goes in pages of at most 14 MiB, cut by bytes (hub/session_pages.rs byte_page, at least one row per page).

  • Order and cursor: pages go in the order of a 128-bit hash of each row's sessionId and peer, not newest first, and the cursor is that hash (h + 32 hex, 33 bytes whatever the ids). A session that updates or leaves while a client pages moves no other row, so none is skipped or repeated; a row listed twice (one id, one peer) comes once; a page never ends between rows of one hash.
  • Cost: a page is cut from the list snapshot's own row spans (FullRows in hub/list_guard.rs), so it parses and serializes only its own rows. A list whose rows alone are over the budget does not build the whole reply only to drop it.
  • Masking: a peer reads the page masked, which can lengthen it: such a page is cut again at half the budget. Raw pages take the same final size check.
  • Oversized row: a single row larger than a page goes as a stub with _meta.acpmux.omitted {bytes, reason: "too_large"}, keeping only its id and cwd (each at most 4 KiB) and no title, the way a record too large for a history page goes. Rows keep their tools.
  • _acpmux/sessions cursors now hash the peer with the id, and ties sort by peer (local first) in both sorts (sort_session_summaries, Rows::finish), so a chat here and its copy on a peer each have their own cursor. Cursors of this daemon's own rows are unchanged; an old cursor for a peer's row falls back to the stamp and may repeat, never skip.

Performance

Synthetic fleet: 25 peers × 210 mirrored chats = 5250 rows, each with a 3.3 KB tool description (the incident's mean row size with tools).

metric before after delta harness
session/list replies to read the whole list 1, refused (message.too_large) 2 pages +1 reply, list readable server/acp_list_tests.rs a_fleet_sized_acp_list_goes_in_pages_under_the_message_limit (red, green)
rows read 0 5250 +5250 same
bytes of the list 20,559,292 in one reply, refused (the client got a ~200 B error) 20,559,390 over 2 pages, each ≤ 14 MiB +98 B (one cursor) same (PERF acp_list rows=5250 bytes=20559390 pages=2)
page 1 time (the first session/list of an oversized list) 278-284 ms 146-149 ms -48% run 41064470951: acp_list_tests::a_fleet_sized_acp_list_goes_in_pages_under_the_message_limit, release, 3 runs each, the same runner and test file; before = this PR at 19689e4 (whole list parsed per page), after = 02632c2
page 2 time 53 ms 30 ms -43% same run

Test

cargo test -p acpmux --lib acp_list_tests:

  • a_fleet_sized_acp_list_goes_in_pages_under_the_message_limit: the 5250-row state above; following nextCursor reads every row once, with tools, and no reply line is over MAX_MESSAGE_BYTES.

  • an_acp_list_that_fits_is_one_reply_with_no_cursor: 40 rows, one reply, no nextCursor.

  • byte_pages_read_each_row_once (hub/session_pages.rs): a chat here, its copies on two peers and a row listed twice, at five page budgets: every row comes once (red without the dedupe).

  • byte_pages_hold_their_place_while_sessions_update_and_leave: one row a page; between pages an unread row becomes the newest, then a read and an unread row leave: every remaining row comes once.

  • a_byte_page_cursor_stays_small_for_a_huge_id: rows with 1 MiB ids; the cursor is 33 bytes (red with the ids in the cursor).

  • copies_of_a_chat_on_peers_page_by_their_own_cursors and the extended streamed_rows_preserve_all_fields_and_stable_ties (hub/list_guard.rs): the _acpmux/sessions cursor and both sorts tell a chat's peer copies apart (each red without its change).

  • an_acp_row_larger_than_its_page_goes_as_a_stub (server/list_reply.rs): a row over its budget, and one with a 1 MB cwd and title, go as stubs under 400 B with no title.

  • Red on the test commit 909561f (run 41026359386): page 0: message too large to send: 20559292 bytes, the limit is 15728640

  • Green on the head 02632c2 (run 41060184119, tested tree b7e0a5a9 = the head's tree): the tests above pass; cargo clippy -p acpmux --all-targets -- -D warnings is clean; in cargo test -p acpmux everything passes except three: hermetic_daemon_guard (every_test_binary_runs_in_the_hermetic_environment), which fails the same way on base before acpmux: a bare session name or id prefix reaches the one live session on any host; several are refused with each HOST/NAME command #28000 and at the tip 7f97b84; and two that passed 5 of 5 when rerun alone at the same tree (run 41062160697), in code this PR does not touch: daemon::state_file_tests::managed_unit_start_avoids_a_symlinked_default_home and held_queue::agent_messages_joined_to_a_send_whose_agent_cannot_start_are_not_held (load flakes). The job's exit 1 is forced so the log prints.

Review

Independent Codex (gpt-6.1-sol, high), merge-base diff: SHIP at 02632c2, no findings, after four BLOCKs fixed in this PR: rows that share a cursor straddling a page edge; a chat's peer copies sharing a cursor; newest-first pages skipping a row that updates between pages; an unbounded cursor and whole-list work per page.

Observed / Not run

Observed: the red and green runs above, on hosted Linux through cmux-ci test, and the release timings. Read, not run: ACP clients in the wild; an ACP client that ignores nextCursor now reads the first 14 MiB of a list it could not read at all before.

What else grows with the fleet (audit of the other list replies)

  • _acpmux/sessions / _acpmux/watch without limit answer whole: with tools that is the same 17 MB (acpmux ls --tools, a non-link watch {withTools: true}); without tools about 2.3 KB per row, so about 6,900 rows at the incident's mix. A filtered _acpmux/sessions without limit (status: closed, excludeStatus) also answers whole. Paging these needs their clients to follow nextCursor; a follow-up.
  • _acpmux/wait with no sessions waits on every running row and answers full rows twice; about 3,000 running sessions fleet-wide to reach the limit.
  • Bounded already: _acpmux/peers, _acpmux/peer_status (problems capped), _acpmux/models (about 1 MB at 26 daemons), _acpmux/chats (MAX_LIMIT 5000 × ~700 B), mailbox, team inbox, history pages (MAX_PAGE_BYTES), peer snapshots, workflows, forum, bg tasks, composers. No notification carries a fleet-wide list.

Changelog

  • Fixed: ACP session/list on a machine that mirrors many peers' chats answers in pages instead of failing as too large.

Agent session: agent session

🤖 Generated with Claude Code

…message limit

Red on base: 5250 mirrored rows with tools are one 17 MB reply, refused as
message.too_large, as one daemon's 5241 were on 2026-10-30.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c11d09c9f41988fc1d93
…ge to send

session/list answered every session, local and mirrored, with tools, in
one reply: one daemon's 5241 rows were 17 MB on 2026-10-30, over the
15 MiB message limit, so it was refused as message.too_large. A list
over 14 MiB now goes in pages of at most that, each with the nextCursor
ACP's client follows (the session list's keyset cursor, so a row that
moves between pages neither repeats nor skips another), and cursor is
read. A list that fits is one reply as before, with no nextCursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c11d09c9f41988fc1d93
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 4 test failures on Blacksmith runners:

Failures

Test View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/
test_download_prints_the_active_profile_only
View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/
Security has no stub
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/
CoreFoundation, missing 1: ['_CFRetain']
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/
CoreFoundation, missing 1: ['_log2']
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@github-actions

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: 3 commit(s) change the sidebar's drawing code without a Sidebar-Look: trailer or an entry in plans/cmux-next/sidebar-owner.md is not red on the base 4154db0; do not merge until green; see lane-rules

…versized row goes as a stub

Review of 669f622: a peer's page was sized before masking, which can
lengthen text; a row larger than a page alone was still sent, page and
all over the limit; and two rows with one id and updatedAt share a cursor,
so a one-row page of the second came forever. A masked page over the
budget is cut again at half the budget (down to 64 KiB), a row larger than
the page goes as a stub with its size (as an oversized record does,
hub/paging.rs), and a cursor the page cannot get past is refused
(list.cursor_stuck) instead of answered again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c11d09c9f41988fc1d93
…CP row's stub keeps no large or masked text

byte_page ends a page before rows a cursor cannot tell apart (one id, one
updatedAt) or, when they are all it holds, after all of them, and a cursor
starts after every such row: each comes once, so list.cursor_stuck is gone.
The stub for a row larger than its page keeps only its id and cwd, each at
most 4 KiB, and no title, so it stays small whatever field was large and
carries no text a peer's reply masks.

Tests: byte_pages_read_rows_one_cursor_names_once (four budgets, the pair
on and across a page edge) and an_acp_row_larger_than_its_page_goes_as_a_stub
(a huge title and cwd); cargo test -p acpmux --lib passes them locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c11d09c9f41988fc1d93
teamleaderleo added a commit that referenced this pull request Oct 30, 2026
…ne/acpmux-sessions-pages

Takes #28161's page cutting (stub rows, cursor groups, re-cut of masked pages) under
this branch's LIST_PAGE_BYTES name, and moves the re-cut into within_limit, which the
_acpmux/sessions pages now use too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c11d09c9f41988fc1d93
Rows with one id and one updatedAt are told apart by their order: a page
that ends on the n-th of them names it <cursor>.<n>, and the next page
starts after it. A page no longer has to hold the whole group, which could
pass the budget (two 7.5 MiB rows on one page).

Test: byte_pages_read_rows_one_cursor_names_once now checks that no page
over its budget holds more than one row (red on the group code at budget 5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c11d09c9f41988fc1d93
A chat here and its copies on peers share an id and an updatedAt. The
cursor told them apart only by their order, so a page ending on the
second of three skipped the third once the first left. The cursor key
now hashes the peer too, and ties sort by peer, so each copy has its
own cursor. This daemon's own rows keep the key they had.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c11d09c9f41988fc1d93
@teamleaderleo

teamleaderleo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review (the owner lane's independent review, at its land request): SHIP at 02632c2

Recorded by lane-gh-broker from this PR's owner lane, which asked to land this exact head after its independent review. next-autoland lands it once every red is the base's; a BLOCK at this head, a HOLD line or a hold label still stops it.

…-acp-list-pages

# Conflicts:
#	cmux-tui/crates/acpmux/src/server/mod.rs
#	cmux-tui/crates/acpmux/src/server/requests.rs
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Red classification at 8c35a0a (merge-base 7deba73). This PR changes only cmux-tui/crates/acpmux/src/{hub/list_guard.rs, hub/session_pages.rs, server/acp_list_tests.rs, server/list_reply.rs, server/mod.rs, server/requests.rs}. Every red is base's.

Check Failing step / tests at this head Base Verdict
protocol contract (job) check-resource-api-boundary.py: 5 diagnostics, all crates/cmux-tui/src/cli.rs:767-771 boundary.surface (list-surfaces, list-pane-surfaces, new-surface, rename-surface, close-surface) the same 5 diagnostics, same lines, on base at #28248 (inventory job), which is in this PR's merge-base; fixed on base by #28349 (3736365), after it base red (cmux-tui cli.rs, untouched here)
inventory (job) the same script, the same 5 diagnostics same as above base red
tooling-tests (job) validate_test_execution_registry.py: tests/test_cmux_tui_owned_mini_caches.py: linux-guard lane is not run by any workflow; plus FAIL stubs: CoreFoundation.tbd ... ['_log2'], ... ['_CFRetain'], Security.framework ... has no stub identical step and identical lines on base 3736365 (job) and its neighbours a6ca7ea, 6ce1bb3, edd2126 base red (test_reds.py classify gives "not a test red: the log names no failing test", so compared by step and line)
link (job) PATCH /repos/manaflow-ai/cmux/issues/comments/6518669209: HTTP 403 ... API rate limit exceeded for installation (4 tries) n/a infra (installation rate limit), not a test; not rerun
nightly-guards / release-notary, Guard status known known base reds base red

Review: Sol (gpt-6.1-sol, high) SHIP at 02632c2 (no findings), and SHIP on the base merge at this head (resolutions keep base's session_open::load/resume and key_route_tests; acp_list paging unchanged). The full acpmux cmux-ci run at this head could not dispatch (the dispatch gave no run, twice); at 02632c2, whose own code is identical, it was green apart from hermetic_daemon_guard (base red) and two load flakes that each passed 5/5 alone (run 41060184119, rerun 41062160697).

@teamleaderleo

teamleaderleo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review (the owner lane's independent review, at its land request): SHIP at 8c35a0a

Recorded by lane-gh-broker from this PR's owner lane, which asked to land this exact head after its independent review. next-autoland lands it once every red is the base's; a BLOCK at this head, a HOLD line or a hold label still stops it.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at 8c35a0a

No blocking findings.

@teamleaderleo
teamleaderleo merged commit 4543835 into feat-cmux-next Oct 30, 2026
171 of 178 checks passed
@teamleaderleo
teamleaderleo deleted the lane/acpmux-acp-list-pages branch October 30, 2026 15:11
teamleaderleo added a commit that referenced this pull request Oct 30, 2026
…hits from #28161)

#28161 (4543835) landed while this PR was in review:
- hub/session_pages.rs: byte_page's sort key reads each key with get. Every
  index comes from enumerate, so the order is unchanged.
- server/list_reply.rs: sessions and updatedAt are read with get (a missing
  value is Null, as before), and nextCursor is written via JsonSet::set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d9e7e9f7fb2b87a57d5f
lawrencecchen added a commit that referenced this pull request Oct 30, 2026
… add without overflow (panic ratchet)

The panic ratchet's tracked arithmetic_side_effects count for acpmux grew past
its baseline 1430: three adds in hub/session_pages.rs (4543835, #28161)
and two in daemon/unused_exit.rs (55df640, #28353). They saturate now;
no value they hold can reach usize::MAX or u64::MAX, so behavior is the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lawrencecchen

Copy link
Copy Markdown
Contributor

FYI: hub/session_pages.rs added 3 arithmetic_side_effects hits (acpmux panic ratchet over its baseline). #28418 makes them saturating.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

teamleaderleo added a commit that referenced this pull request Oct 30, 2026
…sor tells a chat's peer copies apart)

Tests of a cursor a chat and its peer copy share read a daemon from
before #28161 (a fake whose cursor names only id and time), as the
fleet's daemons are today.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-4388606a06c8d5c111f8
teamleaderleo added a commit that referenced this pull request Oct 30, 2026
…ing_slice 97 > 94): rewrite the new hits without panics (#28386)

* acpmux: inbox, send and outbox status JSON writes without indexing

Part of the acpmux panic ratchet base red that came back after #28134
(indexing_slicing 703 > 693, string_slice 97 > 94). These writes go through
JsonSet::set instead of IndexMut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d9e7e9f7fb2b87a57d5f

* acpmux: ls globs, dotted fields, inbox threads and match counts without indexing (10 hits from #28012, #27686, #27791, #27970)

- cli/select.rs (911f25f, #28012): field() cuts the dotted path with
  str::get; glob() compares and tails with slice get. Every index was in
  bounds, so the output is unchanged.
- hub/inbox.rs (cbe8e60, #27686): first_pr reads the number after the
  marker with str::get. inbox_unsend matches on (index, noticed, notice is
  none), with the same three outcomes in the same order.
- hub/mod.rs (fe3152a, #27791): local_match_counts bumps its slot with
  get_mut.
- hub/queue.rs (46ed428, #27970): the transfer id is read with get.

Tests add multibyte names, keys and PR markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d9e7e9f7fb2b87a57d5f

* acpmux: ACP session/list paging without indexing (5 indexing_slicing hits from #28161)

#28161 (4543835) landed while this PR was in review:
- hub/session_pages.rs: byte_page's sort key reads each key with get. Every
  index comes from enumerate, so the order is unchanged.
- server/list_reply.rs: sessions and updatedAt are read with get (a missing
  value is Null, as before), and nextCursor is written via JsonSet::set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d9e7e9f7fb2b87a57d5f

* panic ratchet: lower acpmux to the measured counts (indexing_slicing 731 -> 686, string_slice 97 -> 94, unwrap_used 144 -> 143)

#28169 raised the acpmux baseline to the counts base had then, so the hits
fixed here could come back unnoticed. Measured at fdb6218 (fast-checks run
41080915558). PR CI now runs the ratchet too (#28169), so a PR that adds a
hit is caught before it lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d9e7e9f7fb2b87a57d5f

* acpmux: read ls row tags with get (the hit #28369 added to the panic ratchet)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d9e7e9f7fb2b87a57d5f

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lawrencecchen added a commit that referenced this pull request Oct 31, 2026
…back under its baseline (#28418)

* fix(acpmux): daemon_memory's start_logged is Linux-only like its one caller

#27991 (941b7a4) moved Daemon::start onto start_inner, leaving
start_logged used only by the Linux-only retained_session_list_allocation_benchmark,
so clippy -D warnings on macOS failed with 'function start_logged is never used'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(acpmux): the session page budget and the unused-exit client count add without overflow (panic ratchet)

The panic ratchet's tracked arithmetic_side_effects count for acpmux grew past
its baseline 1430: three adds in hub/session_pages.rs (4543835, #28161)
and two in daemon/unused_exit.rs (55df640, #28353). They saturate now;
no value they hold can reach usize::MAX or u64::MAX, so behavior is the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

3 inactive deployments
artifacts — 8c35a0a1 Deployed Oct 30, 2026 by teamleaderleo via add the Windows daemon to the tree #83519
sentry-dsyms — 8c35a0a1 Deployed Oct 30, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #83519
ci-routing-agent — 8c35a0a1 Deployed Oct 30, 2026 by teamleaderleo via cmux-tui tree preflight #83519
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants