Skip to content

acpmux: every session list page and session info fits one message (ls never hits message.too_large) - #28831

Merged
cmux-bot[bot] merged 6 commits into
feat-cmux-nextfrom
lane/ls-whole-cap
Oct 31, 2026
Merged

cmux-bot[bot] merged 6 commits into
feat-cmux-nextfrom
lane/ls-whole-cap

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

On a fleet host, plain acpmux ls fails: acpmux: runtime: message too large to send: 16927852 bytes, the limit is 15728640. No one can list that host's sessions, and the census and stall reconciler can't see it either. The CLI only retries with smaller pages, so the daemon has to send every list reply whole and under the cap.

Where the daemon still built a reply too large to send:

The fix fits every reply as a whole and never refuses one:

  • New hub/reply_fit.rs holds the shared rule and bound. shrink_largest replaces the largest parts first until the value fits. REPLY_BYTES is 1 MiB under rpc::MAX_MESSAGE_BYTES. fit_object trims a row's largest fields: a string keeps a 1 KiB preview, any other value is dropped. It never trims the fields a row is found and opened by (sessionId, id, qualifiedSessionId, sessionReceipt, peer, status, harness, cwd, updatedAt, name; the same set acpmux: a chat whose records together pass one message still opens (attach reply fits) #28754's attach reply keeps), and it marks the row omitted: {bytes, reason: "too_large", fields: [...]}. LIST_PAGE_BYTES is now REPLY_BYTES.
  • Snapshot::page (the limit pages and census pages) holds at most limit rows and at most REPLY_BYTES bytes. A page cut short by size returns the nextCursor of its last row (k…, or s:<id> for a census), and clients already follow it. withTools pages use the same path. They no longer parse a copy of the whole list.
  • A row too large for a page by itself is trimmed, on the byte pages too. Only a row whose kept fields alone are that large still goes as the id stub.
  • _acpmux/info runs its detail through fit_reply_object.
  • A trimmed row still reaches a peer masked (peer_text.rs). lastSeq is a kept field, since masking recognizes a summary by sessionId, harness and lastSeq. A preview is masked over 4 KiB past its cut and only then cut, the way views.rs masked_cut treats titles, so a secret split by the cut is never left as a head.

#28754 adds the same kind of fit for the chat-open reply (_acpmux/attach). The two are built on one rule: paging::fit_reply there is shrink_largest over /events/N/msg and /turnOpener/msg. Whichever PR lands second switches to the shared helper and the single bound.

Related: #28182, #28373, #28161, #28105 (the earlier paging work).

Testing

server/list_fit_tests.rs (new) builds a hub with 1,000 peer rows of 20 KB plus one 16 MiB row, and a local chat with a 16 MiB title. The list as a whole is past the cap, and so is one row alone. Through a real server connection:

  • _acpmux/sessions with {limit: 1000, usage: true}, {limit: 1000, withTools: true}, {} and a census (orderBy): following nextCursor to the end lists every id once, and every reply line is under MAX_MESSAGE_BYTES. The large row keeps its id, peer, status and cwd, its planStep becomes a preview, and it is marked omitted.fields = ["planStep"]. Small rows are unchanged.
  • _acpmux/info on the 16 MiB-title chat answers under the cap, with a title preview and omitted.fields naming title.
  • Over a peer connection (masked), an oversized peer row whose lastPrompt holds one token near its start and one split at the 1 KiB cut keeps lastSeq, and neither token's start appears in the preview (a_trimmed_row_still_goes_to_a_peer_masked). With reply_fit.rs as of 468e477 this test fails: the split token leaked as ghp_CutkQ9mX7pL2vB8n….

Command: cargo test -p acpmux --lib -- list_fit (in cmux-tui/).

  • Base (test commit 2b4c96c alone): both fail. The first page fails with {"limit":1000,"usage":true} page 0: message too large to send: 37190897 bytes, and info fails with message too large to send: 16779433 bytes.
  • Head: all pass. With -- list_fit reply_fit list_guard list_reply acp_list session_pages ls_pages native_history peer_text: 59 passed, 0 failed. cargo clippy -p acpmux --lib --tests -- -D warnings is clean.

Observed / Not run

  • Observed: the red/green above, run locally on a Linux lane VM (cargo test -p acpmux --lib with a filter). Panic ratchet (rust_panic_ratchet.py, -p acpmux): arithmetic_side_effects and cast_possible_truncation report above baseline, but none of the hits in the changed files are on lines this PR adds (they are existing base code).
  • Not run: the full acpmux suite here (CI runs it), and a live acpmux ls on the failing host. That host runs an acpmux older than acpmux ls: read the session list in pages (never one reply), --cursor, titles at most 200 chars #28373 (its error is the raw daemon message, which the paged CLI would have caught and retried smaller), so it needs an update that includes this change first.

Changelog

Fixed: acpmux ls and acpmux session info work on a host whose session list, or one session, is larger than one message; an oversized field shows as a preview and is marked omitted.

Proof

No UI: daemon reply sizes only. The tests above show it.

metric before after delta harness
first _acpmux/sessions {limit:1000, usage} reply, 1,001 rows of 20 KB plus one 16 MiB row refused (37,190,897 bytes) ≤ 14 MiB page with nextCursor listed instead of refused server/list_fit_tests.rs
_acpmux/info of a chat with a 16 MiB title refused (16,779,433 bytes) under 15 MiB, title as a 1 KiB preview answered instead of refused server/list_fit_tests.rs

Agent session: agent session

🤖 Generated with Claude Code

teamleaderleo and others added 3 commits October 31, 2026 02:50
…limit still go

A list whose rows together, and one row alone, pass rpc::MAX_MESSAGE_BYTES:
every _acpmux/sessions read (limit pages, with tools, no limit, a census)
lists every id with each reply under the limit, and _acpmux/info answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-741a93e81636ef5970d4
`acpmux ls` on a fleet host failed with "message too large to send:
16927852 bytes, the limit is 15728640". A page asked for by `limit` (ls
reads pages of 1000, with usage) was cut by count alone, and a row of any
size went whole into it; `_acpmux/info` sent a detail of any size. A
reply refused as too large is refused again on every retry.

- hub/reply_fit.rs: one fit rule (shrink_largest: replace the largest
  parts first until it fits) and REPLY_BYTES, a MiB under the message
  limit. fit_object trims a row's largest fields (a string keeps a 1 KiB
  preview), never its id, peer, status, updatedAt or name, and marks them:
  omitted {bytes, reason: too_large, fields}.
- Snapshot::page (limit pages and the census's orderBy pages) holds at most
  limit rows and at most REPLY_BYTES; a page cut short by bytes hands on
  the cursor of its last row, so a client reads the rest. withTools pages
  go the same way, no longer through a whole parsed copy of the list.
- A row too large for a page alone is trimmed (row_text), on the byte pages
  too; only a kept field that large still goes as the id stub.
- _acpmux/info's detail is fitted (fit_reply_object).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-741a93e81636ef5970d4
…, harness, cwd)

The same set #28754's attach reply never trims, so both fits keep one set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-741a93e81636ef5970d4
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review d2e2428: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28831 -f ref=lane/ls-whole-cap -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)/
test_a_picked_signing_label_has_a_rerun_alternative (main.RunsOnFallbackTests.test_
a_picked_signing_label_has_a_rerun_alternative)
View Logs
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)/
test_no_runs_on_chain_ends_on_the_reserved_side_label (main.RunsOnFallbackTests.tes
t_no_runs_on_chain_ends_on_the_reserved_side_label)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Review (gpt-6-astra) found that a fitted row could reach a peer unmasked:
peer_text masks a summary only when it has sessionId, harness and lastSeq,
and lastSeq was not a kept field. And a preview cut at 1 KiB could split a
secret into a head the scanner no longer knows: the new peer-path test
read ghp_CutkQ9mX7pL2vB8n... at 468e477.

- lastSeq is kept.
- A preview is masked over 4 KiB past its cut, then cut, as a lane's
  title is (views.rs masked_cut).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-741a93e81636ef5970d4
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high fast, router-review): SHIP at d2e2428. Follow-ups: none. (The earlier BLOCK at 468e477, a trimmed row losing peer masking, is fixed in d2e2428 with a red/green peer-path test.)

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Independent review (gpt-6.1-sol high) at 9f40ed4: BLOCK.

  1. P1 (BLOCKS), cmux-tui/crates/acpmux/src/server/list_reply.rs:116-129: an unfiltered, non-withTools request ({}, including the test case) still takes the final list.whole path because whole(params) is true. It never calls Snapshot::bounded, so a large list is sent whole and send_serialized returns message.too_large; the default session list still breaks the PR's contract.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol xhigh, next-review): SHIP at 951dd10

SHIP at 951dd10

Visual verdict: no images; Leo tries the build

No correctness findings. Rust tests were not run because Cargo and rustc are unavailable in this box.

Follow-ups: none

@teamleaderleo teamleaderleo removed the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 951dd10: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@cmux-bot
cmux-bot Bot merged commit 8c35668 into feat-cmux-next Oct 31, 2026
241 of 242 checks passed
@cmux-bot
cmux-bot Bot deleted the lane/ls-whole-cap branch October 31, 2026 19:30

This branch was previously deployed

3 inactive deployments
artifacts — 951dd108 Deployed Oct 31, 2026 by teamleaderleo via add the Windows daemon to the tree #91035
ci-routing-agent — 951dd108 Deployed Oct 31, 2026 by github-actions[bot] via cmux-next macOS placement #55223
sentry-dsyms — 951dd108 Deployed Oct 31, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #91035
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant