Skip to content

acpmux: one serialization per streamed entry; agent host 27 -> 13 us/token (perf100) - #23051

Closed
lawrencecchen wants to merge 2 commits into
feat-cmux-nextfrom
perf100/acpmux-token-copies
Closed

lawrencecchen wants to merge 2 commits into
feat-cmux-nextfrom
perf100/acpmux-token-copies

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What

Per-token copies removed on the claude-stdio path (what Lawrence's chats use):

  • Agent host: each entry is encoded once, from a borrowed view, and kept as its frame bytes. The live send and a resume send the same bytes. Before, each entry was serialized once to weigh it, deep-copied to send it, and serialized again by the writer.
  • Message::from_value moves the fields out instead of copying params. New Message::into_value; the host builds records with it. The raw Claude line moves into its record. It is now translated first, and the raw record still comes first.
  • Message::to_line serializes from the borrowed fields. Before, to_value copied every param for each client line.
  • The preview grows in place instead of being copied on every chunk.

Numbers (rbx, perf100 bench: 10 sessions x 4,000 deltas, one attached client; base 26e79e1; two runs each)

Metric Before After Ratio
Agent host CPU per token (release) 27–28 µs 13–14 µs 2.0x
Daemon CPU per token (release) 89.5–93.5 µs 75.5–85.7 µs 1.1–1.2x
Agent host CPU per token (debug cargo test) 33–35 µs 23 µs 1.5x
Agent host RSS after a 20,000-delta turn (debug) 64–89 MB 36–40 MB ~2x
Log bytes / write syscalls per token 674 B / 2.35 same 1x

Most of the daemon's per-token cost is still in the hub: two records per token, the log line, the event ring's record copy, and the fan-out. That is the next slice; it needs a profiler on rbx (perf is blocked for the rbx user).

Test

tests/claude_stream_budget.rs: a recorded-shape turn of 20,000 deltas from a stand-in Claude Code (tests/fake_claude_stream.py), run through the CLI. It checks that the reply arrives whole. It also checks that the agent host uses at most 70 µs and the daemon at most 330 µs of CPU per delta, about 3x the measured debug values. It is a budget guard: on base the debug numbers (host 33–35 µs, daemon 129–136 µs) also pass.

Review

A correctness review subagent checked this change:

  • The encoded bytes are identical to write_frame(&HostFrame::Entry), and the controller decodes them.
  • from_value gives the same result for every input.
  • The raw record still comes before the translated records and the stdin taps.
  • The writer still does write_all and flush per frame.

Nits that remain open:

  • An entry over the frame limit now becomes a stderr note. If it was a response with an id, that request is not answered. Before, the controller was dropped on every resume. Input lines are capped below the limit, so this is hard to reach.
  • The buffer weight now counts the frame overhead, so the stderr drop threshold triggers slightly earlier.

Changelog

none

Observed / Not run

  • Observed on rbx: the bench, the budget test on head and on base, and cargo clippy -p acpmux --all-targets -D warnings, where the only errors are the base reds cmp_owned (cmux-tui: two base clippy reds (acpmux steer_now_tests, worktree-care store) #22906) and unnecessary_sort_by in cmux-worktree-care. The full suite runs in the landing gate.
  • Not run: macOS. The acp-fork lane's Codex bench was not rerun; the Codex bridge shares from_value and into_value.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Removes per-token copies on the claude-stdio path. The agent host encodes each streamed entry once and keeps its frame bytes, sending the same bytes live and on resume; Message::from_value and the new Message::into_value move fields instead of copying params; Message::to_line serializes borrowed fields; and the streaming preview grows in place. Agent host CPU per token drops from 27–28 to 13–14 µs (release, perf100 bench) and RSS after a 20,000-delta turn from 64–89 to 36–40 MB; daemon CPU per token drops from ~90 to ~78 µs.

Budget test

  • tests/claude_stream_budget.rs runs a recorded-shape 20,000-delta turn through a stand-in Claude Code (tests/fake_claude_stream.py) and asserts the reply arrives whole and that host/daemon CPU per delta stay under 3x the measured debug values.
  • An entry over the frame limit now becomes a stderr note instead of dropping the controller on resume, and the stderr drop threshold triggers slightly earlier because the buffer weight counts frame overhead.

Written for commit 7c1e67d. Summary will update on new commits.

View guided diff Turn on auto-fix

@lawrencecchen lawrencecchen added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@lawrencecchen
lawrencecchen temporarily deployed to cmux-tui-merge-tree October 21, 2026 06:19 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 7c1e67d: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23051 -f ref=perf100/acpmux-token-copies -f platform=macos -f backend_mode=local

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Correctness review (independent subagent, correctness first) and resolution

The review covered #23052 (purge guard and cleanup) and #23051 (per-token copies).

# Finding Severity Resolution
1 A chat that moves A → B → A is purged on A before its import, so the purge guard refused its save and records, and a restart of A lost it blocker Fixed in fda6f6e: transfer_in calls Store::release_purge(id) after it purges the moved-away copy. Red-then-green test a_chat_moved_away_and_back_survives_a_restart_of_its_first_daemon
2 LocalStore::save held the guard lock through its synced write, so every append in the daemon waited should-fix Fixed: the save checks before and after the write; a purge during the write deletes what it wrote
3 MemoryStore was not the same as LocalStore (append had no check; save checked without the lock) nit Fixed: both check under the store's own lock
4 One unreadable events dir stopped the whole startup cleanup nit Fixed: an unreadable log counts as a log, so the session is kept
5 An oversized In response now becomes a stderr note, and its request gets no answer (#23051) nit Open (hard to reach: input lines are capped below the frame limit). Follow-up
6 The buffer weight now counts the full frame (#23051) nit Accepted: the threshold triggers slightly earlier and stays harmless
7 Small copies remain (write_claude_line, error parse) nit Accepted

Checked and fine:

  • Deadlock: writers → purged in both append and purge; delete holds no lock during its retry sleep.
  • Startup order: daemon.lock is taken before load_from_store.
  • The cleanup deletes only directories with status Closed and no log bytes.
  • Fork and adopt use new ids; peers do not write to the store.
  • The guard set stays small: about 50 bytes per purge per daemon run.
  • acpmux: one serialization per streamed entry; agent host 27 -> 13 us/token (perf100) #23051: the frame bytes are identical to write_frame(&HostFrame::Entry); from_value gives the same result for every input; the order of records is kept.

Risk tier: #23052 adds a delete of user data, which is risk:high in REVIEW policy; this repo has no risk:high label. Lawrence chose the deletion (via the perf100 coordinator, 2026-10-21): "delete the 7,885 leftover records".

@teamleaderleo

Copy link
Copy Markdown
Collaborator

Bead cx-5quep created for this PR by the cmux coordinator.

@teamleaderleo

Copy link
Copy Markdown
Collaborator

cmux-next guards / defaults-snapshot is red on 24 of 24 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@cursor

cursor Bot commented Oct 21, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

…nt host and daemon)

perf100. The claude-stdio path (what Lawrence's chats use): a recorded-shape
20,000-delta turn from a stand-in Claude Code, at the CLI. Budget guard about
3x the measured debug values; on base 26e79e1 (debug, rbx) the host used
33-35 us and the daemon 129-136 us per delta.
- agent host: each entry is encoded once (EntryFrame borrows it) and kept as
  its frame bytes; the live send and a resume send those bytes. Before, the
  entry was serialized to weigh it, deep-copied to send it, and serialized
  again by the writer.
- Message::from_value moves the fields out (it copied params); new
  Message::into_value; the host builds records with it. The raw Claude line
  moves into its record (translated first; the raw record still comes first).
- Message::to_line serializes from the borrowed fields (to_value copied every
  param for each client line).
- the preview grows in place instead of being copied per chunk.

rbx, release, 10 sessions x 4,000 deltas (perf100 bench), two runs each,
base 26e79e1: agent host 27-28 -> 13-14 us/token, daemon 89.5-93.5 ->
75.5-85.7 us/token; debug cargo test: host 33-35 -> 23 us, host RSS after the
turn 64-89 -> 36-40 MB.
@lawrencecchen
lawrencecchen force-pushed the perf100/acpmux-token-copies branch from bff738e to 7c1e67d Compare October 22, 2026 00:50
@lawrencecchen
lawrencecchen temporarily deployed to cmux-tui-merge-tree October 22, 2026 00:50 — with GitHub Actions Inactive
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Rebased onto feat-cmux-next 390c286 (head 7c1e67d) with no conflicts. Note for the captain: since #23203 the hub no longer logs raw claude.stream_event lines, so the daemon's per-token numbers in the description (measured before it) are higher than on the current tip; the agent-host change (single encode per entry) is unaffected. Not re-gated after the rebase.

@blacksmith-sh

blacksmith-sh Bot commented Oct 22, 2026

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
nightly cmux-tui resolve/nightly cmux-tui resolve View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

teamleaderleo added a commit that referenced this pull request Oct 27, 2026
…ing every second (idle acpmux 63% -> 0%) (#26255)

* cmux-tui: agent chat notices page the session list and stop reconnecting every second

The notices watcher read the `_acpmux/watch` reply through a 4 MiB cap. On
cosmic-gray the unpaged reply is 14.7 MB (2786 sessions), so every read was
cut short, counted as a success, reset the backoff and reconnected about a
second later. Three cmux-tui servers doing that held the daemon at about 54%
of a core while idle.

- The watcher asks for the list 500 sessions at a time (`limit`, then
  `_acpmux/sessions` from the cursor) and reads whole lines with no cap.
- A cut line, an error reply or a list that never finished is a failure; the
  backoff resets only after a whole list on a connection that stayed up 60 s.
- A connection that ends mid-list keeps what the watch knew of the chats no
  page reached.
- acpmux builds the session page by moving the list in, instead of
  deep-copying it per reply. (The second copy, `Message::to_line` cloning the
  result, is what open #23051 removes.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a

* cmux-tui: a push before its page wins; an unreadable line or cursor is a failure

Review: acpmux turns the watch on before it reads the page, and sends pushes
apart from replies, so a push can come before an older page. A chat a push
reached now follows its pushes alone for the rest of the connection, and
pushes count from the start of the connection. A line that is not JSON or a
cursor that is not a string ends the connection as a failure, and the 60 s a
connection must stay up count from the end of its list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a

* cmux-tui: a push that finds an unknown chat needing nobody reads its notices

Review: a chat whose first sign on a connection is a push skips its page row,
so the daemon-start read the page did never ran for it. The push now does it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Closed by the perf lead (agent, cmux-ad order 2026-10-31): stale, with no owner. This is an agent PR (lawrencecchen account). It was opened on 2026-10-21, its last commit is from 2026-10-22, no person pushed or commented in the last 3 days (automated review and sweep notes do not count), no live session names it in the lead status of the last 24 h, and it conflicts with feat-cmux-next (DIRTY).

The branch perf100/acpmux-token-copies stays. To reopen: click Reopen (or gh pr reopen 23051 -R manaflow-ai/cmux), merge origin/feat-cmux-next into the branch, get a review receipt on the new head, then add ready-to-land.

teamleaderleo added a commit that referenced this pull request Nov 1, 2026
…token (takes over #23051) (#28752)

* test(acpmux): a streamed Claude turn stays within its CPU budget (agent host and daemon)

perf100. The claude-stdio path (what Lawrence's chats use): a recorded-shape
20,000-delta turn from a stand-in Claude Code, at the CLI. Budget guard about
3x the measured debug values; on base 26e79e1 (debug, rbx) the host used
33-35 us and the daemon 129-136 us per delta.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-2f016c499153c1bead46

* fix(acpmux): one serialization per streamed entry; no per-token copies

- agent host: each entry is encoded once (EntryFrame borrows it) and kept as
  its frame bytes; the live send and a resume send those bytes. Before, the
  entry was serialized to weigh it, deep-copied to send it, and serialized
  again by the writer.
- Message::from_value moves the fields out (it copied params); new
  Message::into_value; the host builds records with it. The raw Claude line
  moves into its record (translated first; the raw record still comes first).
- Message::to_line serializes from the borrowed fields (to_value copied every
  param for each client line).
- the preview grows in place instead of being copied per chunk.

rbx, release, 10 sessions x 4,000 deltas (perf100 bench), two runs each,
base 26e79e1: agent host 27-28 -> 13-14 us/token, daemon 89.5-93.5 ->
75.5-85.7 us/token; debug cargo test: host 33-35 -> 23 us, host RSS after the
turn 64-89 -> 36-40 MB.

* acpmux: the agent host's encoded-entry buffer keeps base's orphan accounting; an over-limit answer is answered

Takes over #23051 onto current feat-cmux-next:
- base's orphaned-host end (orphan.rs) counted retained entries a next daemon needs
  by looking at each Entry; the buffer now keeps encoded frames, so each retained entry
  records that flag when it is pushed (Retained.needed).
- base's claude_stdio::logged_line (workflow snapshots logged as their size) gets an
  owned form, so the raw Claude line still moves into its record instead of a copy.
- an entry over the frame limit that is an answer becomes an error answer to the same
  id, so its request is not left waiting (the open nit on #23051); anything else is
  still a stderr note.

Red/green: cargo test -p acpmux --lib an_entry_over_the_frame_limit (new), plus the
logged_line/workflow/rpc lib tests: 27 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-2f016c499153c1bead46

* acpmux: an agent host entry is never kept or sent as an empty frame

Review (Codex, BLOCK at e074c3d): an answer whose id alone is over the frame
limit made its error answer over the limit too, and the entry was kept and sent as
zero bytes; the controller's reader then waits for a header that never comes, and
the orphan count kept the host for an entry no daemon could log.

encode_entry now falls back to a one-line stderr note when the replacement does not
fit either, takes `needed` from what is kept, and numbers nothing when not even the
note fits. encode_frame_within takes the limit so the test runs with a small one.

Red/green: cargo test -p acpmux --lib an_entry_whose_replacement_is_over_the_limit_too
(new; it does not build without encode_entry), and the agent_host/rpc/logged_line lib
tests: 23 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-2f016c499153c1bead46

* acpmux: test that to_line writes the bytes to_value wrote

Red on the borrowed to_line: it wrote jsonrpc before id, and
server/list_reply.rs compares reply lines byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-2f016c499153c1bead46

* acpmux: to_line writes keys in to_value's sorted order

The borrowed to_line wrote jsonrpc first and RpcError in declaration
order; to_value().to_string() sorts keys (no preserve_order). Reply
lines written by server/list_reply.rs are compared byte for byte with
Message::ok(..).to_line(), so keep the bytes the same.

cargo test -p acpmux --lib -- rpc:: list_reply agent_host::: red on the
previous commit (line_tests and list_reply byte equality), 28 passed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-2f016c499153c1bead46

* acpmux: cover params-less messages and a bare response in the line test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-2f016c499153c1bead46

* acpmux: add no panic-ratchet hits in the encode-once path

The frame encoder and the retained weight added two
arithmetic_side_effects hits and one cast_possible_truncation hit to
acpmux; use saturating adds and u32::try_from (the body is at most
MAX_FRAME). rust_panic_ratchet counts for acpmux are now base's
(1494, 222).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-2f016c499153c1bead46

---------

Co-authored-by: lawrencecchen <54008264+lawrencecchen@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>

This branch was previously deployed

4 inactive deployments
ci-routing-agent — 7c1e67d5 Deployed Oct 22, 2026 by github-actions[bot]
artifacts — 7c1e67d5 Deployed Oct 22, 2026 by lawrencecchen
sentry-dsyms — 7c1e67d5 Deployed Oct 22, 2026 by lawrencecchen
cmux-tui-merge-tree — 7c1e67d5 Deployed Oct 22, 2026 by lawrencecchen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants