Skip to content

idle-chats bench: name the shown renderer's idle work, and count a footprint read as 0 as unread - #34155

Merged
teamleaderleo merged 11 commits into
feat-cmux-nextfrom
lane/agent-pane-idle-cpu
Nov 3, 2026
Merged

teamleaderleo merged 11 commits into
feat-cmux-nextfrom
lane/agent-pane-idle-cpu

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Stop condition (perf coordinator): "cmux NIGHTLY idles under 5% CPU per process with about 40 chats open, measured with the shared perf harness". Data: manaflow-ai/cmuxterm-hq#6338 (comment 6604253564).

The shared idle bench (perf-idle-power.sh, idle-chats, display worker) read the shown agent page's renderer at 5.27% CPU at c004201, and the tree's footprint at 7.3 GB before and 13.8 GB after. This PR gives the bench what it needed to name both, and fixes the footprint read that made the memory jump look real.

What the bench now records (after the measured window, so the window is unchanged)

  • idle_probe: for 3.5 s, what the shown chat's page asks for, by call site: animation frames, timeouts, intervals, DOM mutations, running animations, long animation frames (debug.agent_pane idle_probe, webviews/src/agent-session/acpmux/idleProbe.ts). The wrappers are restored when the window ends.
  • idle_profile: for each shown cmux page, Chromium's page metrics per second (task, script, style, layout) and a 10 s V8 CPU profile by function, through debug.cef_page_devtools {browser, method, params} (new, bench and DEBUG builds only, cmux's own pages only; browser tabs are refused). debug.cef_pages now also gives a cmux page's path.
  • native_samples: sample of each renderer over 1% CPU (main-thread call tree and heaviest frames).
  • Footprint: a live process never has 0 MB. A 0 is a failed read, from a renderer that exits during the window, so it is now None and counted in footprint_unread. Before this change it was summed as 0.

What it found

  1. No periodic work in the shown pane. Same harness on four display workers: the shown chat's page makes 0 animation frames, 0 timeouts, 0 intervals and runs 0 animations in 3.5 s, with at most one timestamp mutation a minute. Every shown cmux page reads 0 ms of Blink tasks in 10 s and an idle V8 profile, both just before and just after the window. A native sample of the busy renderer after the window shows its main thread waiting in mach_msg for all 8 s. The 2-5% the window reads is work right after the 40-chat load that stops on its own: at settle=420 instead of 60, the shown agent renderer is at 0.05%. So there is no loop to stop in the pane code.
  2. The memory jump is not a regression. In the 6500d46 runs, 24 of 46 (one host) and 16 of 47 (another host) renderers read 0 MB and no main thread. These renderers exited inside the window, and they were summed as 0. At c004201 the same exits happen after the window. On the same host, 6500d46 read 9.2 GB with 16 renderers unread, and c004201 read 14.3 GB with none unread. With the unread ones at the mean of the rest, 6500d46 is about 13 GB. At settle=420, once acpmux has hibernated the idle agents (2 min) and their hidden chat pages have gone, the whole tree is at 2.6 GB.

Performance

Same harness, perf-idle-power.sh --arg=scenarios=idle-chats --arg=config=bench --display, 40 local chats, 120 s window, two Apple M4 Pro (Mac16,11) display workers, at ef35ed9 (this branch plus a diagnostic profile before the window, since dropped):

metric settle 60 s settle 420 s delta harness
shown agent renderer CPU % 2.37 0.05 -2.32 idle-chats, display
busiest process CPU % 2.83 (app) 0.24 (app) -2.59 same
total CPU % (all processes) 18.32 0.55 -17.77 same
tree footprint MB 14158 2593 -11565 same
processes 101 23 -78 same
renderer footprints unread 0 0 0 same

And the footprint read on the 6500d46/c004201 pair (one host, before this fix):

metric 6500d46 c004201 harness
renderers that read 0 MB (exited inside the window) 16 of 47 0 of 47 idle-chats, display, one host
tree footprint as summed 9152 MB 14260 MB same
mean footprint of renderers that read 230 MB 259 MB same

Stop and cleanup: no timer, observer or loop is added to the app. idleProbe restores requestAnimationFrame, setTimeout and setInterval, disconnects its MutationObserver and PerformanceObserver when its window ends (at most 30 s), and runs only when the debug socket asks. The bench turns off the Profiler and Performance domains after its window.

Red/green

python3 -m unittest scripts/cmux-next/tests/test_bench_power_loaded.py (macOS; on Linux with libproc stubbed):

  • test_footprint_read_as_zero_is_unread_not_zero: at base AssertionError: 0.0 is not None; passes at head.
  • test_profile_summary_self_time_by_function: at base AttributeError: module 'bench_power' has no attribute 'profile_summary'; passes at head.
    All 12 tests in the file pass at head.

bun test test/agent-pane-idle-probe.test.ts (webviews): "a second probe while one runs is refused and the first still restores" fails on the probe before the review fix (the second probe ran and returned a result) and passes at head; "a probe counts timeouts by call site and restores the page's functions" passes.

Review follow-ups from the first Codex receipt, fixed here: a probe started while another runs is refused (it used to keep the first one's wrappers installed), and debug.cef_page_devtools reads the browser id with Int32(exactly:) instead of trapping.

Risk: low. The Swift adds two debug socket methods under #if DEBUG || CMUX_BENCH_HOOKS (debug.cef_page_devtools, debug.agent_pane idle_probe) and a path field in debug.cef_pages, given only for cmux-page:// pages. Shipped builds have none of them. The bench spends about 35 s more after its window. A Release build's page includes idleProbe.ts, but nothing calls it.

Observed / Not run

  • Ran: the idle-chats bench at this branch's intermediate heads on four display workers (probe, profile and sample runs), settle=420 at the final tree, the bench unit tests (red at base, green at head), CmuxNext compile checks at 687f2e7 and the final head, scripts/web-lint.sh (the only failure is check:design-tokens on chief-web.css, which is red on base b0b87e8 too).
  • Not run: a settle=420 pair at 6500d46 and c004201 on one host (queued; numbers go on Align cmux Codex CLI hook setup with current event surface #6338). A NIGHTLY with Leo's real chats: his 95% readings need long transcripts, remote peers and browser tabs, which this load lacks.

Changelog

  • none

Agent session: agent session

🤖 Generated with Claude Code

teamleaderleo and others added 8 commits November 3, 2026 13:02
…e, by call site

The idle-chats bench reads the shown agent page's renderer at 5.27% CPU
with nothing streaming. debug.agent_pane idle_probe counts that page's
animation frames, timeouts, intervals, DOM mutations, running animations
and long animation frames by call site for a window; bench_power.py
idle-chats runs it after the measured window and keeps it in the JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
… unread

The page host's script call ends at 5 s, so a 10 s probe timed out.
A display run read 0 MB for 24 of 46 live renderers and summed them
as 0: the role total said 7.3 GB where the same host now reads 12.9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
…t idle

debug.agent_pane devtools runs a DevTools method on the chat's Chromium
page. The idle-chats bench uses it after the measured window: Performance
metrics per second and a 10 s V8 profile by function, so the renderer's
idle main-thread time has a name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
The focused chat read 0 ms of tasks in 10 s while a shown agent renderer
sat at 2.6%: the busy page is another one. debug.cef_page_devtools runs
a DevTools method on one of cmux's own pages by browser id, and
debug.cef_pages gives a cmux page's path, so the bench profiles each
shown page in one window and names it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
Every shown cmux page read 0 ms of tasks and an idle V8 profile in 10 s
while renderer-client-id=6's main thread still took 2.5%: the work is
native. The bench samples each renderer over 1% for 8 s after the window
and keeps its main thread's call tree and heaviest frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
…efore the window

The native samples after the window found every busy renderer's main
thread waiting: the idle cost may be work that ends by then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
@teamleaderleo teamleaderleo added the risk:low Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@github-actions github-actions Bot added risk:normal Risk tier (docs/ci/risk-tiers.md) and removed risk:low Risk tier (docs/ci/risk-tiers.md) labels Nov 3, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 7dd177d

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 3. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 7dd177d
Visual verdict: no images; Leo tries the build

  1. [P2] [follow-up] webviews/src/agent-session/acpmux/idleProbe.ts:107: Overlapping probes permanently leave wrappers installed. The second probe saves the first probe’s wrappers, then restores them after the first finishes. Reproduced with overlapping 100/200 ms probes. Serialize probes or reject overlapping calls so stack capture and retained maps stop afterward.

  2. [P2] [follow-up] Packages/macOS/CmuxNext/Sources/CmuxNextApp/Control/DebugCEFPages.swift:31: Int32(browser) traps for valid JSON integers outside Int32’s range, crashing debug/bench builds when a page exists. Validate with Int32(exactly:) and return an input error.

  3. [P2] [follow-up] review/images/selection.json:2: No after images are available; complete the visual check in the build.

All 12 bench tests pass with macOS libraries stubbed. Native compilation was unavailable.

Follow-ups:

  • Guard overlapping idle probes and verify restoration.
  • Reject browser IDs outside Int32’s range.
  • Leo checks the build visually.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 7dd177d: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

The idle probe loads on demand like the other debug fixtures (#33740).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review a217cf2: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr34155 -f ref=lane/agent-pane-idle-cpu -f platform=macos -f backend_mode=local

Gallery before/after for a217cf2: 16 states rendered, 0 images differ, 1271 more touched states not rendered. Images are in the run's artifact, not on this public PR; the independent review of this head reads them on its own, so the PR needs no hand-made screenshots of these states.

gh run download 41710422935 -R manaflow-ai/cmux -n gallery-advisory-34155-a217cf242eee700af01bb2298622916437a53c09 (then read gallery-review.json)

States
State Theme Status Diff Picked by
agent-pane.chat-width/standard Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.chat-width/standard Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.chat-width/wide Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.chat-width/wide Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.chat-width/wide-blocks Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.chat-width/wide-blocks Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-edge/short-edge Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-edge/short-edge Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-edge/wide-edge Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-edge/wide-edge Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-edge/no-edge Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-edge/no-edge Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-floats/composer-grown Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-floats/composer-grown Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-rest-gap/gap-40 Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-rest-gap/gap-40 Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-rest-gap/gap-56 Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-rest-gap/gap-56 Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-rest-gap/gap-72 Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.composer-rest-gap/gap-72 Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/shipped Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/shipped Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/two-thirds Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/two-thirds Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/half-tucked Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/half-tucked Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/before Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.continuation-spacing/before Apple System Colors unchanged 0% imports debug.ts (1 hops)
agent-pane.copy-controls/output Apple System Colors Light unchanged 0% imports debug.ts (1 hops)
agent-pane.copy-controls/output Apple System Colors unchanged 0% imports debug.ts (1 hops)

Pick other states with a Gallery: entry.id, entry.id/variant line in the PR body.

Passes: CI passes on a217cf242e.

CI passes on a217cf242e (run 41710423135 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base be63d3b; does not block

Shipped in:

…32 is an input error

A probe that started while another ran saved the first one's wrappers
and put them back for good; it is now refused. debug.cef_page_devtools
read the browser id with Int32(_:), which traps past Int32's range.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cb01117327fc358e9ba1
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 77ff2e9

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 3. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 77ff2e9

Visual verdict: no images; Leo tries the build

  1. [P2] [follow-up] bench_power.py:419: Comparison tables still conceal failed footprint reads. Totals substitute zero for unread processes, while perf_rows omits footprint_unread. An unchanged 200 MB tree with one failed read compares as 100 → 200 MB, +100%. Show unread counts and flag or suppress incomplete memory comparisons.

  2. [P2] [follow-up] idleProbe.ts:52: Calling debug.agent_pane {action:"idle_probe"} without ms waits 10 seconds, exceeding Chromium’s 5-second DevTools timeout, so the default invocation returns an error. Use a default within that deadline or extend the evaluation timeout.

Follow-ups:

cmux-bot[bot]
cmux-bot Bot previously approved these changes Nov 3, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 77ff2e9: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@blacksmith-sh

blacksmith-sh Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Found 12 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp8jck3669/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpayr970ks/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
nightly still handles the legacy tunnel extension/
nightly still handles the legacy tunnel extension
View Logs
test_skipped_steps_read_no_unkeyed_decide_output (main.Workflow.test_skipped_steps_
read_no_unkeyed_decide_output) (step='- name: Build nightly app (Release)', output='not
ary_probe')/
test_skipped_steps_read_no_unkeyed_decide_output (main.Workflow.test_skipped_steps_
read_no_unkeyed_decide_output) (step='- name: Build nightly app (Release)', output='not
ary_probe')
View Logs
test_skipped_steps_read_no_unkeyed_decide_output (main.Workflow.test_skipped_steps_
read_no_unkeyed_decide_output) (step='- name: Build nightly app (Release)', output='not
ary_probe')/
test_skipped_steps_read_no_unkeyed_decide_output (main.Workflow.test_skipped_steps_
read_no_unkeyed_decide_output) (step='- name: Build nightly app (Release)', output='not
ary_probe')
View Logs
test_skipped_steps_read_no_unkeyed_decide_output (main.Workflow.test_skipped_steps_
read_no_unkeyed_decide_output) (step='- name: Build nightly app (Release)', output='pro
file_channel')/
test_skipped_steps_read_no_unkeyed_decide_output (main.Workflow.test_skipped_steps_
read_no_unkeyed_decide_output) (step='- name: Build nightly app (Release)', output='pro
file_channel')
View Logs

...and 2 more test failures. View all on Blacksmith: run 41710422941, run 41710423137

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 20:12 Inactive
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 20:12 Inactive
@cmux-bot
cmux-bot Bot dismissed their stale review November 3, 2026 20:13

A newer verdict blocks a217cf2 (no independent SHIP at the head).

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 20:13 Inactive

@teamleaderleo teamleaderleo left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (codex gpt-6-astra high, next-review, since=20261103T202138Z): SHIP at a217cf2

SHIP at a217cf2

Visual verdict: unchanged by this diff

  1. [P2] [follow-up] idleProbe.ts:52: The default 10-second probe exceeds callPageScript’s 5-second Chromium DevTools timeout. Calling debug.agent_pane {"action":"idle_probe"} therefore returns a timeout instead of diagnostics; explicit windows above 5 seconds also fail. Use a shorter default and supported maximum, or extend the bridge timeout to cover the requested window.

Follow-ups:

  • Align the probe duration and Chromium bridge timeout.

@teamleaderleo
teamleaderleo merged commit 755e54b into feat-cmux-next Nov 3, 2026
187 of 195 checks passed
@teamleaderleo
teamleaderleo deleted the lane/agent-pane-idle-cpu branch November 3, 2026 21:19
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, sweep, via the review router; full diff from merge base be63d3b to this head): BLOCK at a217cf2

[P1] [blocking] Packages/macOS/CmuxNext/Sources/CmuxNextApp/Control/DebugCEFPages.swift:34 - debug.cef_page_devtools forwards arbitrary DevTools methods to privileged cmux-page:// pages, allowing Runtime.evaluate, navigation, or network methods to read or mutate password/settings pages and bypass AgentURLPolicy - restrict this endpoint to the benchmark’s read-only Performance.*/Profiler.* methods and enforce the existing page policy.

teamleaderleo added a commit that referenced this pull request Nov 3, 2026
…t, not a sleep (#34259)

#34155 (755e54b) added webviews/test/agent-pane-idle-probe.test.ts, whose
two window.setTimeout(() => undefined, 5) calls are what idleProbe counts. The
gate read them as sleep-then-assert and turned 'Validate test determinism gate'
red on feat-cmux-next.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-d750cf10922c785efa64

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — a217cf24 Deployed Nov 3, 2026 by cmux-bot[bot] via wait for same-key tree publisher #107536
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant