Skip to content

acpmux: start agent harnesses at background CPU class on macOS, so cmux and its renderers go first - #27600

Merged
teamleaderleo merged 7 commits into
feat-cmux-nextfrom
acpmux-agent-cpu-class
Oct 30, 2026
Merged

teamleaderleo merged 7 commits into
feat-cmux-nextfrom
acpmux-agent-cpu-class

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Leo (2026-10-28): "Likewise, holy hell, the resizing of the Chromium elements. ... It's still the exact same as it was, like, 24 hours ago, where it's just slow as hell."

Reference: cmuxterm-hq#4947 (private).

Cause, measured. The drag lag is CPU contention, not cmux code. On a fleet bench Mac with 14 busy processes standing in for agents, a Chromium page trails its dragged pane by up to 55 ms at p95. With the same load at background class (taskpolicy -b), it trails by 6-10 ms, and cmux's main thread spends about 45 % less CPU per drag. Profiling agrees: cmux's main thread sat idle in mach_msg in 44 of 51 samples. Agent harnesses, and the builds and tests they run, compete with cmux and its renderers at the same priority. acpmux lowered them only after a session passed its CPU budget.

Fix. On macOS each session's harness now starts with PRIO_DARWIN_BG, the class cmux-git already gives its git children (cmux-git/src/run.rs). The class is set in the harness between fork and exec, both for a direct child (agent.rs) and under an agent host (SpawnSpec.cpu_class, which the daemon decides). So the harness and every tool it starts inherit it. No running process is reniced.

  • Configurable (cpu_class.rs). The first source that sets it wins:
    1. A profile's env ACPMUX_SESSION_CPU_CLASS (normal or background), so one build-heavy profile can opt out.
    2. The daemon's env ACPMUX_SESSION_CPU_CLASS.
    3. Config session.cpuClass (acpmux config --set session.cpu_class=normal).
    4. Otherwise background.
  • Why background, not utility or nice. Utility, nice -n 10 and nice -n 20 gave no steady gain over the default class. On the heavy page they dropped 23-26, 6-36 and 19-32 frames per drag; background dropped 0-7. Only background keeps the agents off the cores the GUI needs.
  • No raise for the focused chat. The daemon knows which chats a client shows (hub/seen.rs). Raising one again would mean renicing its running process group, and children it already started would stay low. So every session starts at background.
  • Not touched: the CPU and memory budget enforcement, the Linux cgroup paths (Linux reads the class and ignores it), and cmux's own app and renderer processes.
  • Compatibility: a spawn spec from an older daemon has no cpu_class and starts its harness as before.

Performance

Harness: scripts/cmux-next/split-resize-bench.py through scripts/measure/split-spring.sh on a fleet bench Mac (bench/resize-harness-tip), --pages=probe,heavy --drag-hz=120 --cpu-load=14 --cpu-load-class=default|background|utility. All runs were on the 60 Hz display. The busy processes at a class stand in for agents at that class. The heavy page is shown, as the median of 4 runs per class (fleet steps listed below).

metric before (load at default class) after (load at background class) delta harness or run link
page trails pane, p95 ms (heavy page) 23.0 (runs: 55.2, 23.1, 22.9, 9.7) 7.8 (runs: 6.0, 9.7, 7.9, 7.7) -66 % split-resize, steps below
page frames dropped per drag (heavy) 13.5 (22, 14, 13, 13) 1.5 (7, 1, 2, 1) -89 % same
page frame p95 ms (heavy) 42.7 (50.3, 66.7, 33.4, 35.1) 17.7 (33.3, 16.8, 18.6, 16.8) -59 % same
cmux main-thread CPU ms per drag (heavy) 245 (250, 225, 240, 258) 134 (155, 140, 128, 128) -45 % same
busiest renderer CPU ms per drag (heavy) 448 (448, 425, 448, 460) 359 (368, 365, 353, 325) -20 % same
agent turn running a cold crate build, s (through acpmux, idle M4 Mac) 18.5 (18.4-19.3) 115.8 (113.1-125.2) +525 % bench=build-turn, cmux-ci step a898e4fd0e06d5836335fd34
the same build run directly, s (default / utility / background) 17.8 119.6 (background; utility 18.0) +572 % same step, ACPMUX-BUILD-RAW lines

The build rows come from scripts/measure/acpmux-overhead.sh bench=build-turn (#27582): Claude Code through acpmux runs one Bash call per turn, a cold debug build of cmux-history in a fresh target dir, with before= at base. The build shell's base priority was 20 before and 4 after.

Runs 5 and 6 of the same harness, heavy page, page trails p95 ms / dropped / main ms (default, nice 10, nice 20, background):

  • default: 117.3 / 49 / 260 and 9.7 / 23 / 250
  • nice 10: 8.4 / 6 / 245 and 23.4 / 36 / 258
  • nice 20: 43.0 / 32 / 240 and 10.5 / 19 / 235
  • background: 9.7 / 0 / 125 and 6.2 / 2 / 113

These are on a scratch copy of the harness that adds --cpu-load-class=nice10|nice20.

Steps (default / background), run 1: 33cca097f4eca66b198f1e00 / 6e0128696605b93a94c61c9a; run 2: 54bbedbce6966501aad74092 / 49b59f0fa56e39d5f04e59f4; run 3: a295cbda82f1726c6d8fae0a / ac79343ddd56d8e24ece5b43; run 4: 6faf28c898ae1741ad13db10 / 2d4ab680341ccb8b6e609c5d. Utility: 8da2362168f642090f9629ee, 9c51a10f9c6ffa1e99cc3bdc. Runs 5-6 (default, nice 10, nice 20, background): 55bd43c2125b46652a0d4648, b2136d2a00ac7f2d62080e85, d80f174564b92f273602f725, 2c0cf1728c3fc31bf127044c; 22430ef0f1dc008153ce0658, 80fd10b30e1725dd44c6a7df, 09e1c94720313ec88204cd55, 854796bcc2d9540f06065267.

Tests (red, then green).

  • The red commit 2b4a97f adds tests/cpu_class.rs, which is macOS only. A direct harness and a hosted one each start a background sleep, and the test reads both processes' base priority with ps -o pri= (4 is darwinbg). The two background tests failed on a fleet Mac in step 449b586ce3a30d6dfe904411.
  • At this head they pass, with the normal opt-out test, a ${env:...} reference that resolves to normal, the resolve-order unit test and the config round trip (step fcf491e6fe72a06b1b5c24dd, macOS). macOS clippy is green (0f69df34c5b3bd6abe07c1cf), and so are Linux fmt, clippy and tests (b273610db33137ec07df61c1).

Risk: medium.

  • The cost is real. Background class confines an agent's processes to the efficiency cores and throttles their disk I/O, even on an idle Mac. A cold build inside an agent turn took 6.3x as long (18.5 s to 115.8 s on an M4). Agents on interactive Macs are meant to send builds to the fleet, so most local agent work is light, but a lane that builds locally pays this.
  • The opt-out is one env key per profile (ACPMUX_SESSION_CPU_CLASS=normal, for a build-heavy lane), or one config value for the host (acpmux config --set session.cpu_class=normal).
  • macOS only. Linux and Windows spawn as before.

Agent session: agent session

🤖 Generated with Claude Code

teamleaderleo and others added 3 commits October 29, 2026 13:20
… class on macOS (red)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
Agent harnesses and the tools they run competed with cmux and its Chromium
renderers at the same priority; under load a dragged pane's page trailed it.
Each session's harness now starts with PRIO_DARWIN_BG, set between fork and
exec (direct children and agent hosts), so everything it runs inherits it.
config session.cpuClass, the daemon's ACPMUX_SESSION_CPU_CLASS and a profile's
env pick normal to opt out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review eb6d68c: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27600 -f ref=acpmux-agent-cpu-class -f platform=macos -f backend_mode=local

web and size checks: green

@blacksmith-sh

blacksmith-sh Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Found 6 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp7c6f09w5/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpca6sh9ry/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…d wait the test can release

Review: a profile's ${env:...} reference that resolves to normal was read
unresolved, so the harness started at background class anyway. The class is
now read from the command's own environment, as the harness gets it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
teamleaderleo and others added 2 commits October 29, 2026 14:40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rocess env, not set_var

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	cmux-tui/crates/acpmux/src/cli/command.rs
#	cmux-tui/crates/acpmux/src/config.rs
#	cmux-tui/crates/acpmux/src/config/tests.rs
#	cmux-tui/crates/acpmux/src/hub/catalog_reload.rs
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at eb6d68c

No blocking findings.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves eb6d68c: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo teamleaderleo left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (codex gpt-6-astra high fast, independent review): SHIP at eb6d68c

Rounds:

  • 8a0720b: BLOCK. The class was read from the unresolved profile env, and the FIFO wait could hang. Both fixed in 11bc618.
  • b549c72: BLOCK. set_var was called during tests; fixed in abd6b68.
  • abd6b68: SHIP.
  • eb6d68c: SHIP. This head only merges feat-cmux-next (#27555's task caps); the review checked the conflict resolution in config.rs, config/tests.rs, cli/command.rs and catalog_reload.rs.

Reds, compared per test against base:

  • cmux-next/compile, and the daemon tests job built on it: the one error is JoinConsent+App.swift:222:29: 'async' call in a function that does not support concurrency. The same error appears at the merge parent 1b67652 (fleet swift-check ac092aec), and #27758 fixed it on base afterwards. This PR changes no Swift.

  • release-notary test_the_repository_highlights_validate, and Guard status: the test flags release-notes highlights (chat-find-keeps-place, inline-pr-pills, inspector-error-colours) that lacked a tryIt or docs link at the merge parent. #25839 (0693908) fixed them on base. This PR changes no release notes.

  • Linux cmux-tui-rust-check all session: the same three tests fail at base 1b67652 with the same assertions:

    • transcript_events::orchestration_fields_reach_subagent_events_and_session_summary
    • send_relay_cli a_target_nobody_here_resolves_goes_to_the_relay_and_a_session_here_never_does
    • vm_child_sessions_journey session_children_journey_lists_follows_and_opens_a_vm_chats_child

    These come from fleet steps 7e11795e (head) and ef092519 (base).

Green at this head: macOS test cpu_class, 6 of 6 (fleet step dc5de660); macOS clippy (6c4c56d0); rust workspaces.

@teamleaderleo
teamleaderleo merged commit f5e762b into feat-cmux-next Oct 30, 2026
203 of 212 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40993317688 at f5e762b failed: cmux-next checks (god files, concurrency, crash safety, l10n).
Those jobs last passed at 9498e0c. This pull request is one of 4 merged since: #26997, #27604, #27819, #27600.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

teamleaderleo added a commit that referenced this pull request Oct 30, 2026
…e daemon guard (#27983)

#27600 added tests/cpu_class.rs, which starts the agent host the same way
agent_host.rs and cwd_pin.rs do (its child gets daemon_env::vars as
SpawnSpec.env), but did not list it, so hermetic_daemon_guard fails on base.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-710a4a518c009a6ad126

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lawrencecchen added a commit that referenced this pull request Oct 31, 2026
…names its harness and a background one runs once more (red)

Behavior tests through an in-process daemon's ACP client (macOS): the harness
answers session/new above background class and runs at it once started; a tool
its turn starts inherits it; an agent_start deadline error names the agent
process; a start held at background class by another process is retried once.
Red on the base: #27600 starts the harness at PRIO_DARWIN_BG before its exec.
lawrencecchen added a commit that referenced this pull request Oct 31, 2026
… its session's start is over

#27600 started every harness at PRIO_DARWIN_BG. On 2026-10-30 (load 125 on 18
cores, du and git scans of a 3.3 TB disk) a claude harness at background class
spent 90 s in trust-settings reads with under 1 s of CPU, every Chief turn
failed ("session/new: no answer within 60 s", "no harness session within 90s").

- The harness starts at its starter's class and clears an inherited
  background class at its exec (start_at_normal). Once ensure_child's start is
  over, the daemon puts the running harness in its class with
  setpriority(PRIO_DARWIN_PROCESS, pid, PRIO_DARWIN_BG); later tools inherit it.
  The run class rides on ChildAgent; SpawnSpec.cpu_class is gone.
- A cold start that fails or misses agent_start with its harness at background
  class anyway runs once more, its harness's external class also cleared, and
  logs agent_start_retried_at_normal.
- agent_start and the pool's start deadline errors name the agent process:
  pid, class, priority, run time, CPU time, threads running (proc_pidinfo of
  the one pid; no process listing). data.harness carries the same fields.
- The hosted behavior test: under a background agent host
  (ACPMUX_QOS=background) the harness starts above it and runs below once lowered.
lawrencecchen added a commit that referenced this pull request Oct 31, 2026
…names its harness and a background one runs once more (red)

Behavior tests through an in-process daemon's ACP client (macOS): the harness
answers session/new above background class and runs at it once started; a tool
its turn starts inherits it; an agent_start deadline error names the agent
process; a start held at background class by another process is retried once.
Red on the base: #27600 starts the harness at PRIO_DARWIN_BG before its exec.
lawrencecchen added a commit that referenced this pull request Oct 31, 2026
… its session's start is over

#27600 started every harness at PRIO_DARWIN_BG. On 2026-10-30 (load 125 on 18
cores, du and git scans of a 3.3 TB disk) a claude harness at background class
spent 90 s in trust-settings reads with under 1 s of CPU, every Chief turn
failed ("session/new: no answer within 60 s", "no harness session within 90s").

- The harness starts at its starter's class and clears an inherited
  background class at its exec (start_at_normal). Once ensure_child's start is
  over, the daemon puts the running harness in its class with
  setpriority(PRIO_DARWIN_PROCESS, pid, PRIO_DARWIN_BG); later tools inherit it.
  The run class rides on ChildAgent; SpawnSpec.cpu_class is gone.
- A cold start that fails or misses agent_start with its harness at background
  class anyway runs once more, its harness's external class also cleared, and
  logs agent_start_retried_at_normal.
- agent_start and the pool's start deadline errors name the agent process:
  pid, class, priority, run time, CPU time, threads running (proc_pidinfo of
  the one pid; no process listing). data.harness carries the same fields.
- The hosted behavior test: under a background agent host
  (ACPMUX_QOS=background) the harness starts above it and runs below once lowered.
lawrencecchen added a commit that referenced this pull request Oct 31, 2026
…names its harness and a background one runs once more (red)

Behavior tests through an in-process daemon's ACP client (macOS): the harness
answers session/new above background class and runs at it once started; a tool
its turn starts inherits it; an agent_start deadline error names the agent
process; a start held at background class by another process is retried once.
Red on the base: #27600 starts the harness at PRIO_DARWIN_BG before its exec.
lawrencecchen added a commit that referenced this pull request Oct 31, 2026
… its session's start is over

#27600 started every harness at PRIO_DARWIN_BG. On 2026-10-30 (load 125 on 18
cores, du and git scans of a 3.3 TB disk) a claude harness at background class
spent 90 s in trust-settings reads with under 1 s of CPU, every Chief turn
failed ("session/new: no answer within 60 s", "no harness session within 90s").

- The harness starts at its starter's class and clears an inherited
  background class at its exec (start_at_normal). Once ensure_child's start is
  over, the daemon puts the running harness in its class with
  setpriority(PRIO_DARWIN_PROCESS, pid, PRIO_DARWIN_BG); later tools inherit it.
  The run class rides on ChildAgent; SpawnSpec.cpu_class is gone.
- A cold start that fails or misses agent_start with its harness at background
  class anyway runs once more, its harness's external class also cleared, and
  logs agent_start_retried_at_normal.
- agent_start and the pool's start deadline errors name the agent process:
  pid, class, priority, run time, CPU time, threads running (proc_pidinfo of
  the one pid; no process listing). data.harness carries the same fields.
- The hosted behavior test: under a background agent host
  (ACPMUX_QOS=background) the harness starts above it and runs below once lowered.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…c environment

Since #27600 a harness goes to PRIO_DARWIN_BG once its start is over. The
stand-ins time turns with short sleeps; on a loaded fleet Mac (load1 13-21 on
10 P-cores) a background-class Codex-steer stand-in did not end its turn in
18-30 s, so stop::a_slow_stop_does_not_fold..., forum's steered rule and
steer_end's race failed only on macOS (diag: 7057c820, ad77491f, 8432fc15;
the same 3 pass with the class normal: a35af9a6, 0b8d6072, 1935c6fb).
daemon_env::vars sets ACPMUX_SESSION_CPU_CLASS=normal; tests/cpu_class.rs asks
for background in its process and its real daemon. Product default unchanged.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
… reds on loaded fleet Macs) (#30057)

* test(acpmux): a test's stand-in harness runs at normal CPU class (red on macOS)

* acpmux tests: stand-in harnesses keep normal CPU class in the hermetic environment

Since #27600 a harness goes to PRIO_DARWIN_BG once its start is over. The
stand-ins time turns with short sleeps; on a loaded fleet Mac (load1 13-21 on
10 P-cores) a background-class Codex-steer stand-in did not end its turn in
18-30 s, so stop::a_slow_stop_does_not_fold..., forum's steered rule and
steer_end's race failed only on macOS (diag: 7057c820, ad77491f, 8432fc15;
the same 3 pass with the class normal: a35af9a6, 0b8d6072, 1935c6fb).
daemon_env::vars sets ACPMUX_SESSION_CPU_CLASS=normal; tests/cpu_class.rs asks
for background in its process and its real daemon. Product default unchanged.

* acpmux: the steer race unit test's stand-in runs at normal CPU class

This branch was previously deployed

3 inactive deployments
artifacts — eb6d68cc Deployed Oct 30, 2026 by teamleaderleo via add the Windows daemon to the tree #79315
ci-routing-agent — eb6d68cc Deployed Oct 30, 2026 by github-actions[bot] via cmux-next macOS placement #47298
sentry-dsyms — eb6d68cc Deployed Oct 30, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #79315
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: performance Latency, CPU, memory, launch time risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant