[lawrencecchen]

Summary

Fixes the Chief outage on Lawrence's laptop, 2026-10-30 20:40 PDT (NIGHTLY acpmux a112726). Every Chief turn failed with session/new: no answer within 60 s, and the daemon log said pooled session failed to start: no harness session within 90s harness=claude. #27600 starts every harness at PRIO_DARWIN_BG. The machine had load 125 on 18 cores and du/git scans on a 3.3 TB disk. The claude harness was already past its exec, but its main thread spent 90 s in SecTrustSettingsCopyTrustSettings -> open() / plist parse at disk tier 3, with under 1 s of CPU. Evidence: #28442 (comment).

#28442 (open, P-AUSTIN B10) moves the background class from darwinbg before the exec to the posix_spawn QoS clamp. That fixes a slow exec, but the harness still runs at background I/O after the exec, so this hang stays. A clamp also cannot be removed from a running process. This PR changes the order instead: the harness starts at normal class, and the daemon lowers it after the start.

1. A start never runs at background class. The harness starts at its starter's class (an agent host's utility clamp, or the daemon's class). Between fork and exec it clears a background class that it would inherit (cpu_class::start_at_normal, for example an agent host under ACPMUX_QOS=background). When ensure_child's start is over (initialize and session/new or session/load answered, including a pool claim and Claude's fork priming), the daemon lowers the running harness with setpriority(PRIO_DARWIN_PROCESS, pid, PRIO_DARWIN_BG) (ChildAgent::enter_run_class). Every process that the harness starts after that inherits the class, so the builds and tests of its turns stay below the GUI (acpmux: start agent harnesses at background CPU class on macOS, so cmux and its renderers go first #27600's goal). A pooled harness stays at normal class until a chat claims it. Model probes are never lowered (they never run a turn). The "a running process is never reniced" rule is changed on purpose: now the harness itself is reniced, once. A process that the harness started during its start (an MCP server) keeps its class, because acpmux does not list processes (check-no-process-scan.py). The class decision moved from SpawnSpec.cpu_class to the ChildAgent. The field is removed: serde ignores it from an older daemon, and a host is always the daemon's own binary.
2. Retry. If a cold start fails or misses agent_start and its harness is at background class anyway (from a cause outside acpmux), the start runs one more time. The daemon also clears the second harness's external background flag right after the spawn, and logs agent_start_retried_at_normal (with the harness state) plus a warning.
3. A late start says why. The agent_start and pooled-start deadline errors end with, for example, ; its agent process (pid 123) is alive at background CPU class (priority 4), ran 92 s, used 0.8 s of CPU, 0 of 14 threads running, or is not running / had not started. The text comes from proc_pidinfo of that one pid, with no process listing. data.harness carries the same fields. The text names no vendor.
4. Chief waits past acpmux's deadline. Chief's turn session/new waited 60 s, which is less than acpmux's 90 s agent_start, so Chief never saw acpmux's reason. It now waits 100 s (SESSION_NEW_DEADLINE). Other control requests keep 60 s.

macOS facts used: setpriority(PRIO_DARWIN_PROCESS, pid, …) on another process of the same uid sets that task's external darwinbg flag (xnu do_background_proc: curp == targetp gives internal, otherwise external). A self call in pre_exec clears the internal flag. Darwinbg is task policy, so it survives exec and children forked later inherit it. Children that already exist are not changed. The behavior tests below check these on a fleet Mac.

Docs: cpu_class.rs module doc, docs/background-processes.md ("Start (macOS)"), config.rs cpuClass.

Owner: PL-acpmux (cpu_class.rs, agent.rs, agent_host); Chief (optchat-chief rpc/acpmux). Does not touch hub/idle_census.rs.

Testing

Behavior tests at the daemon's ACP client (an in-process daemon, serve_connection) and the agent host link, macOS (tests/cpu_class.rs, stand-in tests/fake_agent.py):

• cpu_class_a_session_starts_at_normal_class_and_runs_at_background_once_started: the harness answers session/new above pri 4. After the answer it is at pri 4, and a tool that its turn starts is at pri 4.
• cpu_class_a_profile_can_keep_its_harness_at_normal_class, cpu_class_a_profile_reference_that_resolves_to_normal_keeps_normal_class: the opt-out still works.
• cpu_class_a_start_that_misses_its_deadline_says_what_its_harness_was_doing: a harness that cannot finish its start (FAKE_START_GATE) gives an agent_start error that names its pid, class, run time, CPU and threads. A normal-class start is not retried.
• cpu_class_a_start_that_fails_at_background_class_runs_once_more: another process puts the first harness in background class during its start, and the harness fails initialize but stays alive. The start is retried once, logged with background: true, timedOut: false, and the session starts at normal class.
• cpu_class_a_daemon_runs_a_hosted_harness_at_background_once_its_start_is_over: a real acpmux daemon run with agent hosts on (as in the app), driven by acpmux run "class-probe: …". The chat harness answers session/new above pri 4, then runs at pri 4, and a tool from its turn is at pri 4.
• cpu_class_a_hosted_harness_starts_at_normal_under_a_background_host_then_runs_below: the agent host runs in background class (ACPMUX_QOS=background, pri 4). Its harness and a tool started during the start run above pri 4. After enter_run_class the harness and a tool its turn starts are at pri 4.

The test of a harness under load is the mechanism, not a timing: a start that has a deadline does not run at background class.

Focused command: cmux-ci run --class isolated --label macos-27 --script scripts/ci/cmux-tui-rust-check.sh --arg test --arg cpu_class

Results (fleet Macs, cmux-ci isolated, label macos-27; Linux on rbx):

• Red at edf10ec (tests only, on base 2b30971): step 898fcb69 on aws-m4pro-4, 3 failed, 4 passed. …runs_at_background_once_started failed with "the harness answered session/new at background class". The deadline test failed with the bare agent_start did not finish within 3s. The retry test failed with "the retried start failed: agent_start did not finish within 5s".
• Green at 29be556: step d46d625f, 8 of 8 passed. macOS clippy step 0d8cc078: pass.
• Review delta at c972b11 (gate step 656de634, cmux10's Mac mini): 9 tests ran. All passed except the in-process hosted test: it read the background host at pri 46 once (a short kernel priority promotion), and it passed alone (FLAKY). Fixed in e83e364: a class read is now the lowest of 5 reads, and the test ends its host before its checks. macOS clippy at c972b11 (gate step 8b3181ce): pass. rbx at c972b11: cargo clippy -p acpmux --all-targets -D warnings, plus hub_integration agent_start, session_pool and agent_host tests: pass.
• Gate at e83e364 (receipt /tmp/gates/e83e364b…v6.json, all 8 steps green):
  • macOS cpu_class, cmux-ci f68b6198 on aws-m4pro-3: 9 of 9 passed, including the real-daemon hosted test.
  • macOS clippy b72d87e4: pass. macOS never_answers_its_start 6668d34f: pass.
  • rbx Linux: acpmux clippy, hub_integration agent_start, session_pool (34 of 34), agent_host: pass.
  • Chief: rbx tests pass; macOS Chief tests (session filter) 71f8354c, clippy and fmt: pass.
  • Chief first-turn cost 47ee33a9: PASS, 18,680 of 50,000 tokens.
  • check-no-process-scan: ok.
• Base reds, skipped or narrowed by name. Each one is red at feat-cmux-next 02e24e8 too, and the owner lane is acpmux-base-reds:
  • session_pool on macOS: cx-n1nrjk.
  • transfer_guard after_a_restart: cx-q0b1zh.
  • flow_trace_bind: cx-fel6hd.
  • peer_restart close_1012: cx-85z095.
  • setup keychain: cx-c8ouws.
  • startup_load: cx-ij2jg6.
  • subsessions: cx-b0utmu.
  • Chief acpmux_peer_moved: cx-2hwias.
  • Chief eval_upsell: cx-2ct95t.
  • Chief toolpack_mcp: cx-71g32y.
  • agent_host orphan_exit::a_daemon_connecting_while_the_harness_ends_is_told_it_exited: cx-o8vggt. It fails 17 of 20 runs alone at the base and 15 of 20 at the head.
• Landing gate at 458e0d2 (receipt /tmp/gates/458e0d21….json): all 8 steps green.
  • macOS cpu_class 9 of 9, macOS clippy, macOS deadline test.
  • rbx acpmux clippy and tests. Chief tests on rbx and macOS.
  • Chief first-turn cost: pass. The stand-in fault on some hosts is cx-rkzfny.
  • no-process-scan: ok.
• Landed directly as 458e0d2 in merge train 676611b on feat-cmux-next.

Observed / Not run

• Not run: an A/B under real disk load on a laptop (load 125 + a 3.3 TB scan). The tests prove the mechanism (a start runs above pri 4; a running session is at pri 4), not a timing.
• Not run: the full macOS workspace suite. Only the cpu_class and start filters, clippy, Chief's own tests and the Chief first-turn cost guard ran.
• Not checked: the real agent of a launcher harness (a node shim, an ACP adapter that starts the CLI as a child). By design it stays at normal class (documented; follow-up).

Changelog

Fixed: On a busy Mac, starting an agent chat no longer waits behind other disk work, and a start that times out now says what the agent process was doing.

Proof

No capture: the change is in the acpmux daemon and Chief, and has no UI.

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 458e0d2: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28923 -f ref=feat-cmux-next-harness-start-normal -f platform=macos -f backend_mode=local

[lawrencecchen]

Review (independent review subagent, Claude Opus 5.5, correctness first, read-only): FIX at 6a0ec27. No blocking finding, 7 should-fix. The reviewer probed the macOS facts on Darwin 27 with tail -f /dev/null processes:

• An external setpriority(PRIO_DARWIN_PROCESS, pid, PRIO_DARWIN_BG) gives pri 4, also under a utility clamp, and later children inherit it.
• The pre_exec self-clear clears an internal darwinbg (taskpolicy -b), but not an external one inherited from the parent. A clear from another process clears it.
• proc_pidinfo(PROC_PIDTASKINFO) returns 96 bytes, which equals size_of::<proc_taskinfo>().
• pti_priority is 31 at default, 4 at background, 20 under the utility clamp.
• The timebase is 125/3, so the tick conversion is needed.

Resolution, fixed at c972b11:

1. The retry doubles the start time, but Chief waited only 100 s. Fixed:
   • Right after every harness spawn (direct, hosted, pooled), the daemon now clears an external background class (clear_running). So a retry happens only when something lowers the harness again during its start.
   • Chief's SESSION_NEW_DEADLINE is now 200 s (two 90 s budgets plus margin). Its turn limit is 180 min.
   • A retried timeout no longer writes agent_start_timed_out.
2. Only the harness pid is lowered, so the real agent of a launcher harness (a shim or adapter that starts the agent during its start) stays at normal class. Documented in the cpu_class.rs module doc and docs/background-processes.md. Not changed in code: a fix needs either a process listing (refused by check-no-process-scan.py) or a shim-specific resolution. Follow-up below.
3. Off macOS every timeout said "its agent process is not running". Fixed: HarnessSeen keeps NotStarted / Ended / Unread / Running apart, and Unread adds no clause.
4. The production wiring was not tested at the daemon boundary. Fixed: new test cpu_class_a_daemon_runs_a_hosted_harness_at_background_once_its_start_is_over uses a real acpmux daemon run with agent hosts on and acpmux run "class-probe: ...". It checks the chat harness at session/new (above pri 4), after the start (pri 4), and a tool of its turn (pri 4).
5. The retry test used a short 5 s budget. Fixed: the first background harness now fails initialize with an error and stays alive. The test uses the default budget and covers the error branch of the retry.
6. A runner at external background class makes the checks meaningless. Fixed: the tests fail first with "the test process itself runs at background class (pri N)".
7. "Normal" still has utility disk I/O. Documented: the daemon and its hosts keep IOPOL_UTILITY (qos.rs), which is utility, not throttled. The 14.5 s from the laptop was measured with session.cpu_class=normal under the same daemon, so it is the class that this PR gives the start.

NITs:

• Retry on any error while the harness is at background class: kept. It is bounded to two tries, and the first harness always ends before the second.
• Pid reuse in the window between the host reaping the harness and exit_seen: accepted. The risk is low: same user only, and macOS pids increase.
• A readopted host after a daemon restart has no run class, so its harness stays at normal class: accepted, follow-up.
• Model probes stay at normal class: on purpose, they have a deadline and run no turns.
• A config agentStartTimeoutMs of 100 s or more defeats Chief's margin: the doc comment names the 90 s default.

Follow-ups (not in this PR): put the real agent of a launcher harness in background class without a process scan (for example, the agent host lowers its own process group's members as they report through the holder track); give a readopted host's harness its run class.

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}