[lawrencecchen]

Summary

On the fleet Macs, 3 acpmux tests failed only on macOS, on the STILL-RED list (STATUS 2359):

• stop::a_slow_stop_does_not_fold_the_next_prompt_into_the_stopped_turn
• forum a_rule_steered_into_a_running_turn_counts_as_told
• hub::steer_end::race_tests::an_unsent_steer_gets_its_own_turn_after_another_steer_finishes

Cause (test under load).

• Since acpmux: start agent harnesses at background CPU class on macOS, so cmux and its renderers go first #27600, once a session's start is over, the daemon puts its harness in PRIO_DARWIN_BG (cpu_class.rs, default background).
• The stand-in agent (tests/fake_agent.py) times its Codex-steer turn with short sleeps: it polls 300 x 10 ms for a joined prompt, then waits 0.2 s.
• On a loaded fleet Mac (load1 13 to 21 on 10 performance cores, from the daemon's own host-budget warning), a background-class stand-in did not end that turn in 18 to 30 s.
• The diagnostic event dumps show the turn's first chunk ("turn: X"), then nothing: jobs 7057c820 (stop), ad77491f (steer race) and 8432fc15 (forum: no answer to "warm up").
• The scratch commit 093a4d2 changed only the default class to normal. With it, all 3 pass on macOS: a35af9a6, 0b8d6072, 1935c6fb.

Fix (test environment only; the product default stays background).

• daemon_env::vars (the hermetic environment of every acpmux test process and of the daemons the tests start) sets ACPMUX_SESSION_CPU_CLASS=normal.
• tests/cpu_class.rs, which checks the product's class, asks for background in its own process and in its real daemon.
• The steer race unit test (library, no hermetic environment) names normal in its stand-in's profile env.
• New check stop::a_stand_in_harness_runs_at_normal_class_in_tests (macOS).

Cause detail (review). The class change alone is proved (093a4d2 changes only DEFAULT). Host load is only inferred: the daemon logged high load in the diagnostic runs, but not in the 2 reds at 33f01ce. Darwin background timer coalescing of the stand-in's 10 ms sleeps is another possible mechanism, and it does not need load. To decide, one run of the steer race test at background class on an idle mini (owner follow-up).

Note for the owner of cpu_class.rs. A real harness that times its work with short sleeps can also run slowly at background class on a loaded Mac. That is the product's choice (#27600, cmuxterm-hq#4947), and this PR does not change it. It only stops the tests from measuring it by accident.

Testing

run	ref	result
red: new check	33f01ce (test only)	macOS 3256ec1b (m4pro-7): FAIL, Background != Normal
red: stop / steer race at the test-only commit	33f01ce	macOS 2fd97413 (m4pro-7) FAIL (hub_integration.rs:1323 timeout); a29da46e (m4pro-4) FAIL (steer_end_race_tests.rs:73)
green stop::a_slow_stop_does_not_fold… 5 of 5	37defdb	macOS f2ccd79c m4pro-3, 66d4bc66 m4pro-8, 69f5abda m4pro-3, 5f443a81 m4pro-7, a77653ba m4pro-2 (each compiled acpmux, 1 passed, ~9.9 s)
green forum a_rule_steered… 5 of 5	37defdb	macOS b8dfe9bf m4pro-7, 7d2cba69 m4pro-2, b0da60be m4pro-6, 14a9237c m4pro-7, 899bf74d m4pro-8
green steer race 5 of 5	37defdb	macOS a49ada96 m4pro-2, 26b79ad4 m4pro-7, f78ba0e6 m4pro-3, 793bafaf m4pro-7, 2e8a6d05 m4pro-9
new check	37defdb	macOS efe04dc8: pass
cpu_class tests (product class still background)	37defdb	macOS f0c3f4af: 9/9
clippy	37defdb	89db498a red only on base hub/processes.rs:256 (fixed on feat by #30009)
after merging feat (0a45903)		clippy Linux d0a4e04f: pass; cpu_class macOS 5dd8ef18: see the check comment

Observed / Not run

Observed: everything above. Not run: Linux (Linux has no CPU class: for_session returns normal there).

Changelog

none

Proof

No capture: a test-environment change.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes three acpmux tests that timed out only on loaded fleet Macs because their stand-in harnesses ran at background CPU class once a session's start was over. This test-only change keeps the stand-ins at normal class; the product default (background) is unchanged.

• The hermetic environment (daemon_env::vars) now sets ACPMUX_SESSION_CPU_CLASS=normal for test processes and the daemons they start.
• tests/cpu_class.rs explicitly requests background in its own process and real daemon, so it still verifies the product's actual class.
• The steer race unit test sets normal on its stand-in's profile env.
• Adds a macOS check that a stand-in harness runs at normal class in tests.

^{Written for commit 0a45903. Summary will update on new commits.}

[cubic-dev-ai]

2 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cmux-tui/crates/acpmux/tests/cpu_class.rs">

<violation number="1" location="cmux-tui/crates/acpmux/tests/cpu_class.rs:28">
P3: Use the `acpmux::cpu_class::ENV` constant here, as the new `RealDaemon::cmd` line does. A literal would go stale silently if the variable is renamed.</violation>
</file>

<file name="cmux-tui/crates/acpmux/tests/hub_integration/stop.rs">

<violation number="1" location="cmux-tui/crates/acpmux/tests/hub_integration/stop.rs:717">
P3: This test checks only the resolved CPU class for the test process's env, not the class of a running stand-in harness. Rename it to describe the resolution (for example `the_test_environment_resolves_cpu_class_to_normal`), or add a real priority check.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P3: Use the acpmux::cpu_class::ENV constant here, as the new RealDaemon::cmd line does. A literal would go stale silently if the variable is renamed.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At cmux-tui/crates/acpmux/tests/cpu_class.rs, line 28:

<comment>Use the `acpmux::cpu_class::ENV` constant here, as the new `RealDaemon::cmd` line does. A literal would go stale silently if the variable is renamed.</comment>

<file context>
@@ -20,8 +20,13 @@ mod daemon_env;
+const HERMETIC: daemon_env::Process = daemon_env::Process::stand_in().with_env(&[
+    ("ACPMUX_TEST_CPU_CLASS_REF", "normal"),
+    ("ACPMUX_QOS", "background"),
+    ("ACPMUX_SESSION_CPU_CLASS", "background"),
+]);
 
</file context>

Suggested change

	("ACPMUX_SESSION_CPU_CLASS", "background"),
	(acpmux::cpu_class::ENV, "background"),

[cubic-dev-ai]

P3: This test checks only the resolved CPU class for the test process's env, not the class of a running stand-in harness. Rename it to describe the resolution (for example the_test_environment_resolves_cpu_class_to_normal), or add a real priority check.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At cmux-tui/crates/acpmux/tests/hub_integration/stop.rs, line 717:

<comment>This test checks only the resolved CPU class for the test process's env, not the class of a running stand-in harness. Rename it to describe the resolution (for example `the_test_environment_resolves_cpu_class_to_normal`), or add a real priority check.</comment>

<file context>
@@ -705,3 +705,15 @@ async fn a_long_replayed_history_stays_out_of_the_resumed_sessions_last_message(
+/// `tests/cpu_class.rs` asks for background itself and checks the product's class.
+#[cfg(target_os = "macos")]
+#[test]
+fn a_stand_in_harness_runs_at_normal_class_in_tests() {
+    assert_eq!(acpmux::cpu_class::CpuClass::for_session([]), acpmux::cpu_class::CpuClass::Normal);
+}
</file context>

Suggested change

	fn a_stand_in_harness_runs_at_normal_class_in_tests() {
	fn the_test_environment_resolves_cpu_class_to_normal() {

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 0a45903: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30057 -f ref=acpmux-test-harness-normal-class -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 821fe22; does not block

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4130052256501 (28aa638015)

[lawrencecchen]

Review (independent review subagent): SHIP at 0a45903

I reviewed only the PR's own changes (33f01ce, 90e1b98, 37defdb). The merge 0a45903 adds nothing to the 4 files: git diff 0a45903b6f1^2 0a45903b6f1 shows only these 4 files, and they did not change between 37defdb and the head. I did not build or run anything locally. I read the fleet logs with cmux-ci log.

1. Cause: background class is proven as the cause. Host load is only inferred.

• The A/B is clean. 093a4d2 has one parent, cb983f5 (the diag commit that produced reds 7057c820, ad77491f and 8432fc15). It changes one line: cpu_class.rs DEFAULT from Background to Normal. Each of a35af9a6 (stop, 9.9 s), 0b8d6072 (steer race, 5.2 s) and 1935c6fb (forum, 9.4 s) is green at 093a4d2.
• The event dumps agree with the PR body. 7057c820 shows turn: slow-stop, then session/cancel, then nothing. 8432fc15 shows FORUMDIAG no answer to session/prompt for "warm up".
• Load: only the 3 diag reds log host-budget warnings (load1 10.5 to 22.5). The two reds at 33f01ce (2fd97413 and a29da46e) log no load. No green run logs any load. Every red attempt failed ("failed in the full run and failed alone"), and every green took the same time (stop 9.8 to 9.9 s, steer race 4.8 to 5.1 s). That pattern looks like the class alone causes the delay, not the class plus load. A possible cause that this PR does not test: XNU gives darwinbg threads the largest timer-coalescing leeway, so each time.sleep(0.01) in fake_agent.py (codex_prompt, 300 polls) can stretch far past 10 ms with no load at all. The fix is correct either way. Only the explanation in the comments is not proven.

2. Product bug? Not blocking, but the note to the owner should say more.

• Background is the documented product choice (src/cpu_class.rs module docs and DEFAULT at :62, acpmux: start agent harnesses at background CPU class on macOS, so cmux and its renderers go first #27600 by teamleaderleo, cmuxterm-hq#4947), so a test-only exemption is correct here.
• The PR's owner note says "slowly ... on a loaded Mac". If timer coalescing is the cause, a real harness that relies on short timers (stream flushes, poll loops, a Codex steer join window) is slow at background class on any Mac, not only on a loaded one. Recommend: tell the PL-acpmux owner (OWNERS.md row "acpmux agent-session daemon"), and run the steer race test once at background on an idle mini to find out which cause it is. This is a follow-up, not a reason to hold this PR.

3. cpu_class.rs is still covered.

• resolve() order is profile env, then daemon env (std::env::var(ENV)), then config, then DEFAULT (src/cpu_class.rs:78-81, :101). The unit test at the end of that file still covers the order.
• The daemon's own ACPMUX_* variables are removed from the harness command (src/agent/command.rs:124-128), so the hermetic normal reaches for_session only as the daemon env, never as a profile value.
• tests/cpu_class.rs:25-29: enter() does retain and then push for injected keys (daemon_env.rs:198-201), so background replaces the normal that vars pushed. The in-process hubs read std::env, so they see background. The profile-normal and reference tests (:221, :235) now beat a daemon env of background, not only the default. That proves "the profile wins" more strongly than before.
• RealDaemon::cmd (:435): .env(ENV, "background") comes after apply's envs(vars), so the last value wins and the daemon gets background.
• Hosted test (:323): SpawnSpec.env now carries ACPMUX_SESSION_CPU_CLASS=normal to the harness, but nothing on that path reads it. The test sets the class itself with agent.set_run_class(CpuClass::Background) (:360), the agent host does not read the variable, and fake_agent.py does not read it. So the check does not change.
• f0c3f4af at 37defdb: 9 of 9 green, including both product-class checks (:205, :457) that assert pri 4.

4. The regression check is meaningful, but it is a proxy.

• stop.rs:717-719 is red at 33f01ce (3256ec1b: left: Background, right: Normal at stop.rs:718) and green at 37defdb (efe04dc8).
• It proves that the hermetic environment resolves to normal with no config. It does not cover the library steer-race test, which has no hermetic env and relies on the profile env at steer_end_race_tests.rs:12-17. That test has its own red (a29da46e at 33f01ce, steer_end_race_tests.rs:73) and its own greens. This is acceptable for a test-environment change.

5. Evidence is confirmed.

• Every listed job ran at 37defdb, compiled acpmux ("Compiling acpmux v1.0.0" in each log) and passed its 1 test with no flaky retry.
• stop: f2ccd79c, 66d4bc66, 69f5abda, 5f443a81, a77653ba.
• forum: b8dfe9bf, 7d2cba69, b0da60be, 14a9237c, 899bf74d.
• steer race: a49ada96, 26b79ad4, f78ba0e6, 793bafaf, 2e8a6d05.
• I did not check host names from the logs; the logs do not show them. 5dd8ef18 (cpu_class after the merge) was still compiling when I read it.

Nits (not blocking)

• daemon_env.rs:298-301, stop.rs:709-713 and steer_end_race_tests.rs:12-13 say the cause is "a loaded fleet Mac". Consider "at background class (darwinbg; load or timer coalescing)" until the idle-Mac run decides it.
• daemon_env.rs is also included by 4 cmux-tui/tests/* files. They already use acpmux:: items, so this compiles, and their stand-ins now also run at normal. That is intended.

[lawrencecchen]

Check classification at head 0a45903:

• resolve head and base, fast gates, panic ratchet (lowered baseline), post cmux-next/fast-gates (run 41269123166): cancelled ("Canceling since a higher priority waiting request for fast-gates-30057 exists"), no step ran. I re-ran the same SHA.
• Fleet cpu_class macOS 5dd8ef18 at this head: 8 pass, 1 TIMEOUT in cpu_class_a_start_that_misses_its_deadline_says_what_its_harness_was_doing. Base finding: the same test times out at feat a30fe2b (job fae79b08, m4pro-7). Its cause is acpmux: the person's new chat never fails on a slow host; agents never take the person's warm agent #28963: the person's agent start has no deadline, and this test's client is the person's. The fix is in its own base-red PR (branch acpmux-cpu-class-deadline-agent).

Fleet at the head: clippy Linux d0a4e04f pass. The other 8 cpu_class tests pass, so the product class is still background.
Review: SHIP at the head (issuecomment-6552932844).

[blacksmith-sh]

Found 10 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmpva7h2nxi/summary-fail.json (live):/summary-fail.json (live):	View Logs
/tmp/tmpxnjuauq7/summary-missing.json (live):/summary-missing.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs
test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)/ test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)	View Logs
test_no_runner_picker_returns_a_hosted_mac (main.NoHostedMacInNightlies.test_no_run ner_picker_returns_a_hosted_mac)/ test_no_runner_picker_returns_a_hosted_mac (main.NoHostedMacInNightlies.test_no_run ner_picker_returns_a_hosted_mac)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}