[teamleaderleo]

Adds a warning-only NEXT perf harness for Leo's constant-scaling requirement. It runs the existing new-surface, sidebar stress and automation typing probes at 1, 100 and 1000 seeded workspace counts, records p50/p95/p99 rows, and warns when 1000-workspace p95 exceeds the one-workspace value by more than 10%.

Changelog

• Added warning-only 1/100/1000 workspace interaction coverage to the NEXT perf harness.
• Added topology validation and incomplete-sample reporting so partial probes cannot look flat.

Leo said: "It should just never, ever, ever, ever be f***ing acceptable for there to be some sort of linear time growth degradation in performance just because the app somehow has tons of workspaces, thousands, whatever workspace." This keeps the budget advisory so it reports regressions without slowing merges. It covers the contention baselines tracked in #34383 and #34384.

Performance	before	after	delta	harness
Workspace-count scaling coverage	0 interaction rows	15 interaction/count rows (p50/p95/p99)	+15 rows	workspace-scaling-bench.py
Scaling warning budget	none	1.10x p95 (warning-only)	new advisory	workspace-scaling-bench.py
Large topology seeding	terminal-only (PTY capped)	empty rows plus bounded active terminals	safe at 1000	--empty-workspaces, --workspace-count

Validation: python3 -m unittest scripts/cmux-next/tests/test_workspace_scaling_bench.py scripts/cmux-next/tests/test_perf_completeness.py (run through cmux-ci test at this head).

🤖 Generated with Codex

Agent session: agent session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 8eec2c9: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr36079 -f ref=lane/workspace-scaling-bench -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4210546330101 (bcf992a1a9)

[teamleaderleo]

Review (codex gpt-6-astra high, next-review, since=20261105T041857Z): BLOCK at 473cc77

BLOCK at 473cc77

Visual verdict: no images; Leo tries the build

Targeted unit tests and Python compilation pass.

1. [P1] [blocking] [category:tooling] [finding:F1] [outcome:pending] scripts/cmux-next/perf-cron.py:161 adds a suite without bumping SPLIT_SPRING_MEASURE_VERSION (scripts/measure/split-spring.sh:39). Historical commits remain eligible for bisects even though their dispatcher lacks this suite, so those runs fall through to another bench and emit no workspace_scaling.* rows.

2. [P2] [follow-up] [category:performance] [finding:F2] [outcome:pending] workspace-scaling-bench.py:163 labels sidebar measurements as 1/100/1000, but sidebar-stress-bench.py first creates an uncounted bench start workspace before grow(size). The actual topologies are therefore size+1.

3. [P1] [follow-up] [category:reliability] [finding:F3] [outcome:pending] workspace-scaling-bench.py:147 ignores probe exit codes, and main() still returns success with missing reports or partial rows. An incomplete GUI probe can therefore be recorded as a successful scaling result instead of being marked incomplete.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 9e05d63

BLOCK at 9e05d63

Visual verdict: no images; Leo tries the build

• [P1] [blocking] [category:correctness] [finding:F1] [outcome:pending] The 1-workspace sidebar run is invoked with --sizes 0 (workspace-scaling-bench.py:167-170), leaving only the startup row. scenario_switch therefore returns “fewer than two visible rows,” sidebar_rows emits no switch row, and the completeness check records that probe as missing (workspace-scaling-bench.py:188-195). Every scheduled run is consequently incomplete and cannot produce the advertised 15 rows; perf-cron also treats the report as an incomplete budget part.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex, router-review): SHIP at c4565b2

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 1. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at c4565b2

Visual verdict: no images; Leo tries the build

Checked the targeted unittest suite and Python compilation; both pass.

• [P2] [follow-up] [category:performance] [finding:F1] [outcome:pending] The typing comparison changes the active workload with the workspace count: 1 workspace runs zero agents and disables the browser, while 100/1000 run 10 agents plus an unpaced browser loop (workspace-scaling-bench.py:177, workspace-scaling-bench.py:180). A 1000-vs-1 warning can therefore be caused by automation load rather than workspace scaling; keep the workload invariant or compare like-for-like probes.

Follow-ups:

• [finding:F1] Keep the typing workload invariant across counts.

[cmux-bot]

cmux-bot approves c4565b2: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (codex, router-review): SHIP at 5a2a951

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 5a2a951

Visual verdict: no images; Leo tries the build.

• [P1] [follow-up] [category:correctness] [finding:F1] [outcome:pending] The one-workspace sidebar run intentionally omits switch (workspace-scaling-bench.py:195), while scaling_warnings() requires a workspace-1 baseline (:118-124). Consequently switch scaling can never produce a warning, and the harness emits 14 rows rather than the advertised 15. Define a valid baseline or remove/explicitly exclude this interaction from the promised coverage.

• [P2] [follow-up] [category:reliability] [finding:F2] [outcome:pending] Report-only mode silently accepts mismatched or empty report lists because zip(COUNTS, ...) truncates without validation (workspace-scaling-bench.py:222-224). A malformed invocation can therefore write an empty, successful report with no incomplete entry; validate that exactly three reports per probe are supplied and mark missing counts incomplete.

Follow-ups:

• F1: restore switch-baseline coverage or correct the advertised row/budget scope.
• F2: reject or mark incomplete truncated report-only inputs.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex, router-review): SHIP at 5128710

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 1. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 5128710

Visual verdict: no images; Leo tries the build

• [P2] [follow-up] [category:reliability] [finding:F1] [outcome:pending] scripts/cmux-next/automation-typing-bench.py:167-169 ignores every workspace create failure and reports the requested count as seeded. A partial 1000-workspace seed can therefore produce a “1000” measurement at a smaller topology without marking the scaling report incomplete; check return codes and verify the actual workspace count before recording the result.

Follow-ups:

• F1 — validate empty-workspace creation and actual topology size before accepting the typing probe result.

[cmux-bot]

cmux-bot approves 5128710: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex, router-review): SHIP at 4113ed2

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 4113ed2

Visual verdict: no images; Leo tries the build

The focused unit tests pass. One incomplete-sample case should be tracked:

• [P1] [follow-up] [category:correctness] [finding:F1] [outcome:pending] workspace-scaling-bench.py:166 only checks probe exit status and report existence. The instant probe returns success even when runs are missed, and the harness then accepts partial p95 samples without recording incomplete; perf-cron can therefore treat a degraded or under-sampled topology as a valid baseline. Propagate each probe’s completeness/missed-sample status into failures before emitting rows.

Follow-ups: F1.

[teamleaderleo]

Review (codex, router-review): SHIP at 9073800

Reviewer: codex gpt-6-astra high, fast tier.

SHIP at 9073800

Visual verdict: no images; Leo tries the build

Checked the harness wiring, report parsing, topology validation, warning behavior, and the supplied unit tests (15 tests, all passing). No correctness, reliability, security, or regression findings.

Follow-ups: none

[teamleaderleo]

Review (codex, router-review): SHIP at 8eec2c9

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 8eec2c9

Visual verdict: no images; Leo tries the build

1. [P1] [follow-up] [category:performance] [finding:F1] [outcome:pending] scripts/cmux-next/workspace-scaling-bench.py:151 accepts undersized phases because keys is the number attempted, not the requested 30 samples; sidebar rows and report-only instant fallback likewise accept one-sample reports. Partial probes can therefore publish plausible scaling baselines instead of being marked incomplete.

2. [P2] [follow-up] [category:reliability] [finding:F2] [outcome:pending] scripts/cmux-next/workspace-scaling-bench.py:317 assumes nested reports have the expected types after only validating the top-level object. A valid report such as {"summary":"bad"} or {"sizes":"bad"} raises AttributeError during run_app, aborting the harness instead of recording an incomplete probe.

Follow-ups:

• [F1] Enforce requested sample/event counts for every probe before publishing rows.
• [F2] Validate nested report shapes and convert malformed reports into incomplete failures.

[cmux-bot]

cmux-bot approves 8eec2c9: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (codex, router-review): SHIP at 8eec2c9

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 8eec2c9

Visual verdict: no images; Leo tries the build.

• [P2] [follow-up] [category:performance] [finding:F1] [outcome:pending] workspace-scaling-bench.py:302 sets agents=0 and disables the browser, so automation-typing-bench.py:375 runs the “automation” phase with no loads at all. The reported typing rows therefore duplicate idle typing and do not cover the advertised automation contention workload.

• [P2] [follow-up] [category:reliability] [finding:F2] [outcome:pending] workspace-scaling-bench.py:317 and :340 assume nested summary/sizes values are dictionaries after only validating the top-level report. A syntactically valid malformed report such as {"summary": "bad"} crashes app mode instead of being recorded as incomplete; guard these nested shapes consistently with replay mode.

Follow-ups:

• [finding:F1] Restore a fixed nonzero bounded automation load.
• [finding:F2] Harden app-mode nested report validation.

[teamleaderleo]

Independent review (Claude Opus 5.5, medium), read-only, at exact head 8eec2c95bbe89064dd9fc5f777e4941c65aa5a65 (merge-base 1ca554bdabd). Nothing was run; this comes from reading the diff and the probes it drives.

Fixed since my BLOCK at 9073800

• Exact topology: the typing and instant probes now close restored rows, then seed to the exact count. The sidebar probe grows count-1 from its one bootstrap row. rows_listed and the typing workspace count are checked exactly.
• One-workspace cases: sidebar_scroll and switch at 1 are explicit undefined rows (p95 null, so rows_of_part skips them). Their ratio baseline is 100, so switch now gets a 100-to-1000 check.
• Typing workload is invariant: --agents 0 --no-browser --layouts light --samples 30 --rounds 1 at every count. A phase counts only when complete >= keys.
• Incomplete reports: missed, mixed or all-missed instant samples, frame replies that aren't frame_metrics.complete, partial typing phases and malformed or non-object JSON all drop the row. App mode records the probe in incomplete, and report-only mode gets the same check from coverage_failures. report_is_incomplete keeps such a run out of the last-good baseline. The RPC-for-frames substitution is gone.
• CI safety: target_of returns None for workspace_scaling.*, so rows aren't judged against the 10 ms target (a test covers it). The cron part is gated on needs: workspace-scaling, i.e. the dispatcher and the bench file exist at the commit. Probe timeouts add up to at most about 3000 s, inside run_seconds 3600 and --vdisplay-hold=3600. A timeout kills only the probe's own process group. The scaling budget never fails a job.

Non-blocking, fix forward

1. reset_restored_workspaces in the typing probe waits for snapshot.get to list zero workspaces. If the topology lists the store's hidden home workspace (home_not_closable), it never gets there: type is then missing at every count. That shows up as incomplete, not as a false pass, but confirm it on the first fleet run.
2. The sidebar check rows_listed != count is exact, while the probe's own grow accepts 95%. If grouped or collapsed rows aren't listed, 100 and 1000 come out incomplete. Again this fails safe.
3. sidebar_scroll is frame cadence (intervals_ms), so it shows dropped frames, not input-to-frame latency. Fine for a trend row; label it that way.
4. The repo rule against adding CI perf rows and budgets: this adds a cron part (advisory, idle capacity, opens the tracking issue only, never gates a PR). The SPLIT_SPRING_MEASURE_VERSION 4→5 bump rebases every part's history, even though the needs gate already stops older commits from running this suite. Worth reconsidering, but neither breaks base or loses data.
5. The PR body still shows no fleet 1/100/1000 run. Add one when it exists.

Review (claude opus-5.5 medium): SHIP at 8eec2c9

[teamleaderleo]

Review (claude-opus-5-5): SHIP at 8eec2c9

[teamleaderleo]

Review (codex, router-review): SHIP at 8eec2c9

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 8eec2c9

Visual verdict: no images; Leo tries the build

Targeted tests passed: 191 tests; Python files compile.

• [P1] [follow-up] [category:correctness] [finding:F1] [outcome:pending] typing_rows falls back to a complete light-idle phase when the automation phase is missing or incomplete, so partial automation data can publish a valid type row and avoid coverage_failures (scripts/cmux-next/workspace-scaling-bench.py:151-163).
• [P1] [follow-up] [category:performance] [finding:F2] [outcome:pending] run_app invokes automation typing with --agents 0 --no-browser, leaving the automation phase with no background loads; the advertised automation/contention probe therefore measures idle typing (scripts/cmux-next/workspace-scaling-bench.py:299-305).

Follow-ups:

• [finding:F1] Require a complete automation phase before publishing type.
• [finding:F2] Use a fixed nonzero active workload or revise the harness claim.

[teamleaderleo]

Review (claude-opus-5-5): SHIP at 8eec2c9

[teamleaderleo]

Review (claude-opus-5.5): SHIP at 8eec2c9

[teamleaderleo]

Review (claude-opus-5-5): SHIP at 8eec2c9

Independent Claude Opus 5.5 (medium) review at exact head 8eec2c9. Full findings: #36079 (comment). No blocking issues: exact-topology seeding, explicit undefined 1-workspace scroll/switch rows with a 100-workspace baseline, invariant typing workload, incomplete/malformed reports kept out of baselines, scaling rows exempt from the 10 ms target, and a capability-gated cron part. Fix-forward items are listed in the comment.

Review (claude opus-5.5 medium): SHIP at 8eec2c9

[teamleaderleo]

Review (claude opus-5.5 medium): SHIP at 8eec2c9

[teamleaderleo]

Review (claude opus-5.5 medium): SHIP at 8eec2c9