Skip to content

cmux-next tests: the Extensions menu suite scales its budgets by the runner's reference workload - #24157

Merged
teamleaderleo merged 16 commits into
feat-cmux-nextfrom
browser-import-2/extmenu-calibrate
Oct 28, 2026
Merged

teamleaderleo merged 16 commits into
feat-cmux-nextfrom
browser-import-2/extmenu-calibrate

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Leo’s request

Browser import and Extensions menu behavior should remain correct.

Change

Share the existing FrameBudgetCalibration with the Extensions menu performance suite. Keep the nominal 16 ms open, 50 ms first-open and 16 ms hover budgets; scale them by the same runner reference workload already used by sidebar tests. Runners over 4× nominal speed cost, or with excessively noisy samples, print a reason and skip the timed assertions. Functional assertions still run, including icon reuse and opening without waiting for Chromium.

The helper moves to Tests/CmuxNextTestSupport; only the browser and app test targets depend on it. Sidebar callers gain the import and retain the existing calibration logic. The build-input guard recognizes this reviewed test-only target because no shipped product links it, and the architecture map records it as a core helper. Two screen-independent tests exercise slow and fast runner behavior, and the old duration-assertion allowlist entry is removed.

Risk: test-only package wiring and performance-budget calibration. No product sources, visual design, new timers, observers or animation loops change. Calibration samples are synchronous and bounded to 15 measurements per checkpoint.

Validation

  • Reconciled feat-cmux-next at 47b4dbb1c154c4c29c4802a2e0416ec450ff51b4; resolved Package.swift by retaining both sides’ dependencies. Diff: 11 files, +137/−40.
  • python3 tests/test_dev_release_product_key.py: all 9 tests pass after recognizing the shared test-support target.
  • The changed-file pre-push gate passes all scanners, SwiftFormat, SwiftLint and the architecture ratchet in 3.3 seconds.
  • The fleet Release compile and cmux-next static gate passed at final head 323876d01f2088dc1a4cd9aa294c395c658bcc8a.
  • Earlier same-harness scaling regression: disabling scaling failed at 6.06 ms against a 2.82 ms budget; with scaling the slow-runner stand-in passed at 5.28 ms against 2.59 ms nominal / 7.8 ms scaled. These are calibration tests, not evidence of a product speedup.

Known limit: the CPU reference cannot explain extra AppKit drawing cost on a nominal-speed runner; earlier first-open measurements of 51.3–54.6 ms still exceeded 50 ms with factor 1.00. Those base timing failures are separate from this calibration/wiring change. Related follow-up: #26303.

Agent session: agent session

teamleaderleo and others added 4 commits October 21, 2026 17:45
…t target both test targets can use

CmuxNextBrowserTests cannot see a file in CmuxNextAppTests, and its
Extensions menu budgets need the same runner calibration the sidebar
suites use. The helper moves unchanged into a small regular target under
Tests/ (no product links it), made public, and takes threadCPU and
percentile with it; SidebarScrollBench's copies forward to them.
CmuxNextAppTests and CmuxNextBrowserTests depend on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d18-98cd-7a42-ac32-a70f43729c15&host=cosmic-gray
…e by its reference workload

A stand-in runner does every timed open and every reference sample three
times, with no screen, so the test runs on every runner. Its opens break
budgets set on this host's own opens, and must hold them once they scale
by the runner's FrameBudgetCalibration. Budgets.scaled is a stub that
never scales, so the test is red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d18-98cd-7a42-ac32-a70f43729c15&host=cosmic-gray
…rkload

aws-m4pro-9, a runner 3-4x slower at every sidebar frame, read a 51.3 ms
first open of the Extensions menu against 50 ms with no change in the
code. The open and hover tests now time FrameBudgetCalibration's
reference workload before and after their timings and hold the 16 ms
open, 50 ms first open and 16 ms hover budgets scaled by it, or skip them
with the reason printed when the runner is over 4x nominal or too noisy
to read. The budgets themselves are unchanged; the icon-reload and
no-Chromium-wait checks stay unconditional.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d18-98cd-7a42-ac32-a70f43729c15&host=cosmic-gray
…ntry goes

Its budgets are no longer literal duration asserts: they go through
Budgets.overruns, scaled by the runner's calibration, and the checker
finds nothing in the file without the entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d18-98cd-7a42-ac32-a70f43729c15&host=cosmic-gray
@teamleaderleo teamleaderleo added the risk:low Risk tier (docs/ci/risk-tiers.md) label Oct 22, 2026
@teamleaderleo

teamleaderleo commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Owned by cx4-bi24157 on huge-blue

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent): SHIP at f6fdee3

  • Package.swift: CmuxNextTestSupport is a regular target under Tests/, linked into no product, and the dependency lists are sorted.
  • The move: FrameBudgetCalibration is unchanged in logic apart from public and the Self.* helpers.
  • Scaling: the factor is floored at 1, so a fast runner keeps the 16/50/16 ms budgets.
  • Skips: only over 4x or on noisy samples, each with a printed reason.
  • Tests: the stand-in tests are deterministic.
  • Scope: no shipped code changes. The determinism allowlist finds the same set as base.

Not fixed here: aws-m4pro-9 is slow at drawing, not CPU (factor 1.00), so its first-open red stays (see "Not in this PR").

@github-actions github-actions Bot added risk:normal Risk tier (docs/ci/risk-tiers.md) and removed risk:low Risk tier (docs/ci/risk-tiers.md) labels Oct 22, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-gx5fb created for this PR by the cmux coordinator.

…ort-2/extmenu-calibrate

# Conflicts:
#	Packages/macOS/CmuxNext/Package.swift
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 323876d: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr24157 -f ref=browser-import-2/extmenu-calibrate -f platform=macos -f backend_mode=local

web and size checks: green

@blacksmith-sh

blacksmith-sh Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Found 4 test failures on Blacksmith runners:

Failures

Test View Logs
build_only must be a conjunctive exclusion on the helper, signing, and publication jobs
, and must not gate the unsigned app build/
build_only must be a conjunctive exclusion on the helper, signing, and publication jobs
, and must not gate the unsigned app build
View Logs
nightly.yml must derive the embedded key and sign every appcast through the helper/
nightly.yml must derive the embedded key and sign every appcast through the helper
View Logs
test_the_nightly_is_covered (main.NightlyStepCoverageTests.test_the_nightly_is_cove
red)/
test_the_nightly_is_covered (main.NightlyStepCoverageTests.test_the_nightly_is_cove
red)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent): SHIP at 3a67073

The head is a merge of feat-cmux-next into the earlier reviewed head f6fdee3. The one conflict was in Package.swift: base added the test-only CmuxNextTestExitTrace target in the same place as this PR's CmuxNextTestSupport. The resolution keeps both.

  • Merge: base's changes to Package.swift come through line for line, with nothing lost or duplicated. Both targets are listed in CmuxNextAppTests in sorted order, and check-package-dependency-lists.py exits 0.
  • Diff against the merge base: the same 7 files (+133/−39) as the reviewed change. Nothing else came in through the merge.
  • Name clashes: base has no other CmuxNextTestSupport and only one FrameBudgetCalibration, so nothing clashes.

…pport, where FrameBudgetCalibration now lives

Base added SidebarHoverCardRetargetBenchTests after this branch moved FrameBudgetCalibration out of CmuxNextAppTests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d18-98cd-7a42-ac32-a70f43729c15&host=cosmic-gray
@cursor

cursor Bot commented Oct 22, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent): SHIP at 9380e81

Since 3a67073 (shipped), this head merges feat-cmux-next again to pick up #24227's test-compile fix (the earlier fast tier and daemon reds were that base break, HoleRecordingPage). It also adds import CmuxNextTestSupport to SidebarHoverCardRetargetBenchTests.swift, a file base added that uses FrameBudgetCalibration after this PR moved the type into the new target.

  • Imports: every test file that uses FrameBudgetCalibration or its moved helpers imports CmuxNextTestSupport. Every member they use is public, and both test targets list the module, in sorted order.
  • Old helpers: SidebarScrollBench.threadCPU/percentile still exist and call through to the moved helpers. FrameBudgetCalibration is defined once.
  • Merge: it brings in base changes only. The diff against base is the PR's 7 files plus the one import line.
  • Dependency lists: check-package-dependency-lists.py exits 0.

Package.swift: CmuxNextBrowserTests keeps base's CmuxNextDesign and this
branch's CmuxNextTestSupport. SidebarLagTests (new on base) imports
CmuxNextTestSupport, where FrameBudgetCalibration now lives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15d18-98cd-7a42-ac32-a70f43729c15&host=cosmic-gray
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › Checks job structure is red on 12 of 14 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › No fleet host names in gallery fixtures is red on 11 of 14 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next / cmux-next checks (god files, concurrency, crash safety, l10n) › No new SwiftUI (macOS UI is AppKit) is red on 12 of 14 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / cmux-next page strings generate is red on 13 of 13 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 12 of 13 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: no activity for 54 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in Packages/macOS/CmuxNext/Package.swift. Listed for a lane to resolve.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra xhigh, next-review): SHIP at e5308c8

No blocking findings.

The helper move preserves its logic, and callers have the required imports and dependencies. macOS tests could not be run in this Linux environment.

@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo teamleaderleo removed the hold cmux-next batch queue: keep this PR out of the batch (docs/ci/cmux-next-batch.md) label Oct 28, 2026
@teamleaderleo
teamleaderleo merged commit 9616df3 into feat-cmux-next Oct 28, 2026
177 of 186 checks passed
@teamleaderleo
teamleaderleo deleted the browser-import-2/extmenu-calibrate branch October 28, 2026 04:20
@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40631829534 at 09b9b3f failed: cmux-next checks (god files, concurrency, crash safety, l10n).
Those jobs last passed at 03c0c73. This pull request is one of 2 merged since: #26400, #24157.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 323876d0 Deployed Oct 28, 2026 by teamleaderleo via namespace autofix #5035
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant