ci: reuse the compiled app and UI tests for runtime regressions by teamleaderleo · Pull Request #13166 · manaflow-ai/cmux

teamleaderleo · 2026-09-20T07:20:56Z

Summary

tests-build-and-lag waits for compile admission, then builds the app again. The existing artifact contains the app and unit-test bundles, but lacks the UI-test runner needed by the display regressions.

Build the cmux UI-test scheme in compile admission's existing DerivedData alongside the unit-test schemes. Include its manifest and bundle in the same artifact. The lag job restores that exact-run, checksummed artifact and runs the display tests with -xctestrun and test-without-building, removing its separate dependency resolution and app build. Both consumers use one restore helper and retain source revision, Xcode, architecture, and relocated-path validation. UI target app paths are validated too.

The Swift warning budget remains enforced on the app/UI build output. That scheme runs first and writes a separate log, avoiding broadening the budget to include unit-test diagnostics. The nightly compilation-cache seeder uses the same build helper and settings.

The prior artifact-reuse investigation estimated about 12 minutes saved in the lag job after download. Net savings are unmeasured: adding UI compilation to admission increases producer time and artifact size. This PR's hosted build, UI execution, and timing are the remaining validation.

Related: #13160 changes cache handling in the same workflow and may need reconciliation; #13165 removes the separate notification build. No changes to the release build or test assertions.

Testing

Added failing UI artifact-handoff tests first, then the implementation in a separate commit.
Seven artifact tests pass, covering producer/consumer relocation, revision/toolchain/architecture mismatch, missing UI bundles, and unscoped UI app paths.
Compilation helper tests pass with all three schemes and separate app/UI warning capture; package-resolution retry tests remain green.
Passed CI change-area tests, self-hosted runner guards, Swift warning-budget tests, reusable-workflow permissions, actionlint, shell syntax checks, and git diff --check.
No app runtime code changed. Hosted macOS CI must validate the real Xcode UI products and execution before merge; the fixture tests do not establish that or the timing benefit.

Summary by cubic

Reworks the macOS runtime regression CI job to reuse the app and UI test products built during compile admission, instead of resolving dependencies and building the app again.

Changes

Compile admission now builds the cmux UI-test scheme alongside the unit-test schemes and packages its manifest and UI runner bundle into the shared artifact.
The runtime regression job restores the exact checksummed artifact and runs display tests with test-without-building, removing its separate dependency resolution and app build.
Both artifact consumers share scripts/ci/restore-app-host-test-product.sh, and validation now covers UI target app paths.
The Swift warning budget stays scoped to app/UI output because that scheme logs separately from unit-test diagnostics.
The nightly compilation-cache seeder uses the same build helper and settings.

Notes

Net timing savings are unmeasured; compile admission and the artifact both get bigger.
Hosted macOS CI still needs to validate the real Xcode UI products and timing, since fixture tests don't cover them.
This may need reconciliation with ci: stop pull requests evicting the main cache seeds, and add an optional Warp cache store #13160, which changes cache handling in the same workflow.

^{Written for commit 7522486. Summary will update on new commits.}

Summary by CodeRabbit

CI Improvements
- Streamlined macOS validation by reusing compiled test products across verification stages.
- Added validation for Swift warning budgets and archived test-product integrity.
- Display and browser UI regression checks now run from prebuilt test bundles.
Test Coverage
- Expanded coverage for app, unit, numeric-locale, and UI test configurations.
- Added validation for UI test bundle paths and product restoration scenarios.
Reliability
- Improved detection of missing, invalid, or incorrectly scoped test artifacts.

github-actions · 2026-09-20T07:21:14Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

coderabbitai · 2026-09-20T07:21:57Z

Warning

Review limit reached

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 1a364871-1dc1-49a5-bfdc-10374bbd0833

📥 Commits

Reviewing files that changed from the base of the PR and between 17c2498 and 7522486.

📒 Files selected for processing (10)

.github/workflows/ci.yml
scripts/ci/app_host_test_products.py
scripts/ci/compile-app-host-test-product.sh
scripts/ci/restore-app-host-test-product.sh
scripts/ci/run-display-ui-regressions.sh
tests/test_app_host_test_products.py
tests/test_ci_change_areas.py
tests/test_ci_self_hosted_guard.sh
tests/test_ci_swift_warning_budget.sh
tests/test_ci_test_compilation_cache_seed.sh

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 56448bf9-6b81-4e5e-bdd8-aa63d00b0e64

📥 Commits

Reviewing files that changed from the base of the PR and between 02cd032 and 17c2498.

📒 Files selected for processing (10)

.github/workflows/ci.yml
scripts/ci/app_host_test_products.py
scripts/ci/compile-app-host-test-product.sh
scripts/ci/restore-app-host-test-product.sh
scripts/ci/run-display-ui-regressions.sh
tests/test_app_host_test_products.py
tests/test_ci_change_areas.py
tests/test_ci_self_hosted_guard.sh
tests/test_ci_swift_warning_budget.sh
tests/test_ci_test_compilation_cache_seed.sh

Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

CI now compiles app and UI test products during compile admission, validates their manifests, restores shared artifacts, and runs display and runtime tests without rebuilding in the runtime job.

Changes

Shared app-host test product

Layer / File(s)	Summary
Compile admission products and warning logs `.github/workflows/ci.yml`, `scripts/ci/compile-app-host-test-product.sh`, `tests/test_ci_test_compilation_cache_seed.sh`, `tests/test_ci_change_areas.py`, `tests/test_ci_swift_warning_budget.sh`	The admission build now includes `cmux`, `cmux-unit`, and `cmux-numeric-locale`. Each scheme writes a separate build log. Swift warning validation uses the app build log.
Manifest coverage and product restoration `scripts/ci/app_host_test_products.py`, `scripts/ci/restore-app-host-test-product.sh`, `tests/test_app_host_test_products.py`, `.github/workflows/ci.yml`, `tests/test_ci_change_areas.py`	The product manifest includes the UI test bundle and validates `UITargetAppPath`. The new restoration script verifies, extracts, stages, and restores the compiled product. Tests cover missing and unscoped UI paths.
Shared product runtime verification `.github/workflows/ci.yml`, `scripts/ci/run-display-ui-regressions.sh`, `tests/test_ci_self_hosted_guard.sh`, `tests/test_ci_change_areas.py`	`tests-build-and-lag` restores the admission artifact instead of building locally. Display and browser tests run from `CMUX_UI_XCTESTRUN` with `test-without-building`.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant CompileAdmission
  participant ArtifactStorage
  participant TestsBuildAndLag
  participant RunDisplayUIRegressions
  CompileAdmission->>ArtifactStorage: publish compiled app-host test product
  TestsBuildAndLag->>ArtifactStorage: download product
  TestsBuildAndLag->>TestsBuildAndLag: restore product
  TestsBuildAndLag->>RunDisplayUIRegressions: pass CMUX_UI_XCTESTRUN
  RunDisplayUIRegressions->>RunDisplayUIRegressions: run prebuilt UI tests

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 5.88% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 9 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (24 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: reusing compiled app and UI tests for runtime regressions.
Description check	✅ Passed	The description provides a detailed summary, rationale, testing results, validation limits, and related work. It omits the Demo Video, Review Trigger, and Checklist sections from the template, but the…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input	✅ Passed	PASS: The PR changes only CI workflows, app-host test-product packaging/restoration, UI regression invocation, and related tests. It does not change Cloud terminal creation, cmux-tui transport, manual…
Cmux Swift Actor Isolation	✅ Passed	The reviewed range changes only CI configuration, Python helpers, shell scripts, and CI tests. `git diff --name-only` reports no `.swift` files, and the added patch contains no Swift actor-isolation d…
Cmux Swift Blocking Runtime	✅ Passed	PASS: The reviewed range changes only one YAML file, three Python files, and six shell files. It introduces no production Swift file changes. The patch contains no added blocking or timing primitives …
Cmux Browser Automation Off-Main	✅ Passed	The pull request does not change browser socket automation routing. No Swift source or policy-test path changed, and both policy source-of-truth files remain unchanged. The only browser-related change…
Cmux Expensive Synchronous Load	✅ Passed	PASS: The authoritative PR diff changes no production Swift files. It changes only CI workflow, CI scripts, Python, shell tests, and test files. The patch also contains none of the custom-check load o…
Cmux Cache Substitution Correctness	✅ Passed	PASS: The authoritative PR diff changes only CI workflow/configuration, Python CI/tests, and shell CI/tests. It contains no changed production Swift, TypeScript, or JavaScript files, so the cache-subs…
Cmux No Hacky Sleeps	✅ Passed	No covered hacky sleep was introduced or worsened. The changed shell scripts add no sleep, timer, polling, or fixed backoff logic. `restore-app-host-test-product.sh` has no delay logic. `compile-app-h…
Cmux Algorithmic Complexity	✅ Passed	No explicit algorithmic-complexity failure is introduced. The diff changes CI workflows and CI scripts; it does not change Swift, TypeScript, JavaScript, or application runtime code. The new shell bui…
Cmux Swift Concurrency	✅ Passed	PASS: The pull request changes only CI, shell, Python, and test files. The authoritative diff contains no changed Swift paths, no added Swift concurrency constructs, and no added legacy async patterns…
Cmux Swift `@Concurrent`	✅ Passed	PASS — The reviewed range changes only CI YAML, shell scripts, and Python/tests; it contains no `.swift` files or added/removed `async`, `nonisolated`, `@concurrent`, `MainActor`, or actor-isolation c…
Cmux Swift Package Boundaries	✅ Passed	PASS — the pull request changes only CI workflow configuration, Python helpers/tests, and shell helpers/tests. The authoritative diff contains no changed `.swift` files, Swift package manifests, or `S…
Cmux Swiftpm Lockfiles	✅ Passed	PASS. The review-scoped diff changes CI, scripts, and tests only. It does not change any `Package.swift`, `Package.resolved`, `.gitignore`, or Xcode project package-reference file. The root Xcode `Pac…
Cmux Swift Logging	✅ Passed	PASS: The authoritative PR diff changes only CI workflows, CI scripts, and tests. It contains no changed Swift files or production source paths. The only token match in added lines is shell `find ... …
Cmux User-Facing Error Privacy	✅ Passed	PASS: The pull request changes only CI workflows, CI restoration/build scripts, and tests. The new messages and errors are CI/developer diagnostics, such as `CMUX_UI_XCTESTRUN is required`, manifest v…
Cmux Full Internationalization	✅ Passed	PASS: The authoritative diff changes only CI workflow, CI scripts, and CI tests. It adds no Swift source, app string catalog, Info.plist localization, web UI, API response, markdown, changelog, or loc…
Cmux Swiftui State Layout	✅ Passed	PASS: The pull request changes only CI YAML, shell scripts, Python helpers, and tests. The authoritative diff contains no Swift or SwiftUI files and introduces none of the state, layout, row-store, or…
Cmux Architecture Rethink	✅ Passed	PASS: The custom check is not applicable. The pull request changes only CI YAML, shell scripts, Python helpers, and tests; the reviewed diff contains no Swift or Objective-C source changes. No added S…
Cmux Swift Auxiliary Window Close Shortcuts	✅ Passed	PASS: The reviewed range changes only CI YAML, Python, shell scripts, and tests. It changes no Swift source and adds or modifies no NSWindow, NSPanel, NSWindowController, SwiftUI Window, WindowGroup, …
Cmux Source Artifacts	✅ Passed	All 10 changed paths are tracked workflow, source, script, or test files. The only added path, `scripts/ci/restore-app-host-test-product.sh`, is a hand-written CI restoration script. The diff adds no …
Cmux No Test Or Debug Seam In Production Source	✅ Passed	The authoritative pull-request diff changes 10 CI, shell, Python, and test files. It contains no changed Swift file under a production `Sources/` path. Therefore, the custom check has no applicable pr…

Full details: Docstring Coverage

Explanation

Docstring coverage is 5.88% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 9 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

greptile-apps · 2026-09-20T07:24:42Z

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule violations identified.

Summary

The PR extends the compile-admission artifact to include the cmux UI-test products and reuses that artifact for display/runtime regressions instead of rebuilding the app.

Builds the UI, unit, and numeric-locale schemes in a shared DerivedData directory.
Packages and validates the UI-test manifest, runner bundle, target app, and existing app-host products.
Centralizes checksummed artifact restoration in a shared helper.
Runs display regressions from the restored .xctestrun manifest with test-without-building.
Keeps the Swift warning budget scoped to the app/UI scheme’s separate build log.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[macOS compile admission] --> B[Build cmux UI scheme]
    A --> C[Build unit-test schemes]
    B --> D[Shared DerivedData]
    C --> D
    D --> E[Validate manifests and product paths]
    E --> F[Checksummed artifact]
    F --> G[App-host test shards]
    F --> H[Runtime regression job]
    G --> I[Restore and relocate products]
    H --> I
    I --> J[test-without-building]

_{Reviews (3) · Last reviewed commit: "Merge origin/main into ci-reuse-build-fo..."}

4b18fc9 ci: track package test helper inputs (manaflow-ai#13188) b210493 Merge pull request manaflow-ai#13181 from manaflow-ai/ci-focused-app-host-tests ced163c Merge pull request manaflow-ai#13180 from manaflow-ai/ci/remote-tmux-crash-diagnostics 9faf726 Merge pull request manaflow-ai#13178 from manaflow-ai/ci-reject-incomplete-test-runs 14c3ca1 Merge pull request manaflow-ai#13166 from manaflow-ai/ci-reuse-build-for-runtime-regressions 7c3574a ci: let the pre-merge Release check compile arm64 only (manaflow-ai#13195) 887839a ci: drop a stalled GhosttyKit download and resume it (manaflow-ai#13197) 70a244d Merge pull request manaflow-ai#13177 from manaflow-ai/ci-fast-static-preflight fcad43f build: read Xcode projects with Foundation and drop XcodeProj and PathKit (manaflow-ai#13111) 76d80b1 ci: skip Release and its helper for test-only pull requests (manaflow-ai#13122) fdc63e9 Merge pull request manaflow-ai#13176 from manaflow-ai/ci-reuse-queue-build-products 3162fee test: split an expression Xcode 27 cannot type-check (manaflow-ai#13126) d10aa64 test: use consistent XCTest imports to stop compiler diagnostic flood (manaflow-ai#13163) 1b69bf9 test: stop real-Git reftable tests depending on a 2s wall clock (manaflow-ai#13186) cad333b Merge origin/main into ci-fast-static-preflight 7522486 Merge origin/main into ci-reuse-build-for-runtime-regressions 43210e1 Bound automatic terminal titles before session persistence (manaflow-ai#13009) 674a0db ci: retire Depot macOS runners (manaflow-ai#13162) f48ef36 Merge pull request manaflow-ai#13183 from manaflow-ai/ci-early-cli-smoke ebbb17f ci: skip app-host teardown when setup never started (manaflow-ai#13179) 5fb6d8c Merge pull request manaflow-ai#13168 from manaflow-ai/ci-cache-r2-store a348064 ci: skip compile admission when an earlier run compiled the same build inputs (manaflow-ai#13139) 88e102c reload: let a reused checkout keep one warm DerivedData across tags (manaflow-ai#13131) 7a049e9 Merge origin/main into ci-reuse-build-for-runtime-regressions cd05c6e Merge origin/main into ci-fast-static-preflight 5cf41fa Merge origin/main into ci-reuse-queue-build-products 5a6322e test: guard early CLI smoke ordering 0438552 fix: pass R2 public URL through workflow environment 0716c59 test: bound app-host replay subprocesses 2e0b9b5 ci: terminate cancelled focused discovery 14bbad4 ci: keep R2 public URL configuration inside the cache actions e342c67 ci: make focused run discovery cancellable cf3984b test: avoid hard timeout in app-host classifier replay 251b050 ci: allow privileged crash report collection 7d9a7f2 Merge main after landing cache backend and suite policy b6853ee ci: allow manual cache-only seeding for R2 rollout 81d3026 test: require manual cache seeding to skip app publication 6980f8e ci: harden remote tmux diagnostics collection c24d77f ci: publish R2 cache pointers conditionally and repair failed writes 2432805 test: cover R2 pointer repair and out-of-order saves 4779d01 ci: continue past unusable build artifact candidates 9dd1579 test: reproduce corrupt candidate blocking product reuse bfb43f5 ci: check CLI version and help before app-host fan-out f2b0fae docs: use an existing suite in focused launcher example 2c04b6a ci: drain tar streams portably with BSD tar c5e1d59 ci: pin focused tests to a commit and track the requested run 1a44bde ci: consume tar padding when restoring zstd caches 31d4fd9 test: cover padded R2 archives on macOS a46567a ci: isolate R2 cache writes from release credentials ce26a8e test: require early CLI smoke gate to propagate probe failures 18a67fb test: reproduce focused launcher revision and run attribution bugs 4dd543e test: require cache-only R2 credentials for cache saves 2c8412c ci: make product reuse attempt-safe and bound archive expansion 59fb526 ci: preserve remote tmux mirror crash diagnostics d8107e4 test: cover artifact reruns, expansion limits and producer source checks 28a03e3 ci: reject interrupted app-host runs despite later passing summaries 302551d test: reproduce false-green app-host timeout and restart runs bc3a63a ci: reject invalid static inputs before expensive validation baf65d9 test: require successful static preflight before macOS admission 6cea5f0 ci: fall back when build identity cannot be established 7632c7e ci: reuse compatible compiled products in merge groups 8be0c54 ci: add an R2 bucket as a cache store every runner can read 25f50c3 test: behaviour of an R2-backed cache store script 17c2498 ci: reuse compiled app and UI products for runtime regressions 6ed96f4 test: require UI products in the shared CI build artifact 230ad52 ci: drop a timeout note about a DerivedData cache that no longer exists b5ec5cc ci: stop restoring DerivedData in pull request jobs a42ad38 Merge remote-tracking branch 'origin/main' into ci-cache-backend-switch 477fb0f ci: choose the cache store per dispatched run, and cover the nightly app build 03363d7 ci: let a repository variable move the seeded caches to the Warp store f791f87 ci: pull request jobs restore caches and never save them 5b65bb1 test: pull request jobs must restore caches read-only

teamleaderleo added 2 commits September 20, 2026 00:17

test: require UI products in the shared CI build artifact

6ed96f4

ci: reuse compiled app and UI products for runtime regressions

17c2498

teamleaderleo requested a review from lawrencecchen as a code owner September 20, 2026 07:20

teamleaderleo enabled auto-merge September 20, 2026 07:22

This was referenced Sep 20, 2026

[RFC] CI structure: thin router, reusable platform workflows, merge queue, failure ratchet #13095

Open

ci: reuse identical compiled products in the merge queue #13176

Merged

teamleaderleo added this pull request to the merge queue Sep 20, 2026

teamleaderleo mentioned this pull request Sep 20, 2026

Merge queue and CI cost: admin-only settings to finish the rollout #13182

Closed

6 of 10 tasks

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Sep 20, 2026

teamleaderleo added 2 commits September 20, 2026 03:13

Merge origin/main into ci-reuse-build-for-runtime-regressions

7a049e9

Merge origin/main into ci-reuse-build-for-runtime-regressions

7522486

teamleaderleo merged commit 14c3ca1 into main Sep 20, 2026
37 of 38 checks passed

Navigation Menu

Uh oh!

ci: reuse the compiled app and UI tests for runtime regressions - #13166

ci: reuse the compiled app and UI tests for runtime regressions#13166
teamleaderleo merged 4 commits into
mainfrom
ci-reuse-build-for-runtime-regressions

teamleaderleo commented Sep 20, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

github-actions Bot commented Sep 20, 2026

Uh oh!

coderabbitai Bot commented Sep 20, 2026 •

edited

Loading

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

greptile-apps Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

teamleaderleo commented Sep 20, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Summary by cubic

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Sep 20, 2026

Uh oh!

coderabbitai Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

greptile-apps Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teamleaderleo commented Sep 20, 2026 •

edited by cubic-dev-ai Bot

Loading

coderabbitai Bot commented Sep 20, 2026 •

edited

Loading

greptile-apps Bot commented Sep 20, 2026 •

edited

Loading