ci: run E2E tests on the runner that built them

[teamleaderleo]

An E2E dispatch used to queue for a macOS runner twice. The build job compiled (or adopted) the app-host and UI test product and uploaded it. Then the test job queued again for a runner of the same label, downloaded the product and ran the tests. With the pools saturated, that second wait decided how long a run took. Across the 20 successful runs on 2026-09-25, the test job itself ran for 2–3 min. The ~11 runs from 09:19Z on then waited 1.3–96 min for its runner. Run 36130355245 finished its build at 12:14, and its test job was still queued when it was cancelled at 12:26.

Now build runs the tests itself, on the runner that just built the product.

• Test steps move into a composite action. They live in .github/actions/e2e-run-tests and cover the virtual display, ffmpeg and TCC, the capture preflight, bun and tmux, the product restore into a RUNNER_TEMP DerivedData, the app-host home, selector resolution, the test run, recording, uploads, the summary and cleanup. The always() upload and cleanup steps keep their if:.

• Where the tests run:

  • build runs the action after it uploads the product and cleans the canonical DerivedData. It restores from the archive it just packaged (CMUX_PRODUCT_FROM_PRODUCER, the same path compile admission uses for changed suites), so the tests run the exact uploaded bytes.
  • test stays as the fallback. It runs only when needs.build.outputs.tested != 'true', and CI_E2E_TEST_IN_BUILD=0 hands the tests back to it.

• Old refs still work. Both jobs sparse-check the action out at github.workflow_sha into .e2e-workflow, so a dispatch (or bisect probe) of a ref older than the action still runs its tests. The scripts the action calls come from the tested revision, as before.

• Test environment. The action exports CMUX_CI_APP_HOST_ISOLATION_REQUIRED=1, CMUX_APP_HOST_SHARD=1 and CMUX_APP_HOST_CAPTURE_XCRESULTS=1 after the compile, wherever the tests run. CMUX_CI_XCODE_APP is the build's own pin.

• Timeout. build's timeout is 2 × job_timeout, since it now covers compiling and testing. The filter job computes it and rejects a non-integer value.

• Product upload stays. Later dispatches adopt it. Since build now ends after the tests, e2e_sibling_build.py and reuse_app_host_products.py count an E2E producer as done once Upload the compiled test product has succeeded. A sibling no longer waits out another run's tests, and a failing test no longer makes an intact product unadoptable.

• Bisect verdicts. main_regression_bisect.py reads a probe's verdict from failed top-level step names, and the jobs API doesn't list a composite's inner steps. So both jobs name the action step Run selected tests, and a top-level Resolve selectors against the built tests step fails when the action recorded an unresolvable selector. classify() now checks that step first.

• Cleanup after a skipped preparation. When the build job has removed its canonical DerivedData, it clears CMUX_DERIVED_DATA_PATH. A failure that skips the tests' own preparation (a failed screen-capture preflight, as in run 36168944875) then leaves their always() cleanup nothing to refuse.

Guard tests that read the old test job now read the action. The app-host home, pipe-capture and no-masking guards also scan composite actions, so the E2E steps can't drop out of their coverage silently.

Known cost: the Cache Swift packages post-save only runs when the job succeeds, so a run whose tests fail no longer saves a SwiftPM cache miss. The E2E compilation cache is saved by an explicit step before the tests and is unaffected.

Owned Macs: glaeda's runner hook gives test-e2e's build and test jobs the unknown-job "compile" class, which never takes the mini's gui token. That was already true of the old test job. The fix belongs in glaeda and is being done separately.

Verification

• Local guard tests: all 170 linux-guard tests from tests/test-execution.toml ran on this branch and on main at the same base. The results match, apart from test_swift_testing_suite_timeout.py, which failed on main and passed here.
  • Seven tests fail identically on this Mac on both, including test_ci_e2e_compilation_cache.py::test_a_missing_manifest_fails_instead_of_silently_compiling (macOS bash 3.2, ONLY_TESTING[@] under set -u).
  • A mutation check confirmed the no-masking guard now rejects continue-on-error inside the action.
• CI dispatches used this branch's workflow at main's 4f0ac55da15:
  • cmuxTests, 36168890047, passed. Attempt 1 was refused by an owned Mac; the rescue re-ran it on blacksmith-6vcpu-macos-26. There, build queued 2.1 min and ran 10.4 min: a 5.7 min compile, then a 0.8 min in-job test step. The product restore took 14 s. WindowTitleTemplateTests executed 12 tests. test was skipped, so no second macOS queue.
  • UI, video off, 36171193999. Restore, virtual display, automation mode, app launch and xcresult capture all ran on the build runner. The test failed on "Failed to activate application … Running Background", which require_selected_test_execution.sh rightly rejects. That's the open Blacksmith activation problem (test-e2e lane: XCUITest activation fails on both default and warp runners (no logged-in GUI user) #8588, cmuxUITests that launch the app report false passes in CI (XCTExpectFailure swallows the launch abort) #9667), not this change.
  • UI, video on, 36168944875. It failed at "Verify screen capture before dependency setup". Main's separate test job fails the same way on these pools today (36124174448, 36125021766, 36124921773).
  • Not proven: the video recording path, and UI tests on an owned Mac (the only owned attempt was refused at runner setup).

Before and after, single-class runs. Across the 20 successful runs on 2026-09-25, build ran 6–22 min. test ran a further 2–3 min, most of it checkout, product download and restore, after a second queue of 1.3–96 min from 09:19Z onward. With this change there is one queue and one job. In the cmuxTests run above, the tests added 0.8 min to build, against the 2–3 min the old test job spent. Per run, that saves about 2 macOS runner-minutes (the second checkout and download) and the entire second queue, which was the part that decided the wall time on saturated pools.

• Review: a read-only subagent reviewed the change. Its one regression (the bisect step names) is fixed in the second commit.

— Quillmoor g1 🌊

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 6a6321f5-e10a-4abb-ad25-e55b481910cb

📥 Commits

Reviewing files that changed from the base of the PR and between ce2a516 and d42b52a.

📒 Files selected for processing (13)

• .github/actions/e2e-run-tests/action.yml
• .github/workflows/test-e2e.yml
• scripts/ci/e2e_sibling_build.py
• scripts/ci/main_regression_bisect.py
• scripts/ci/reuse_app_host_products.py
• tests/test_ci_app_host_home_isolation.py
• tests/test_ci_app_host_pipe_capture.py
• tests/test_ci_e2e_compilation_cache.py
• tests/test_ci_focused_test_selectors.py
• tests/test_ci_main_regression_bisect.py
• tests/test_ci_self_hosted_guard.sh
• tests/test_e2e_sibling_build.py
• tests/test_reuse_app_host_products.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_paid_and_downstream_jobs_checkout_only_the_resolved_sha (main.IOSWorkflowDispa tchRefTests.test_paid_and_downstream_jobs_checkout_only_the_resolved_sha)/ test_paid_and_downstream_jobs_checkout_only_the_resolved_sha (main.IOSWorkflowDispa tchRefTests.test_paid_and_downstream_jobs_checkout_only_the_resolved_sha)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[github-actions]

Merge receipt for d42b52a3ff: every check was green at merge (12 verified; 15 skipped by policy). Full suite runs on main after merge.