Skip to content

ci: advisory fleet test-compile check for the test targets a change reaches - #23423

Merged
teamleaderleo merged 6 commits into
feat-cmux-nextfrom
ci-speed/compile-check-tests
Oct 21, 2026
Merged

teamleaderleo merged 6 commits into
feat-cmux-nextfrom
ci-speed/compile-check-tests

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

fleet/compile-check compiles only the app executable, which links no test target, and merge_compile_guard.py accepts its pass in place of a queued Release compile. Two PRs passed it today and then broke feat-cmux-next's test build: #23360 (a throwing call inside #expect without try, CmuxNextActionsTests) and #23163 (a FakeClient missing the requirement #21258 added, CmuxNextAppTests). On #23360 every Mac job that builds tests was still queued when it merged.

This adds an advisory check as a bridge, and leaves the merge gate alone:

  • fleet/compile-check is unchanged: still the Linux app compile, same timing, same github-actions[bot] status. fleet-compile-check.yml and merge_compile_guard.py are byte-identical to feat-cmux-next.
  • New fleet-test-compile.yml runs on pull requests into feat-cmux-next that touch Packages/ or vendor/. A Linux job plans with scripts/ci/compile_test_plan.py over the merge commit's diff: the CmuxNext test targets that depend on a changed target in the existing SwiftPM target graph (cmux_next_route.route, the swift test tier's routing), every one for a manifest or graph change, plus the macOS package lane's packages whose sources or path dependencies changed. An empty plan passes on that Linux runner in seconds and queues no Mac. A plan runs package-test-lane.sh compile-tests BASE as a light fleet step, which builds each planned CmuxNext test target with swift build --target (no link, no run) and a lane package with --build-tests. The check, fleet/test-compile (advisory), goes red only when a planned test target fails to compile. The lane prints its failure marker as soon as a target fails, so a later package's setup failure can't hide it. Anything else (queue or run timeout, setup, the client) is a warning, and the job's 55-minute timeout outlasts the client's 20-minute queue plus 25-minute run timeouts, so a slow fleet never reads red. If SwiftPM refuses --target, the package falls back to one --build-tests in place of its remaining targets.
  • Nothing waits for it: the guard reads only the Release compile and fleet/compile-check; gh-merge-green is main-only and this workflow runs only on feat PRs.

It's a bridge until the Linux pool compiles test targets. The pool's darwin SDK can't find Testing today (job 1382fbdb06c0fc7c023643d3 at df5e2eb: no such module 'Testing'); cmuxterm-hq #2750 adds Xcode's platform search paths and reaches the pool with the next image bake. The per-target pool argument is manaflow-ai/cmuxterm-hq#2759.

Testing

Red/green on a fleet Mac with the build the new phase runs (the reviewed package-test-lane.sh suite phase, which runs swift build --build-tests over CmuxNext before one suite):

head result fleet step
#23360's head 4f867f5e57c2d943ac1f4e9743d42cc89730e683 red, exit 1: PaletteMenuVocabularyTests.swift:74:106 and :79:101, "call can throw, but it is not marked with 'try'" ed72e6e6f72d215befe9bb9b (aws-m4pro-8)
feat-cmux-next df5e2ebb1c03db048843083530e86feeda2c1b2a green: Build complete! (89.11s), then CmuxNextActionsTests (136 tests) passed 54d316aebe3d339a9a2126eb (aws-m4pro-7)

cmux-ci run --class light --script scripts/ci/package-test-lane.sh --ref <sha> --arg=suite --arg=Packages/macOS/CmuxNext --arg=CmuxNextActionsTests; cmux-ci log <step> on the tailnet prints each log. Each ran about 1.5 to 2 minutes after 11 minutes in the light-class queue, which is why this is advisory and off the merge gate.

tests/test_ci_compile_test_plan.py (linux-guard) was committed red first (ec4b51c) and passes: #23360's diff selects CmuxNextActionsTests and its reverse dependents and nothing that doesn't reach CmuxNextActions; a new CmuxNextAppTests file selects only that target; the manifest selects every target; a CMUXAuthCore change selects it, its lane dependents and the CmuxNext targets that use it; iOS-only, docs, CI and app-host changes plan nothing; the phase refuses a bad merge base. The same file runs the compile-tests phase with a fake swift (committed red first in a9201f8): a target that doesn't compile exits 1 with the marker; an earlier target failure stays red when a later package's setup fails; a setup failure alone has no marker; a refused --target falls back to --build-tests once per package; an empty plan builds nothing. It also runs the workflow step with a fake client to check how exits map to red, warning or green, and that the job timeout outlasts the client's. cmux-next.yml now runs it with the merge guard tests. tests/test_merge_compile_guard.py adds that a red or pending advisory check never holds a merge or stands in for the Release compile, and that the advisory workflow is not the fleet check the guard trusts. Passing on Linux (huge-blue): those two, test_ci_cmux_ci_client_guarded, test_ci_workflow_guards_are_wired, test_supplychain_security, test_runner_label_policy, test_ci_pr_runner_pool, test_ci_fork_runner_routing, test_ci_merge_queue_required_checks, test_ci_workflow_path_filter_parity, test_ci_workflow_run_sources, the reusable-workflow and cache guards, package-test-lane-suite-args.test.sh; actionlint; prepush-gate.py.

Observed / Not run

  • Ran: the two fleet steps above, the pool probe (no such module 'Testing'), the tests listed above, compile_test_plan.py on cmux-next palette: titles end in "…" exactly when the action asks first, outside Phone #23360's diff (12 CmuxNext test targets including CmuxNextActionsTests and CmuxNextAppTests).
  • Not run: the compile-tests phase and fleet-test-compile.yml on the fleet. A changed allowlisted script runs on the fleet only after it lands (hq step_reviewed.go), so the first real run is the first feat PR touching Packages/ after merge. Unverified there: swift build --target <test target>; if SwiftPM refuses it, the phase falls back to --build-tests for that package.
  • Known: next_autoland holds a PR on any red check, including this advisory one, until teamleaderleo/leo-workspace#746 (skip checks named ... (advisory)) lands. Expected red here: this PR's own swift-package-tests fleet step is refused until package-test-lane.sh lands, as for every allowlisted-script change.

Changelog

none

Agent session: cosmic-gray

🤖 Generated with Claude Code

teamleaderleo and others added 2 commits October 21, 2026 05:51
…can break (red)

fleet/compile-check compiles only the app executable, so #23360 (a
throwing call in #expect without try, CmuxNextActionsTests) and #23163
(a FakeClient that missed #21258's requirement, CmuxNextAppTests) passed
it and broke feat-cmux-next's test build. This test pins the plan the
check must compile and the new package-test-lane.sh compile-tests
argument check; neither exists yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
…ches

fleet/compile-check compiled only the app executable, which links no test
target, and merge_compile_guard accepts it in place of a queued Release
compile. #23360 (CmuxNextActionsTests) and #23163 (CmuxNextAppTests)
passed it while every Mac job that builds tests sat queued, and broke the
base's test build.

The workflow now reads the head's merge base with feat-cmux-next from the
compare API and runs `package-test-lane.sh compile-tests MERGE_BASE` on a
fleet Mac beside the Linux app compile; success needs both. The plan
(scripts/ci/compile_test_plan.py) is the CmuxNext test targets that depend
on a changed target in the existing SwiftPM target graph (the routing
cmux_next_route.py gives the swift test tier), plus the macOS package
lane's packages whose sources or path dependencies changed. Each CmuxNext
test target builds with `swift build --target`, so nothing links or runs.
The Linux pool cannot do it: its SDK has no Testing module.

A head whose package-test-lane.sh predates the phase gets no verdict
(error), with "merge feat-cmux-next and dispatch again".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by ci-speed on cosmic-gray · session

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 3a2851b: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23423 -f ref=ci-speed/compile-check-tests -f platform=macos -f backend_mode=local

web and size checks: green

@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 7 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp1karsm2b/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpgs7o6ys4/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
scripts the package test job runs are not global inputs: ['scripts/ci/
compile_test_plan.py']
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-rkwre created for this PR by the cmux coordinator.

… is the app compile again

coord-deputy: an ~11 min Mac queue on every PR's merge gate costs too much.
fleet-compile-check.yml and merge_compile_guard.py are back to feat-cmux-next's
(the Linux app compile, same status semantics).

fleet-test-compile.yml runs on pull requests into feat-cmux-next that touch
Packages/ or vendor/: a Linux job plans with compile_test_plan.py over the
merge commit's diff; an empty plan passes there in seconds and queues no
Mac; a plan runs `package-test-lane.sh compile-tests BASE` as a light fleet
step, and the check `fleet/test-compile (advisory)` goes red only when a
planned test target does not compile. It is a bridge until the Linux pool
image carries cmuxterm-hq #2750 (Testing search paths).

Tests: the guard ignores the advisory check, red or pending, and it never
stands in for the Release compile; CI and app-host changes plan nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / defaults-snapshot is red on 17 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

@teamleaderleo teamleaderleo changed the title ci(fleet compile-check): also compile the test targets the change reaches ci: advisory fleet test-compile check for the test targets a change reaches Oct 21, 2026
teamleaderleo and others added 2 commits October 21, 2026 06:41
…llback per package; advisory verdicts

Red on c150a5c: a target failure followed by a later package's setup
failure exits before the failure marker, so the check only warns; a refused
--target falls back to --build-tests once per target; the job's 45 min
timeout is shorter than the client's queue plus run timeouts, so a slow
fleet ends red; exit 128 after the marker maps to a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
…er does

- package-test-lane.sh compile-tests prints the failure marker as soon as a
  target fails, so a later package's setup failure cannot hide it.
- A refused --target falls back to --build-tests once per package, in place
  of its remaining targets.
- fleet-test-compile.yml: red on the marker whatever the exit; the job
  timeout (55 min) outlasts the client's queue plus run timeouts (45 min),
  so a timeout is a warning, never red.
- cmux-next.yml runs tests/test_ci_compile_test_plan.py with the merge guard tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent, correctness first): SHIP at b96188a

…age test job

package-test-lane.sh compile-tests runs it, so test_ci_select_package_tests.py
requires it in GLOBAL_INPUTS (red on b96188a: "scripts the package test
job runs are not global inputs").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent, correctness first): SHIP at 3a2851b. The SHIP at b96188a still stands: the only change since then adds scripts/ci/compile_test_plan.py to GLOBAL_INPUTS in select_package_tests.py, which fixes this PR's own 'Validate package test selection' red. The defaults-snapshot check-menu-actions failure (FileExplorerSectionView.swift) is red on base 68a3517 too.

@teamleaderleo
teamleaderleo merged commit dbc1528 into feat-cmux-next Oct 21, 2026
@teamleaderleo
teamleaderleo deleted the ci-speed/compile-check-tests branch October 21, 2026 14:02
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

teamleaderleo added a commit that referenced this pull request Oct 21, 2026
…id, reload shim test, fleet-test-compile runners) (#23553)

* fix(cmux-next media): a display step's shot passes the app's pid to the screen route

Screen.use_display takes the app's pid since #22046 (it finds the window
server id by owner pid), and run_tour passes it; display_capture still
called use_display() with no argument, so every `display` tour step died
with a TypeError. tests/test_cmux_next_tour.py's display-step cases were red
on feat-cmux-next (preflight guard group).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

* fix(ci): the reload DerivedData test loads the shim writer from scripts/lib/cmux-dev-shim-install

reload.sh now sources write_dev_cli_shim and reload_socket_is_live from
scripts/lib/cmux-dev-shim-install (the one source of the managed shim), so
the test's awk extraction from reload.sh found nothing and failed with
"write_dev_cli_shim not found in reload.sh" on feat-cmux-next (preflight
guard group). The test sources the library as reload.sh does, and checks
reload.sh still sources it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

* fix(ci): fleet-test-compile.yml runs on Blacksmith, not a GitHub-hosted label

#23423's plan job and its empty-plan compile leg selected ubuntu-24.04, which
tests/test_ci_self_hosted_guard.sh refuses (a GitHub billing block must not
stop CI): "Validate macOS runner guards" was red on feat-cmux-next (preflight
guard group). Both take the standard selector: the fork branch, then
vars.LINUX_RUNNER with the Blacksmith fallback; a needed plan still takes
vars.CI_COMPILE_CHECK_GATEWAY.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

* fix(ci): fleet-test-compile.yml sends fork pull requests to Blacksmith before any runner variable

The previous commit put vars.LINUX_RUNNER (and the compile gateway) right after
the owner branch, so tests/test_ci_fork_runner_routing.py flagged both lines: a
fork pull request could reach an owned runner label. Both selectors now use the
standard form (agent-brands.yml, backend.yml): the fork pull-request branch
first, then the gateway and LINUX_RUNNER.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant