[teamleaderleo]

Builds on #28249 (merged as 38c2c9a). This branch is this PR's own 7 commits rebased onto current feat-cmux-next; none of #28249's commits remain in its diff.

What

When every owned and Blacksmith macOS label the fast tier could take is full or dead, the tier no longer waits: it runs as a thin cmux-ci client on the build-fleet gateway, and cmux-next fast tier keeps its check name and exit code.

#28249 added fleet_jobs to the placement (filtered to FLEET_CLIENT_JOBS, which it left unset, and read only through label_full_or_dead) and reserved the gateway outputs without wiring any job to them. This PR wires the fast tier with the client + report pattern swift-canary-fleet / swift-canary already use:

• fast-tier-fleet (new, name cmux-next fast tier (build fleet)): runs on needs.macos-placement.outputs.gateway_runner only when fleet_jobs lists fast-tier and placement saw an idle gateway in this attempt. permissions: {}, no checkout, no action, no secret. One step: cmux-ci run --class light --script scripts/ci/package-test-lane.sh --ref <REF> --arg=suite ... for each package of fast-tier.json's Swift entries (CmuxHomeRender: MotionTokenTests, IntentMappingTests; CmuxNext: ModelMatrixTests, PaneZoomMotionTests, CEFProfileStorageLogTests, AgentSubagentPanesFocusTests), in parallel, --no-supersede. REF is the exact commit fast-tier itself checks out: github.sha, a pull request's merge commit (as swift-canary-fleet does), so a head that breaks against the current base is still caught. On a push it is the pushed sha. It then reads cmux-ci status <step> and requires every ref in the record (and source_digest) to equal REF. Output verdict is pass, fail or unavailable. The controller's record decides: pass needs state done with exit_code 0 (and a client exit 0). A step that ended with a nonzero exit is a fail even if the client exited 0. Step ids are read as 6 to 64 hex, as the fleet tools read them. Fail, no verdict, or a step at another commit all fail the job; only a pass at REF exits 0.
• fast-tier: needs fast-tier-fleet. When that job ran (success, failure or cancelled), it runs on blacksmith-2vcpu-ubuntu-2404 (4 vCPU under CI_CLOUD_OVERFLOW_SAVED, as swift-canary does) with a 5 minute timeout. Its first step reports the client's verdict (pass + success: exit 0; fail: exit 1; anything else: exit 1 with "no verdict"), and every other step is off (CMUX_FAST_TIER_FLEET). Its runs-on no longer names gateway_runner. When the client job skipped, nothing changes.
• Push route (with ci: anchor push routing to finished fast tier #28260): fast-tier-fleet honours needs.path_route.outputs.fast_tier on pushes as well as pull requests, so a push whose diff since the last finished fast tier reaches no Mac tier skips the client as it skips the tier. test_the_client_runs_only_when_placement_listed_its_tier_with_an_idle_gateway checks the two gates agree on both events (red without the gate).
• macos-placement: FLEET_CLIENT_JOBS: fast-tier in the place step. next_macos_cascade.py is unchanged.
• Gateway room (side_lane_placement.fleet_gateway()): the client runs on gateway_runner beside GATEWAY_JOBS (app-compile-fleet, swift-canary-fleet, release-compile-fast), and the gateway is judged for those alone. Placement now keeps fleet_jobs only while the gateway has an idle runner for each listed job beyond GATEWAY_JOBS and the run's own jobs (counted as gateway_route() counts them). Otherwise it lists nothing and fast-tier keeps its route, rather than its client queueing behind another gateway job until its timeout. A re-run emits no gateway, so it lists none. Tests: GatewayRoute.test_a_fleet_client_needs_its_own_idle_gateway_runner, test_main_lists_a_fleet_job_only_with_a_gateway_runner_for_its_client (caught when the call in main() is removed), test_a_rerun_lists_no_fleet_job. No label is hard-coded for the gateway; gateway_runner comes from placement.
• No whole job on the gateway: ci: keep cmux-next fast tier off dead side queues #28249's FleetGateway.test_gateway_output_is_reserved_without_wiring_raw_mac_jobs becomes test_no_tier_runs_on_the_gateway_itself: no tier's runs-on names fleet_jobs or gateway_runner, or resolves to the gateway when placement lists it. Its check that FLEET_CLIENT_JOBS is unset is dropped, since this PR sets it.

Tiers with and without a client

• fast-tier: client added. Its Swift entries map one to one to package-test-lane.sh suite filters, the allowlisted light script. One difference: fast-tier.py runs a failed entry once more to label it FLAKY; suite mode fails on the first failure. Both are red, so the gate is the same.
• swift-test: no client. It is not a cascade job (CASCADE_JOBS is fast-tier, swift-canary, cmux-scheme-compile, release-compile), so place() never leaves it unplaced and fleet_jobs can never list it. Its route-selected suites and side steps need a checkout to plan.
• daemon-test: no client. It is not a cascade job either, and it needs the same-tree cmux-tui daemon. A fleet suite step without that daemon reports "no result", so a client could never honestly pass.

.github/runners.json classifies cmux-next.yml#fast-tier-fleet as apple-swift-tests, beside the other gateway clients (the routing table reads the gateway label as macOS).

Gateway label allowlist

The client job's id is fast-tier-fleet and its name is cmux-next fast tier (build fleet). The glaeda gateway allowlist for cmux-next.yml needs this id (the reporting job fast-tier never runs on the gateway).

Guard tests

GatewayRunsOnlyTheFleetClient in tests/test_cmux_next_checks_workflow.py:

• Every job whose runs-on can reach gateway_runner through fleet_jobs has runs-on equal to the gateway output, permissions: {}, no container, services, environment or secrets, and only run steps. Every command word is the client or a shell builtin (a shlex-based parse), with no absolute path and no eval, exec, source, command, builtin or env.
• In report mode every fast-tier step but the report is off, checked by evaluating each step's if with every earlier step skipped.
• The client step is run with PATH empty against a fake cmux-ci. It passes only on a pass at REF, and fails with the right verdict for: a suite failed, a step at another commit, a queue timeout, no worker, no client, and a REF that is not a full sha. Any non-builtin it ran would show as "command not found".
• FLEET_CLIENT_JOBS and the <tier>-fleet jobs match one to one. The tier never names gateway_runner, reports on Linux only when the client ran, and has every other step off. The report step's exit matches the client's.
• The client's suites equal fast-tier.json's gating Swift entries by package.
• BuildFleetGatewayJobsFallBack.FALLBACKS now includes fast-tier-fleet → fast-tier.

FleetClientJobs in tests/test_ci_next_macos_cascade.py: with every label full or dead, FLEET_CLIENT_JOBS puts exactly the listed jobs in fleet_jobs (written fast-tier). Nothing is listed when no client is configured, when a Blacksmith label is under its cap, when the side label has a live listener, or when there is no snapshot. The workflow lists only cascade jobs that have their <job>-fleet.

Red/green: with these test files on #28249's earlier head (d3289ff), GatewayRunsOnlyTheFleetClient + BuildFleetGatewayJobsFallBack fail (7 failures, 12 errors) and FleetClientJobs fails (1 failure, 1 error). On this branch they pass. On an earlier #28249 head (be77727), whose runs-on sent whole Mac jobs to the gateway, test_a_client_job_runs_nothing_but_the_client failed for each of those seven jobs. Two mutations of this branch are also caught: an ungated Run the fast tier step, and a git fetch added to the client step.

Same-host runs, this branch vs its base (330eebb): test_cmux_next_checks_workflow.py 173 OK (base 165 OK), test_ci_next_macos_cascade.py 125 OK (base 121 OK), test_ci_side_lane_placement.py 81 OK (base 78 OK). test_ci_runs_on_fallback.py, test_cmux_next_checkout_retry.py, test_ci_macos_xcode_selection.py, test_ci_fork_runner_routing.py and test_ci_fast_tier_anchor.py pass on both. test_ci_change_areas.py fails the same one test (test_repository_helpers_route_by_where_they_run) on both. test_ci_self_hosted_guard.sh and actionlint give the same output on both. Every test file that ci-guards.yml or the cmux-next CI groups run (502 files naming cmux-next.yml, placement, cmux-ci or runners.json) gives the same result by name on this branch and on its base.

Risk

Low. Behaviour changes only when placement lists fast-tier in fleet_jobs, which needs the placement script from the base (after #28249) and every macOS label full or dead. With no idle gateway the client skips and fast-tier takes its usual route. No new permission, secret or runner label. The client job holds no token.

Agent session: agent session

🤖 Generated with Claude Code

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 2944b6f: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28391 -f ref=lane/fast-tier-fleet-client -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 87b4c90; does not block

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4131171124701 (fe91ebac50)
• cmux NEXT nightly 1.0.0-nightly.4131261222601 (1c7090a384)
• cmux NEXT nightly 1.0.0-nightly.4131344025701 (c7caf1e32a)
• cmux NEXT nightly 1.0.0-nightly.4131523971401 (fa5830f6d5)
• cmux NEXT nightly 1.0.0-nightly.4131541733001 (49aa2917ea)

[teamleaderleo]

Owned by an agent session · session

[blacksmith-sh]

Found 13 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmpatqo0syu/summary-fail.json (live):/summary-fail.json (live):	View Logs
/tmp/tmply3g91rs/summary-missing.json (live):/summary-missing.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/ test_download_prints_the_active_profile_only	View Logs
nightly must build and hand off the macOS 26 helper and Xcode 26.5 app before publishin g/ nightly must build and hand off the macOS 26 helper and Xcode 26.5 app before publishin g	View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/ Security has no stub	View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/ CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/ CoreFoundation, missing 1: ['_CFRetain']	View Logs

...and 3 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6-astra high, next-review): BLOCK at 82a6d88

• scripts/ci/owned_pool_rescue.py:656 — assess() now considers only persistent-pool jobs when detecting running siblings. If an owned job is stuck in runner setup while a Linux or Blacksmith sibling is running, it sees no sibling and returns rescue; the rescue then cancels the entire non-PR workflow, killing healthy work. The sibling check must include all in-progress jobs.

[cubic-dev-ai]

2 issues found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_ci_next_macos_cascade.py">

<violation number="1" location="tests/test_ci_next_macos_cascade.py:2195">
P2: This fixture does not test the fresh listener case it claims to cover. Because `generated_at` is newer than the fixed `NOW`, placement treats the snapshot as unreadable and the assertion passes for the wrong reason; derive the timestamp from `NOW` instead.</violation>
</file>

<file name="tests/test_ci_fork_runner_routing.py">

<violation number="1" location="tests/test_ci_fork_runner_routing.py:1067">
P2: This exception validates only the hosted branch for a fork's own repository, not a fork PR targeting `manaflow-ai`. Require the existing fork-PR gate before allowing `vars.LINUX_RUNNER`, otherwise this guard can approve a selector that queues fork PRs on self-hosted Linux.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This fixture does not test the fresh listener case it claims to cover. Because generated_at is newer than the fixed NOW, placement treats the snapshot as unreadable and the assertion passes for the wrong reason; derive the timestamp from NOW instead.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_next_macos_cascade.py, line 2195:

<comment>This fixture does not test the fresh listener case it claims to cover. Because `generated_at` is newer than the fixed `NOW`, placement treats the snapshot as unreadable and the assertion passes for the wrong reason; derive the timestamp from `NOW` instead.</comment>

<file context>
@@ -2148,6 +2158,64 @@ def test_the_sign_runner_reads_the_same_rule(self):
+        self.assertEqual(self.place(shot=shot, live={LARGE: 1, SMALL: 10}, FLEET_CLIENT_JOBS="fast-tier").fleet_jobs,
+                         ())
+        # The side label has a listener, by a fresh health record (a stale record or snapshot reads full).
+        fresh = dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+        listening = self.shot(listening=1, held_offline={"units": 0, "cpu": 0, "livelock": 0}, as_of=fresh)
+        listening["generated_at"] = fresh
</file context>

Suggested change

	fresh = dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
	fresh = (NOW - dt.timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M:%SZ")

[cubic-dev-ai]

P2: This exception validates only the hosted branch for a fork's own repository, not a fork PR targeting manaflow-ai. Require the existing fork-PR gate before allowing vars.LINUX_RUNNER, otherwise this guard can approve a selector that queues fork PRs on self-hosted Linux.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_fork_runner_routing.py, line 1067:

<comment>This exception validates only the hosted branch for a fork's own repository, not a fork PR targeting `manaflow-ai`. Require the existing fork-PR gate before allowing `vars.LINUX_RUNNER`, otherwise this guard can approve a selector that queues fork PRs on self-hosted Linux.</comment>

<file context>
@@ -1050,11 +1050,21 @@ def test_every_fork_exercised_runner_has_a_hosted_path(self) -> None:
                         saw_linux += 1
                         self.assertTrue(
-                            hosted_linux,
+                            hosted_linux or (fork_short_circuits and hosted_macos),
                             f"{path.name}:{number} has no GitHub-hosted Linux fork branch",
                         )
</file context>

Suggested change

	hosted_linux or (fork_short_circuits and hosted_macos),
	hosted_linux or (
	fork_short_circuits
	and hosted_macos
	and fork_pull_request_gate_error(line) is None
	),

[teamleaderleo]

Review (codex, router-review): SHIP at cc11800

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

Follow-ups: 4. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at cc11800

Visual verdict: no images; Leo tries the build.

Focused tests passed except the same pre-existing webviews test failure reported for the base.

1. [P1][follow-up] .github/workflows/cmux-next.yml:3057 only rejects a mismatching non-empty source_digest. A missing or empty digest still permits state=done, exit_code=0, and an exact ref to produce verdict=pass, despite the stated requirement that the digest must be verified.

2. [P2][follow-up] scripts/ci/next_macos_cascade.py:889-894 checks only SIDE_LABEL, while side_route() can widen a light label to a compatible std label. If the light label is full/dead but the std label is merely busy, placement can incorrectly send fast-tier to the fleet even though its actual owned route is not full/dead.

3. [P2][follow-up] .github/workflows/cmux-next.yml:2957 does not gate the push branch on needs.path_route.outputs.fast_tier != 'false', contrary to the PR description. A push whose route skips the fast tier can still launch the fleet client when placement lists it.

Follow-ups:

• [P1][follow-up] Require a present, non-empty source_digest equal to REF.
• [P2][follow-up] Evaluate all compatible owned labels when deciding whether every route is blocked.
• [P2][follow-up] Apply the fast-tier route gate to pushes.
• [P2][follow-up] No after images were supplied.

[cmux-bot]

cmux-bot approves cc11800: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (codex, router-review): SHIP at 40a4d4e

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

SHIP at 40a4d4e

Visual verdict: no images; Leo tries the build

No findings. Focused workflow, placement, cascade, and routing checks passed. Broader failures traced to unchanged code or omitted fixtures.

Follow-ups: none

[cmux-bot]

cmux-bot approves 40a4d4e: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

A newer verdict blocks 40a4d4e (no independent SHIP at the head).

[teamleaderleo]

Review (carry): SHIP at 2944b6f (base merge only since 40a4d4e)

A newer verdict blocks 2944b6f (no independent SHIP at the head).