ci: anchor push routing to finished fast tier by teamleaderleo · Pull Request #28260 · manaflow-ai/cmux

teamleaderleo · 2026-10-30T10:15:42Z

A feat-cmux-next push now routes its Mac fast tier over the diff since the newest finished fast-tier marker on its history. A tier that reaches success or failure uploads the marker; cancelled, skipped, and reused jobs do not. Missing markers, lookup errors, and timeouts run the fast tier.

Swift test remains unchanged: workflow_dispatch and the tip Swift suite still run, while feat-cmux-next pushes still skip it. Cargo fast-tier routing is unchanged. No workflow permissions or secrets change; the lookup uses path_route's existing actions: read permission.

Request: "on feat-cmux-next pushes, cmux-next.yml's path routing must decide from the commits since the last FINISHED 'cmux-next fast tier' (success or failure; not cancelled or skipped), not since the previous push."

Design context: https://github.com/manaflow-ai/cmuxterm-hq/issues/5123#issuecomment-6515215902

Validation: all 245 targeted tests passed remotely at 42bd82a (11 anchor, 69 routing, 165 workflow tests). Independent Astra review returned SHIP at that exact head, checking failure marker conditions and a shallow checkout. Marker retention is one day; the new upload passes the retention guard. Routing uses the existing fast_tier output, preserving daemon-only skips from #28334.

Risk: routing may conservatively run extra Mac work when marker lookup is unavailable or history has no recorded anchor. This does not repair runner starvation or superseded-push preflight skipping.

Agent session: agent session

blacksmith-sh · 2026-10-30T10:20:46Z

Found 2 test failures on Blacksmith runners:

Failures

Test	View Logs
`test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt` `epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/` `test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt` `epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)`	View Logs
`test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl` `ights_validate)/` `test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl` `ights_validate)`	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

github-actions · 2026-10-30T10:24:01Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review d78a17d: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28260 -f ref=lane/fast-tier-anchor-last-finished -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4116264351601 (f8c1ccbeda)

teamleaderleo · 2026-10-30T10:46:07Z

Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at eae49f0

BLOCK

.github/workflows/cmux-next.yml:2902-2903 — fast-tier-cargo now depends on path_route, but its condition lacks always(). If path routing fails, GitHub skips this job automatically because a required job failed, so the Linux cargo/script fast tier is silently lost. This contradicts the workflow’s fail-open behavior and its comment that a failed route still runs the cargo half.

teamleaderleo · 2026-10-30T12:36:11Z

SHIP: independent codex-review-box/gpt-6-astra review at exact head 6414a0d found no blocking findings after reviewing all five changed files, 173 tests, and a shallow-clone check. The earlier BLOCK was for the discarded eae49f0 head; current cargo remains independent of path_route, and swift-test remains unchanged.

teamleaderleo · 2026-10-30T12:41:32Z

Review (codex-review-box/gpt-6-astra high, lane-place): SHIP at 6414a0d

SHIP

No blocking findings. Reviewed all five changed files; 173 tests passed. A local shallow-clone check also passed. Live GitHub behavior could not be verified offline.

teamleaderleo · 2026-10-30T13:38:39Z

Review (codex-review-box/gpt-6-astra high, lane-place): SHIP at 2dc987d

SHIP

No blocking findings. All five changed files match the supplied diff; 231 focused tests passed. Live GitHub Actions behavior could not be verified offline.

cmux-bot

cmux-bot approves 2dc987d: an independent review passed this exact head (review box review-28260-refreshed receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

teamleaderleo · 2026-10-30T13:50:19Z

Overlap with #28334 (route: skip the Mac fast tier when no fast-tier input changed), which is in review now:

Both PRs add a fast_tier: output to the path_route job. They sit about 15 lines apart, so git merges them cleanly, and the result is a duplicate fast_tier key in path_route.outputs. GitHub rejects that workflow file, so whichever PR lands second breaks every cmux-next run on base.
Both also edit the same fast-tier job if: line. That one is a real textual conflict.

After #28334, scripts/ci/cmux_next_route.py writes fast_tier=true|false for every event, from Route.fast_tier(). Your anchor step appends its own fast_tier= on pushes after route.py, and the later line wins. So the two can coexist like this:

The output line. Keep the one from ci(cmux-next): a doc-only or daemon-only PR skips the Mac fast tier and swift test #28334 and drop the fast_tier: line this PR adds.
The anchor. It can decide reach with outputs(result)["fast_tier"] instead of ["macos"]. That way a daemon-only push range does not run the Mac fast tier.
The if: line. Merge it to ... pull_request ... && needs.path_route.outputs.macos != 'false' && needs.path_route.outputs.fast_tier != 'false') || (push ... && needs.path_route.outputs.fast_tier != 'false').

If #28334 lands first, this PR needs that rebase. If this one lands first, I'll make the same change on #28334.

A newer verdict blocks bdecf5b (no independent SHIP at the head).

teamleaderleo · 2026-10-30T14:20:01Z

Review (codex-review-box/gpt-6-astra high, lane-place): SHIP at bdecf5b

SHIP

No blocking findings. Reviewed all five changed files; 231 tests passed, plus a local shallow-history fetch/diff check.

Forum acknowledgments could not run: neither the forum tool nor acpmux is available.

cursor · 2026-10-30T14:30:29Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

teamleaderleo · 2026-10-30T14:37:30Z

Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at 10242cb

BLOCK

.github/workflows/cmux-next.yml:2893 sets the new marker artifact to retention-days: 14. The repository’s test_ci_upload_artifact_retention.py guard allows at most 3 days for unallowlisted artifacts, and reports this exact step as an unlisted violation. Set retention to 3 days or add the required allowlist entry.

teamleaderleo · 2026-10-30T14:51:11Z

Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at 3c00d35

BLOCK

.github/workflows/cmux-next.yml:2879-2886 — These marker steps do not use always(). When Run the fast tier fails, GitHub skips both steps before evaluating their outcome == 'failure' conditions, so failed verdicts never publish the anchor marker. Subsequent pushes repeatedly anchor to an older commit and rerun accumulated work. Add an explicit always() && !cancelled() condition.
scripts/ci/fast_tier_anchor.py:157-160 — The decision uses outputs(result)["macos"], but the workflow gates this job on fast_tier. Daemon-only/app-host-only changes produce macos=true and fast_tier=false, so the anchor incorrectly runs the Swift fast tier for changes it should skip. Use the fast_tier output.

A push fast tier on the base branch can end with no verdict: its concurrency group replaces it while pending, or the head preflight skips it. No push run containing c5dd001 (#27931, generated SettingsSchema Swift) finished the fast tier (runs 41012836913 and 41013125180 cancelled, 41014687570 skipped), so that Swift landed with only a Release compile behind it. A push fast tier that runs to a verdict now uploads a cmux-next-fast-tier-finished marker. path_route reads the markers (existing actions: read token) and routes the diff since the newest one on the push history through the pull-request router; the push fast tier skips only when that diff reaches no Mac tier. Any error or missing marker runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-db2d0bd6a321df7a1f3a

teamleaderleo · 2026-10-30T14:57:58Z

Review (codex-review-box/gpt-6-astra high, lane-place): SHIP at 42bd82a

SHIP

No blocking findings at head 42bd82a2fea583e2f9e0666f8d894975031f8da6.

Checked marker conditions at .github/workflows/cmux-next.yml:2879 and :2886: always() preserves failure markers while the remaining guards exclude cancelled, skipped, and reused tiers.

All 245 focused tests passed, plus a local shallow-checkout check.

cmux-bot

cmux-bot approves 42bd82a: an independent review passed this exact head (review box review-28260-final2 receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

teamleaderleo · 2026-10-30T15:28:35Z

The final head 42bd82a retains Swift dispatch/tip-suite behavior and the existing cargo routing. Its independent review passed, and all 245 focused tests passed: https://github.com/manaflow-ai/cmux/actions/runs/41076110329.

The normal pinned landing request completed without merging. The controller reports base red since fixed for release-notary and tooling-tests, although the same jobs remain in its current base-red report and the relevant failing files are unchanged on the current base. The detailed evidence and follow-up are tracked in https://github.com/manaflow-ai/cmuxterm-hq/issues/5379. No status override or permission change was made.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-db2d0bd6a321df7a1f3a

teamleaderleo · 2026-10-30T16:37:32Z

Review (codex gpt-6-astra high, next-review): SHIP at d78a17d

No blocking findings.

cmux-bot

cmux-bot approves d78a17d: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

#28260 routes a feat-cmux-next push's fast tier from the last push whose fast tier finished, and skips the tier when that diff reaches no Mac tier. fast-tier-fleet checked the route only on pull requests, so with every macOS label full such a push still ran every Swift suite on the fleet. Its push branch now requires needs.path_route.outputs.fast_tier != false, as the tier's does, and a test checks the two gates agree on both events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lawrencecchen · 2026-10-31T03:22:32Z

Base red from this PR: the feat-cmux-next "Tier routing" step is red since a827fab. tests/test_cmux_next_base_tier.py (RouteStep) runs path_route's route step under set -u with no GITHUB_SHA, and the new fast_tier_anchor.py --head "$GITHUB_SHA" line fails with "GITHUB_SHA: unbound variable" (3 errors). #28835 reverts this PR (revert first). To re-land, give the route-step test a GITHUB_SHA and copy the anchor script into its stub root.

…)" (#28835) This reverts commit a827fab. #28260 made path_route's route step run `fast_tier_anchor.py --head "$GITHUB_SHA"` on a feat-cmux-next push. tests/test_cmux_next_base_tier.py runs that step under `set -u` with no GITHUB_SHA, so three RouteStep tests fail ("GITHUB_SHA: unbound variable") and "Tier routing" is red on every feat-cmux-next push since a827fab. Revert first (standing order); re-land with the route-step test given a GITHUB_SHA and the anchor script in its stub root. tests/test_cmux_next_base_tier.py: FAILED (errors=3) at the base, OK here. The rest of Tier routing and test_cmux_next_checks_workflow.py: OK.

…S label is full or dead (#28391) * cmux-next: the fast tier runs as a build-fleet client when every macOS label is full or dead When macos-placement finds every owned and Blacksmith macOS label the fast tier could take full or dead, it lists the tier in fleet_jobs (FLEET_CLIENT_JOBS is now `fast-tier`). A new client-only job, fast-tier-fleet, then runs on gateway_runner with permissions {} and no checkout, action or secret: it hands fast-tier.json's Swift entries to the build fleet (`cmux-ci run` of package-test-lane.sh's suite mode, one step per package) at the exact commit, and checks that the controller's record of each step names that commit. No verdict, or a step at another commit, fails it and is never a pass. fast-tier keeps its name: when the client job ran it takes a 2 vCPU Linux runner and only reports the client's verdict (fail and no verdict exit 1); every other step is off. No whole Mac job runs on the gateway any more: the fleet_jobs -> gateway_runner runs-on branches of fast-tier, swift-canary, swift-test, daemon-test, generated-files, release-compile and cmux-scheme-compile are gone, and the FleetGateway test that asserted them is now its inverse. GatewayRunsOnlyTheFleetClient guards the shape: a job that can reach the gateway through fleet_jobs runs only the client and shell builtins (checked by parsing its run text and by running the step with PATH empty against a fake client), and the tier itself never runs there and is off in report mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cmux-next: list a fleet job only while the gateway has an idle runner for its client fast-tier-fleet runs on gateway_runner beside GATEWAY_JOBS, but the gateway was judged for GATEWAY_JOBS alone: with one idle runner for both, the client would queue behind another gateway job until its timeout and report no verdict. side_lane_placement.fleet_gateway() keeps fleet_jobs only while the gateway has an idle runner for each listed job beyond GATEWAY_JOBS and the run's own jobs. Otherwise the job keeps its route. A re-run emits no gateway, so it lists none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cmux-next: the fast tier's client step leaves with exit when the client is missing tests/test_ci_cmux_ci_client_guarded.py requires the missing-client branch to end the step with a literal exit; finish already exits, so this adds the explicit one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cmux-next: the fast tier's fleet verdict reads the controller's exit code A pass now needs the step done at the commit with exit 0; a step that ended with a nonzero exit is a fail even when the client exited 0. Step ids are read as the fleet tools read them (6 to 64 hex), not only 24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: classify cmux-next.yml#fast-tier-fleet in runners.json The routing table reads the gateway label as a macOS runner; the client hands Swift suites to the build fleet, so its class is apple-swift-tests, beside the other gateway clients (app-compile-fleet, swift-canary-fleet). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cmux-next: the fast tier's fleet client tests the merge commit fast-tier builds A pull request's fast-tier checks out its merge commit (github.sha), so the client tests that commit too, as swift-canary-fleet does: a head that breaks against the current base is still caught, never passed on the unmerged head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cmux-next: fast-tier always starts, to report its fleet client or take its queue A tier listed in fleet_jobs has no other placement route by design, so the test pins that fast-tier still starts when its client ran (success, failure or cancelled) to report the verdict, and still starts without one (the base's deferred owned queue): it is never skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cmux-next: the fast tier's fleet client skips a push the route skips #28260 routes a feat-cmux-next push's fast tier from the last push whose fast tier finished, and skips the tier when that diff reaches no Mac tier. fast-tier-fleet checked the route only on pull requests, so with every macOS label full such a push still ran every Swift suite on the fleet. Its push branch now requires needs.path_route.outputs.fast_tier != false, as the tier's does, and a test checks the two gates agree on both events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cmux-next fast-tier-fleet: pass the 60 s entry limit as --env=, which the gateway guard allows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

teamleaderleo requested a review from lawrencecchen as a code owner October 30, 2026 10:15

github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026

This was referenced Oct 30, 2026

acpmux: a peer link hears no change of a row mirrored from another peer #28152

Merged

acpmux: briefs asks the peers at once and names each one left out #28172

Merged

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from eae49f0 to 6414a0d Compare October 30, 2026 11:50

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 11:51 — with GitHub Actions Inactive

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from 6414a0d to 2dc987d Compare October 30, 2026 13:27

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 13:27 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 13:28 — with GitHub Actions Inactive

cmux-bot Bot previously approved these changes Oct 30, 2026

View reviewed changes

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from 2dc987d to bdecf5b Compare October 30, 2026 14:05

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 14:05 — with GitHub Actions Inactive

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from bdecf5b to a79bf30 Compare October 30, 2026 14:24

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 14:24 — with GitHub Actions Inactive

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from a79bf30 to 10242cb Compare October 30, 2026 14:30

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 14:30 — with GitHub Actions Inactive

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from 10242cb to 3c00d35 Compare October 30, 2026 14:38

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 14:38 — with GitHub Actions Inactive

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from 3c00d35 to 4820231 Compare October 30, 2026 14:53

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 14:53 — with GitHub Actions Inactive

teamleaderleo force-pushed the lane/fast-tier-anchor-last-finished branch from 4820231 to 42bd82a Compare October 30, 2026 14:56

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 14:56 — with GitHub Actions Inactive

cmux-bot Bot approved these changes Oct 30, 2026

View reviewed changes

ci: keep anchor regressions on explicit fleet runs

d78a17d

Agent-Session: https://cmux.com/deeplink/agent?id=aid-db2d0bd6a321df7a1f3a

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 15:30 — with GitHub Actions Inactive

cmux-bot Bot approved these changes Oct 30, 2026

View reviewed changes

teamleaderleo merged commit a827fab into feat-cmux-next Oct 31, 2026
134 of 140 checks passed

teamleaderleo deleted the lane/fast-tier-anchor-last-finished branch October 31, 2026 02:22

teamleaderleo mentioned this pull request Oct 31, 2026

cmux-next: the fast tier runs as a build-fleet client when every macOS label is full or dead #28391

Merged

lawrencecchen mentioned this pull request Oct 31, 2026

base red: revert #28260 (fast tier anchor breaks Tier routing: GITHUB_SHA unbound in the base-tier test) #28835

Merged

Conversation

teamleaderleo commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blacksmith-sh Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

cmux-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

cursor Bot commented Oct 30, 2026

Bugbot needs on-demand usage enabled

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

cmux-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

cmux-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lawrencecchen commented Oct 31, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

teamleaderleo commented Oct 30, 2026 •

edited

Loading

blacksmith-sh Bot commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading