ci(cmux-next): the session label comes only from a placement that judged its queue by teamleaderleo · Pull Request #27853 · manaflow-ai/cmux

teamleaderleo · 2026-10-30T02:26:28Z

Leo, 2026-10-28: "Bro, I thought we already had logic to automatically route stuff. How can we still have pins like this? I hate pins. Like, just pin the set of the stuff that isn't hilariously slow." Rulebook: no job waits on one reserved label; "a CI test fails on a runs-on that can resolve to a single-host label with no fallback".

What was wrong

On 2026-10-29, #24295's daemon tests waited 90 minutes on glaeda-session-std-xcode-26.6, and then did it again. Five session runners were online and busy, and the gui label next to it was starting jobs within a minute. daemon-test, generated-files and both autofix action-exports jobs could still put that label into runs-on as a literal (vars.CI_SESSION_LABEL == '1' && 'glaeda-session-std-xcode-26.6' || ...), so no picker ever looked at its queue. GitHub never moves a queued job. ci-owned-pool-rescue.yml only watches cmux-next runs "refusals-only" when Blacksmith has no free slot, and that day it had none, so nothing moved those jobs.

What changed

Routes (cmux-next.yml daemon-test and generated-files runs-on plus CMUX_NEXT_RUNNER, cmux-next-pr-autofix.yml and cmux-next-base-autofix.yml action-exports): the session label is never written literally. When the placement gave no route, the last owned leg is glaeda-gui-std-xcode-26.6, and Blacksmith macOS comes after that, as before. The session label still reaches a job, but only through placement outputs (session_runner, free_session_runner, session_lane_runner).
Picker (side_lane_placement.py role_room(), next_macos_cascade.py Seen.unserved): the janitor's snapshot already writes unserved, the labels with a queued job and none running or started within 10 minutes (starved_labels.py). role_room() now refuses room on such a label unless one of its runners is idle right now. An empty session_runner then sends the jobs to gui_runner, and an empty gui route sends them to gui_overflow (Blacksmith). Since feat-cmux-next added two keeps to healthy_role_label() that skip role_room() (a stalled or dead Blacksmith overflow keeps any online role label, and a known not-full health reading keeps it), the same unserved-and-no-idle-runner check now runs first there too: an unserved queue is not a busy queue that moves. Seen.unserved is read only from a fresh snapshot.
The CI test that should have caught this (tests/test_ci_runs_on_fallback.py): its gui-or-session rule passed these jobs. They had a Blacksmith last leg, were gated on attempt, and sat in a workflow the rescue watches, so the test counted the rescue as their way off. The rescue doesn't move them when Blacksmith is full. The new test_a_session_label_comes_only_from_a_placement_that_judged_it fails when a runs-on names a session label literally, or reads a session variable without the placement's session_lane_runner verdict. Red on base with 8 findings (all four jobs, literal + vars.CI_SESSION_LABEL); green here. A self-test covers the expression forms.
Tests updated to match: test_ci_session_label_routing.py, test_ci_side_lane_placement.py (autofix route), test_cmux_next_checks_workflow.py (generated-files expected route), test_ci_self_hosted_guard.sh (drops the session-selector exemptions that no longer match anything), test_ci_next_macos_cascade.py (unserved parsing: single owned keys only).

Other single-label literals checked

glaeda-gui-std-xcode-26.6 (cmux-next.yml, autofix): it comes after placement and before a Blacksmith leg, and the rescue watches these workflows. role_label_problems already covers it.
glaeda-dev-std-xcode-26.6 (FRESH_LABEL env): placement input only, never runs-on.
glaeda-side-light-xcode-26.6 (cmux-tui-nightly.yml macos_runner): a pool label, and its re-run takes macos_retry_runner (Blacksmith).
glaeda-root-std-xcode-26.6 (ci-compile-attribution.yml OWNED): a probe input with its own OVERFLOW.

None of these is a session label left without a judge, so they stay as they are.

Reds

tests/test_ci_fork_runner_routing.py (SESSION_LANE: ${{ vars.CI_SESSION_LANE_RUNNER }} at cmux-next.yml placement) fails the same way on base. fix(ci): a fork pull request never reads CI_SESSION_LANE_RUNNER #27548 fixes it.
check_no_self_hosted_fleet_runners in test_ci_self_hosted_guard.sh flags the same five runs-on lines on base.

No new timer, loop or service. Nightly, dev and promote workflows are untouched, and no concurrency or cancel behaviour changed.

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Fixes cmux-next jobs waiting hours on session labels whose runners are online but busy, by ensuring the session label only reaches a job through a placement that judged its queue.

daemon-test, generated-files, and both autofix action-exports jobs no longer name the session label literally; their last owned leg is the gui label.
role_room() in side_lane_placement.py declines labels the janitor finds unserved (a job queued, none running or started within 10 minutes) unless a runner is idle, so session_runner falls back to gui_runner; healthy_role_label() applies the same refusal before the stalled-overflow and health-contract keeps bypass it.
The new test_ci_runs_on_fallback.py test fails whenever a runs-on names a session label literally or reads a session variable without the placement's session_lane_runner verdict; it fails on base with 8 findings and is green here.

Migration

No new timers, loops, or services; nightly, dev, and promote workflows are untouched.

^{Written for commit 50820ae. Summary will update on new commits.}

…ged its queue cmux#24295's daemon tests waited 90 minutes, twice, on glaeda-session-std-xcode-26.6 with five session runners online and busy while the gui label beside it started jobs within a minute. The runs-on named the session label literally (behind vars.CI_SESSION_LABEL), so no picker judged it, and the rescue watches cmux-next refusals-only while Blacksmith has no free slot. - daemon-test, generated-files and both autofix action-exports never name the session label literally; their last owned leg is the gui label. - role_room() declines a label the janitor's snapshot lists as unserved (none running or started within 10 min) unless one of its runners is idle, so session_runner falls to gui_runner. - tests/test_ci_runs_on_fallback.py: a session label reaches a job only from a placement output (or vars.CI_SESSION_LANE_RUNNER gated on the placement's session_lane_runner). The gui-or-session rule passed these jobs because it counted the rescue as their way off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

teamleaderleo · 2026-10-30T02:27:12Z

Owned by an agent session · session

github-actions · 2026-10-30T02:27:20Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 50820ae: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27853 -f ref=fix/std-xcode-session-label-picker -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: red check: Script tests (pin-cmux-tui, embed-cef, ensure-cef store) (+1 more) is not red on the base b2ff685; do not merge until green; see lane-rules

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4134851291701 (530f1f2f82)
cmux NEXT nightly 1.0.0-nightly.4135063471001 (c026ef2050)
cmux NEXT nightly 1.0.0-nightly.4135181397101 (0a4c836fe3)

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/41349234750 at 530f1f2 failed: cmux-next crash ends the app.
Those jobs last passed at ed47344. This pull request is one of 4 merged since: #31235, #31213, #31278, #27853.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

…able is not a label Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

blacksmith-sh · 2026-10-30T03:36:36Z

Found 6 test failures on Blacksmith runners:

Failures

Test	View Logs
`/tmp/tmp5mq56i3i/summary-missing.json (live):/summary-missing.json (live):`	View Logs
`/tmp/tmp5ybawpr1/summary-fail.json (live):/summary-fail.json (live):`	View Logs
`[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/` `[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows`	View Logs
`[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames`	View Logs
`[tabs] debug.tab_switch timed 0 of 40 switches/` `[tabs] debug.tab_switch timed 0 of 40 switches`	View Logs
`[tabs] the window draws no frames/[tabs] the window draws no frames`	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

…oken to #28163 On top of dep-fleet-2's placement re-run (4cbd22a): where the placement or full re-run would repeat a finished job but no job of the run is running or held by a runner, cancel (only waiting jobs stop) and re-run only what waited: the one waiting job and its dependents, or the failed jobs when several waited. Run 40997814347 (PR #27853): 28 of 29 jobs finished, the swift test and daemon tests red below the placement, its Release compile queued 4.5 h on glaeda-std-xcode-26.6 while that label ran nothing. watch()'s periodic offer and the refusals-only check consider both failovers. The snapshot-token refresh and read cache are dropped here: they land separately in #28163. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

4cbd22a This carries dep-fleet-2's 4cbd22a (branch lane/fasttier-open-head-replace: the red-sibling watch() failover offer, label_started/label_quiet and the placement re-run), rebased as the commit before this one, and adds the settled-sibling failover on top of it. Where the placement or full re-run would repeat a finished job but no job of the run is running or held by a runner, cancel (only waiting jobs stop) and re-run only what waited: the one waiting job and its dependents, or the failed jobs when several waited. Run 40997814347 (PR #27853): 28 of 29 jobs finished, the swift test and daemon tests red below the placement, its Release compile queued 4.5 h on glaeda-std-xcode-26.6 while that label ran nothing. Review fixes: a refused single-job re-run is reported, never widened to a re-run of the failed jobs; the cancel guard reads the jobs after the run, so a job that started during a slow run read drops the cancel; the receipt names the failed jobs a multi-job re-run repeats; and the CANCEL_SENT mark carries the job id, so a successor of a rescue stopped after its cancel re-runs that job, not the failed ones. watch()'s periodic offer and the refusals-only check consider both failovers. The snapshot-token refresh and read cache stay as #28163 landed them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

4cbd22a This carries dep-fleet-2's 4cbd22a (branch lane/fasttier-open-head-replace: the red-sibling watch() failover offer, label_started/label_quiet and the placement re-run), rebased as the commit before this one, and adds the settled-sibling failover on top of it. Where the placement or full re-run would repeat a finished job but no job of the run is running or held by a runner, cancel (only waiting jobs stop) and re-run only what waited: the one waiting job and its dependents, or the failed jobs when several waited. Run 40997814347 (PR #27853): 28 of 29 jobs finished, the swift test and daemon tests red below the placement, its Release compile queued 4.5 h on glaeda-std-xcode-26.6 while that label ran nothing. Review fixes: a refused single-job re-run is reported, never widened to a re-run of the failed jobs; the cancel guard reads the jobs after the run, so a job that started during a slow run read drops the cancel; the receipt names the failed jobs a multi-job re-run repeats; and the CANCEL_SENT mark carries the job id, so a successor of a rescue stopped after its cancel re-runs that job, not the failed ones; and a failover's receipt (both kinds) is given only once its cancel went out. watch()'s periodic offer and the refusals-only check consider both failovers. The snapshot-token refresh and read cache stay as #28163 landed them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

…ed siblings, judged by real label starts (#28148) * ci(owned-pool rescue): the open-head failover moves cmux-next PR runs stuck on a label that starts nothing The stall-aware failover (#27521) never moved a cmux-next pull request run. Four reasons, all fixed here: - It required that no job of the run had finished other than the placement jobs, but a cmux-next PR run's Linux jobs (checks, the cargo fast tier, web bundles, fleet Release compile) always finish before a Mac job has waited three minutes. A cmux-next PR run's first attempt is now re-run from its macOS placement (POST /actions/jobs/{placement}/rerun): the placement and the jobs that need it run again and read the routes afresh, and every other finished job keeps its result, a red `checks` included. A queued or running job the re-run would not repeat still keeps the open-head rule. Other workflows keep the full re-run rule. - A run left queued behind a red sibling (assess()'s leave_decided) was never offered the failover at all. watch() now offers it every 2 minutes to any PR run with a Mac job queued past 3 minutes. - The snapshot reads used the sweeper's first token, which expires after an hour: from then every read was refused (401) and the failover was off (sweeper run 41006853342: 22 refusals from 06:12Z). The reads now use the client's current, re-minted token, and one read serves all watches for 60 s. - "Started nothing" meant running == 0 in the janitor's pool count, which rolls root jobs into the pool and counts a job started hours ago. The janitor snapshot now records each macOS label's newest job start (label_started), and a label that started nothing since the job began waiting counts as dead too. Runs 40993742146 (#27797) and 41021261503 (#27756) waited hours on glaeda-side-std and glaeda-std, whose online runners took no job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887 * ci(owned-pool rescue): settled-sibling failover on top of dep-fleet-2's 4cbd22a This carries dep-fleet-2's 4cbd22a (branch lane/fasttier-open-head-replace: the red-sibling watch() failover offer, label_started/label_quiet and the placement re-run), rebased as the commit before this one, and adds the settled-sibling failover on top of it. Where the placement or full re-run would repeat a finished job but no job of the run is running or held by a runner, cancel (only waiting jobs stop) and re-run only what waited: the one waiting job and its dependents, or the failed jobs when several waited. Run 40997814347 (PR #27853): 28 of 29 jobs finished, the swift test and daemon tests red below the placement, its Release compile queued 4.5 h on glaeda-std-xcode-26.6 while that label ran nothing. Review fixes: a refused single-job re-run is reported, never widened to a re-run of the failed jobs; the cancel guard reads the jobs after the run, so a job that started during a slow run read drops the cancel; the receipt names the failed jobs a multi-job re-run repeats; and the CANCEL_SENT mark carries the job id, so a successor of a rescue stopped after its cancel re-runs that job, not the failed ones; and a failover's receipt (both kinds) is given only once its cancel went out. watch()'s periodic offer and the refusals-only check consider both failovers. The snapshot-token refresh and read cache stay as #28163 landed them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887 * ci(owned-pool rescue): a failover's cancel never strands a queued job Review fixes for the open-head failovers. The settled-sibling failover now moves a run only when the stalled job is its one unfinished job, and re-runs that job and its dependents. With a second job queued (a Linux job, another Mac job), a re-run of the failed jobs was not shown to bring back every job the cancel stopped. The open head keeps the run until only the stalled job is left, and the guard right before the cancel drops it if another job queued meanwhile. A placement failover's CANCEL_SENT mark now names the placement job (" of job N"), not " of every job". A successor of a rescue stopped between its cancel and its re-run therefore re-runs the placement and the jobs that need it, not every job, which would repeat the finished jobs the failover keeps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887 * ci(owned-pool rescue): the settled guard drops the cancel once the stalled job finished or left settled_guard() now also requires only_job to be listed and unfinished at the read right before the cancel. A stalled job that finished (or left the listing) since the failover's look, while the run still read in progress, would otherwise be cancelled with nothing waiting and re-run as a finished job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887 * ci(owned-pool rescue): a placement failover never widens a refusal or strands a queued job A refused re-run of the cmux-next placement is now reported and stops, instead of falling through to a re-run of every job, which would repeat the finished Linux jobs (a red check included) the failover keeps. An unfinished job counts as brought back by the placement re-run only when every workflow job its name could be needs the placement (PlacementRerun.brings_back()). A queued job with an unmatched or ambiguous name could be left cancelled, so it now keeps the run where it is. Finished jobs keep the conservative reruns() rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887 * ci(owned-pool rescue): fresh job reads before a placement cancel; the failover offer before the refusal wait open_head_guard() reads the run, then the jobs, as settled_guard() does. A job that queued while a slow run read waited now shows in the read the cancel decides from, so the placement re-run never leaves it cancelled. watch() offers the open-head failover before every branch that waits or returns: the refusal wait (when the run also failed on its own, so the refusal path would decline it) and the rescue/refused returns (a hold re-watches and returned before the offer). A run with a genuine failure, a refused Mac sibling and a compile stalled on a dead label now moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

teamleaderleo · 2026-10-31T11:10:10Z

Orphaned: no activity for 32 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in scripts/ci/next_macos_cascade.py, scripts/ci/side_lane_placement.py, tests/test_ci_side_lane_placement.py. Listed for a lane to resolve.

Conflicts kept everything feat-cmux-next has and re-added only this PR's change: the literal session label swapped for the gui label in daemon-test and generated-files, Seen.unserved parsed from a fresh janitor snapshot, and healthy_role_label() declining an unserved label with no idle runner before base's stalled-overflow and health-contract keeps (which bypass role_room()). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

teamleaderleo · 2026-11-01T01:44:54Z

Review (codex gpt-6-astra high, fast): SHIP at 179aea9

No blocking or follow-up findings. The conflict resolution keeps owned_full/owned_full_known, the stalled/dead overflow keep and the stale late_waits paths; fresh-only unserved reaches both role_room() and the health-keep paths; every routing expression keeps a non-empty fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

teamleaderleo · 2026-11-01T20:39:32Z

Review (codex gpt-6-astra high, fast): SHIP at 50820ae

No blocking or follow-up findings. Re-review after merging feat-cmux-next again: the PR's change matches the SHIP at 179aea9; the only dropped hunk is the edit to a test constant feat-cmux-next removed. The merged workflow cascade is kept, every affected route keeps its GUI/Blacksmith fallback, and session labels stay placement-gated.

* ci guards: depth-limited fetches only in shallow checkouts (base red) scripts/measure/fresh-home-launch.sh and workers/cmux-vm/scripts/test-mesh-label-compat.ts passed --depth to a fetch in the caller's checkout, which turns a full clone shallow. Both now pass it only when git rev-parse --is-shallow-repository says the checkout is already shallow. Regressed by #30495 (fresh-home-launch.sh) and #30501 (test-mesh-label-compat.ts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: map the transcript's TurnRows.tsx in place of the deleted MessageGroupRow.tsx (base red) #29800 removed MessageGroupRow.tsx (every sent message is its own card); TurnRows.tsx now draws each sent message row in order (SentMessageLine.tsx is already mapped), so chat-transcript maps it. Regressed by #29800 (deleted the file #28698 had mapped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: scan cmux-next-ios/verify/ for errno-after-assertion (base red) cmux-next-ios/verify/imsg-replies/GestureServerUITests.swift is an XCTest source outside every TEST_ROOTS entry. Scan cmux-next-ios/verify/ (it has no errno use, so nothing else changes). Regressed by 201101a (added the verify/imsg-replies rig; landed by a safe-push train, no PR). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: list the four manual-lane workflow readers in UNWIRED (base red) tests/test_ci_blacksmith_linux_sizes.py, test_ci_overflow_pick_reads_pool.py, test_nightly_next_control_rescue.py and test_ci_cloud_probe_rescue.py read .github/workflows/ but are lane = "manual" in tests/test-execution.toml (no new CI tests; wiring them as linux-guard reddened tooling-tests on every PR). They join UNWIRED with the same reason as the other manual-lane entries, and each says when to run it by hand. Regressed by #30360, #30878, #30657 and #30796 (each added its test without an UNWIRED entry). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: fleet rule lints pass again (base red) "ci: Validate fleet rule lints" failed on feat-cmux-next in three rules. big-red: the personal machine's name left product code and tests. Doc comments say "one lane VM" / "<host>"; the webview tests use my-vm (its machine hue still differs from gpu-box's). Regressed by #28105 (peer_handshake_tests.rs, list_reply.rs, list_reply_tests.rs), #29495 (host_update_tests.rs), #29126 (chatOpenPacedList.test.ts), #31108 (project-chooser-machine-tag). pkill: the iOS verify rigs and the tool-evals remote daemon stop only what they started. imsg-replies records the conversation-sim, the recorders (simctl itself, not the xcrun shim) and the gesture servers in pidfiles and signals those PIDs after checking the command line still matches; pq_browser_events records its two site servers' PIDs; the tool-evals remote stop kills the setsid group whose PID the launcher recorded beside the socket. Regressed by 201101a (imsg-replies, first in #22328), #25915 (pq_browser_events.py), #29080 (tool-evals/session.mjs). Allowlisted as safe, with reasons: scripts/ci/macos-stubs/Security.tbd (a linker stub listing symbol names, no keychain read; #27603) and tests/tool-evals/cases/hard.ndjson (an eval case forbidding pkill; #28738). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: determinize python tests flagged by test-determinism (base red) Regressed by the commits introducing each assertion (see report). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: nightly Xcode selection, unused anchors, daemon retention key (base red) - Validate macOS jobs select a pinned Xcode: the nine flagged nightly.yml jobs already run select-ci-xcode.sh through the `- *select-xcode` step alias; the guard did not expand step aliases. It now resolves `- *name` to the `- &name` step, with self-tests for both an alias to the selector and an alias to another step. Regressed by #30471 (steps turned into aliases to keep nightly.yml under its size limit). - Lint every workflow: drop the five sign-nightly-identity-legacy-* anchors left after the Blacksmith twin that aliased them was removed (a7b6a61, "legacy sign and deltas run on owned Macs only"; the twin came from #27357). No behaviour change. - upload-artifact retention: the 14-day cmux-nightly-remote-daemon upload moved from build-nightly-app to build-nightly-remote-daemon in #25935 with its retention unchanged; move the allowlist key with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: media tours and visual-gate state match their validators (base red) Drop the unknown step key "record" (re-added after #26888 removed it by #27933 and f283a2e), the unimplemented top-level frameInterval in home-chief-shimmer (c022070), the focusLeft right after closePane in palette-chat-toggles (#25405), split pane-polish (31 steps) into pane-polish and pane-polish-themes, and give the app-store-consent-deploy state a synthetic host that is not an email shape (4a05792). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: allowlist remaining test-determinism findings with reasons (base red) Swift/web/tool-evals findings cannot be determinized without Mac or webview runs; each entry names its reason and adding PR. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: Linux pool for the push kicks; record pool readers as unrescued (base red) - cmux-next-base-tier-kick, cmux-next-dev-kick and ios-internal-nightly-trigger waited on one fixed Blacksmith Linux size (!startsWith(LINUX_RUNNER) || overflow 8 || 2 vCPU). They now read fromJSON(vars.RUNNER_POOLS).linux after the fork branches. Their pinning tests (base tier, DEV kick, iOS internal builder) assert the pool read instead. Regressed by #30837. - owned_pool_rescue.py: feat-cmux-next's UNRESCUED_WORKFLOW_PATHS records the secretless Linux pool readers the owned-pool sources test found unlisted (fast gates, gallery health, PR checks, cloud backend check, web validation, and the three kicks above), with the existing _POOL_READ_WHY. Regressed by #29844 and #31138. - test_ci_linux_single_size.py: drop cmux-next-fast-gates.yml from FIXED_PAIR (moved by #31138); test_ci_overflow_pick_reads_pool.py: drop the two kicks from PICKS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: signing-runner and retry-attempt guards follow the owned-Mac signing rule (base red) Signing keychain guard (rule 7): accept owned-pool runner variables only where safety is proven in the job itself: - 7a: a job in glaeda's exclusive signing class whose first step is the verbatim claim gate may fall back to CI_NIGHTLY_NEXT_BUILD_RUNNER / CI_SIDE_LANE_RUNNER (nightly.yml sign-nightly-identity-legacy and generate-nightly-deltas, owned-only by design, Lawrence a7b6a61; deltas' Sparkle key added in #29261). test_ci_signing_owned_mac.py holds the claim script, always() keychain delete and twin/picker rules for them. - 7b: stable-next.yml sign (#27898) reads every signing secret only under env.SIGN == 'developer-id', whose condition picks a literal Blacksmith label before any variable; its owned dry-run holds no key. Proven structurally. Retry guard: cmux-next.yml swift-test/daemon-test/generated-files' side-lane fallback (#29814) is now gated at run_attempt <= 3 for the bot, so the bot's attempt 4 takes blacksmith-6vcpu-macos-26 like every other cmux-next fleet fallback (#22565). The three owned-only nightly jobs are exempt while they keep exactly their owned route. The textual check now forbids naming attempts after 2 (== 3+), as its comment says, instead of also == 2 (#29847 nightly.yml, #29130 cmux-next-guards.yml name the bot's attempt 2, which may stay owned). Regressed by #29814, #27898, #29261, #29847, #29130 and a7b6a61. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: owned Mac pools through the picker/vars pattern; follow nightly step scripts for sentry-cli (base red) check_no_self_hosted_fleet_runners The guard told every workflow to "use a cloud label so required jobs never land on a mini". That is outdated policy: owned fleet Macs now come first for macOS jobs (Linux, then owned Macs through the picker, Blacksmith last), release signing may run on owned Macs, display tests gate nothing, and the NEXT nightly runs on the fleet with the nightly-only minis dedicated to it. The guard kept up only through exact per-workflow selector strings, which drifted with every routing change and turned the guard red on base: - cmux-next.yml swift-test/daemon-test/generated-files: the glaeda-side deferred fallback (regressed by #29814) and the attempt-1 gui selector (#28605, #27853) - cmux-next-base-autofix.yml / cmux-next-pr-autofix.yml: the gui selector gained a re-run predicate (#27853) - nightly.yml sign/prepare jobs: owned pool as the final fleet fallback (#29154) Fix the guard instead of moving those jobs to Blacksmith: an owned POOL label (glaeda-[<lane>-]<class>-xcode-<version>, served by many minis, never one host) is accepted as a quoted literal in a ${{ }} runs-on that routes through the declared picker/vars pattern (reads needs.<job>.outputs.* or vars.*). In a workflow a pull request can trigger, the fork pull request branch (fork heads to Blacksmith) must still come first on the line, so fork code never reaches an owned Mac (runner trust unchanged). Still refused: bare self-hosted/macOS/ARM64, single-host labels (cmux-macos, mac-mini, slot-N, tart-*, ...), macos-26 outside the fork branch, and an owned label named directly with no picker or variable route. The exact gui selector strings are replaced by this rule; self-tests cover each accept and refuse case. check_sentry_cli_install_portability #31013 moved nightly.yml's "Upload dSYMs to Sentry" body into scripts/ci/nightly-steps/app-upload-dsyms.sh (size budget), so the guard no longer saw the ./scripts/upload-sentry-dsyms.sh call. The guard now reads the step's step script too. Regressed by #31013. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: change-area tests follow the RUNNER_POOLS and devEngines routing; helper walk skips non-running namers (base red) - release-admission and remote-daemon-admission read fromJSON(vars.RUNNER_POOLS).linux by design since #30878; the two tests now assert that instead of vars.LINUX_RUNNER. - fast-checks.yml's setup-bun reads the bun pin from webviews/package.json devEngines at run time (#29757); the setup-bun test accepts a steps.<id>.outputs.version whose step reads devEngines, and still requires the literal pin everywhere else. - indexnow.yml pinned bun 1.3.14 when #30697 backported it; it now installs devEngines' 1.4.4. - ci_helper_areas() failed open on files that name a helper without running it, so owned_pool_rescue.py, owned_build_state.py and e2e_warm_derived_data.py stopped routing by where they run. Regressed by #28528 (test_cmux_next_route.py names owned_pool_rescue.py as a routing input and itself), #28513 (docs/ci/cmux-next-tiers.md names that test), #30657 (nightly_next_control_rescue.py's only test is manual-lane), #28568 (.github/warm-sources.json prose) and #30467 (base_red_excuse.py's guard-test table and its test's fixtures). The walk now ignores a test naming itself, docs, a Python test naming it only in a docstring, the listed path tables and warm-sources.json, and treats a test the registry declares in an inert lane that nothing imports as a dead end. An unregistered test nothing names still fails open. - cache_restore_receipt.py now routes precisely (macOS + Release, via ci-macos.yml's cache-restore action) instead of failing open; its assertion says so. python3 tests/test_ci_change_areas.py: 4 of 234 failed before, 0 after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: review fixes for #31366 (daemon pgid, fork-first owned pools, stale pidfiles) - tests/tool-evals/session.mjs: the remote launcher recorded $! of the backgrounded && chain (the wrapping subshell), so the stop signalled a group that did not exist. setsid now starts a shell that writes its own PID and execs the daemon, so the recorded PID is the daemon's process group. Checked locally: the recorded PID equals the daemon's PGID and SID, and kill -TERM -- -PID stops it. - tests/test_ci_self_hosted_guard.sh (security): the owned-pool exemption in pull request workflows only required the fork branch somewhere before the owned literal, so `needs.x.outputs.runner || <fork branch> || 'glaeda-...'` passed. The fork branch must now be the first selector of the expression (after only the hosted repository_owner clause). New negative self-tests for reordered expressions; red with the old match, green with the new. - cmux-next-ios/verify/parity/pq_browser_events.py: stop_site() now stops only servers this run started (serve() reports each pidfile it wrote), kills a PID only while its command line is still that server's, and removes the pidfile after. Checked with a local stub harness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: universal nightly helper runs-on match works under mawk (base red) The helper runs-on expected string passed to awk -v carried `\&\&`, which gawk reads as `&&` (with a warning) but mawk, the default awk on Ubuntu runners, keeps as `\&\&`, so the whole expression never matched and "nightly must build and hand off the macOS 26 helper" failed. Plain `&&` means the same in both. Regressed by #30824. bash tests/test_nightly_universal_build.sh: FAIL before, PASS after (mawk 1.3.4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

teamleaderleo requested a review from lawrencecchen as a code owner October 30, 2026 02:26

github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026

ci(cmux-next): test role_room's unserved check; a session switch vari…

bd3e5bd

…able is not a label Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

teamleaderleo temporarily deployed to ci-routing-agent October 30, 2026 03:30 — with GitHub Actions Inactive

teamleaderleo mentioned this pull request Oct 30, 2026

ci(owned-pool rescue): open-head failover for runs with finished or red siblings, judged by real label starts #28148

Merged

This was referenced Oct 31, 2026

ci: require a GUI-capable pool for frame pacing #28914

Merged

ci(cmux-next): a late pool snapshot still keeps PR jobs off a stalled Blacksmith 6 vCPU queue #29067

Merged

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 01:38 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 01:47 — with GitHub Actions Inactive

Merge feat-cmux-next into fix/std-xcode-session-label-picker

50820ae

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-e61ab7dfe7d69f97f887

teamleaderleo temporarily deployed to ci-routing-agent November 1, 2026 20:35 — with GitHub Actions Inactive

teamleaderleo merged commit 530f1f2 into feat-cmux-next Nov 1, 2026
195 of 202 checks passed

teamleaderleo deleted the fix/std-xcode-session-label-picker branch November 1, 2026 20:43

This was referenced Nov 1, 2026

cmux NEXT incognito: Feed and iPhone push skip incognito windows (#31210) #31235

Merged

acpmux: index live sessions for exclude-closed hot paths #31213

Merged

perf(next): narrow steady-state chat history observations #31278

Merged

teamleaderleo mentioned this pull request Nov 2, 2026

ci(cmux-next): with no idle gateway runner, the push Release fast check reads the fleet compile on Linux instead of skipping to the Mac queue #32234

Merged

Navigation Menu

Repository navigation

ci(cmux-next): the session label comes only from a placement that judged its queue - #27853

ci(cmux-next): the session label comes only from a placement that judged its queue#27853
teamleaderleo merged 4 commits into
feat-cmux-nextfrom
fix/std-xcode-session-label-picker

teamleaderleo commented Oct 30, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

blacksmith-sh Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

teamleaderleo commented Oct 31, 2026

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

teamleaderleo commented Oct 30, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

What changed

Other single-label literals checked

Reds

Summary by cubic

Uh oh!

teamleaderleo commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blacksmith-sh Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

teamleaderleo commented Oct 31, 2026

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

teamleaderleo commented Nov 1, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teamleaderleo commented Oct 30, 2026 •

edited by cubic-dev-ai Bot

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading

blacksmith-sh Bot commented Oct 30, 2026 •

edited

Loading