Repository navigation
ci(cmux-tui): pin pushes run their daemon tests on their own pool, never the side lane - #23660
Conversation
… lane Red until the fix: cmux-next-daemon-tests sends a cmux-tui-pin-* push to CI_PIN_DAEMON_POOL, else CI_LIGHT_LANE_RUNNER, on every attempt. Under no variable values does a pin push land on the side label, and only the light-lane fallback may share the promotion's daemon pool (CI_NEXT_DAEMON_POOL, else CI_NEXT_PROMOTE_POOL). No nightly job reads CI_PIN_DAEMON_POOL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
Every merged pull request pushes a cmux-tui-pin-* branch, and its cmux_next_ daemon tests fell through to CI_SIDE_LANE_RUNNER, flooding the pull request side lane (2026-10-21, claude-sr2). cmux-next-daemon-tests: a manaflow-ai cmux-tui-pin-* push takes vars.CI_PIN_DAEMON_POOL on every attempt unless it names the side lane or the promotion's daemon pool (CI_NEXT_DAEMON_POOL, else CI_NEXT_PROMOTE_POOL); else vars.CI_LIGHT_LANE_RUNNER (owned pools on) unless it is the side lane; else Blacksmith macOS 15. The light minis may serve both pins and promotion until promotion gets its own label; main's ci-repo-variables.yml warns then. The fork gate and the dispatch input stay first; feat-cmux-next and pull request routing are unchanged. owned_pool_sources counts CI_PIN_DAEMON_POOL as an owned pool variable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base cd9b30e; does not block |
|
Owned by ci-speed on cosmic-gray · session |
|
Companion (repo-variable check, into main): #23661 |
|
Found 9 test failures on Blacksmith runners: Failures
|
|
Bead |
|
Review (independent subagent, correctness first): SHIP at a299616 Reds: |
|
Review (independent subagent, correctness first): SHIP at e370347 Re-review of the update-branch merge: the PR's diff against the new merge base matches the reviewed a299616 diff line for line; the only base change to a touched file (cmux-tui-artifacts.yml: daemon_marker skip in the job's if:) does not touch runs-on, and the pinned runs-on string test passes at the merged head (huge-blue: routes 4/4, nightly-off-blacksmith 9 OK, workflow security 78 OK). |
|
Shipped in:
|
…omotion stays on macOS (#23837) * test(cmux-tui): pin pushes run the cmux_next_ daemon tests on Linux Red until the fix. A cmux-tui-pin-* push routes cmux-next-daemon-tests to vars.CI_PIN_DAEMON_POOL, else blacksmith-16vcpu-ubuntu-2404, on every attempt. The job keeps its name and installs the Linux build dependencies only on Linux, its cargo cache is keyed per OS, and only a macOS run writes the tree's daemon-tests marker, so the feat-cmux-next promotion gate stays on its Mac pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16020-82b9-7501-8b24-24eafedababb&host=cmux16 * ci(cmux-tui): pin pushes run the cmux_next_ daemon tests on Linux Every merged pull request pushes a cmux-tui-pin-* branch, and its cmux_next_ daemon tests took a Mac (the light minis since #23660, before that the side lane). The tests are Rust and pass on Linux, so a pin push now takes vars.CI_PIN_DAEMON_POOL, else blacksmith-16vcpu-ubuntu-2404, on every attempt. The job installs clang and libclang on Linux and keys its cargo cache per OS (the macOS key is unchanged). The feat-cmux-next promotion gate stays on macOS: the job outputs its runner OS and publish-tree-tested writes the daemon-tests marker only for a macOS run. A pin usually publishes the tree its feat-cmux-next merge pushes next; without a marker, that push's preflight finds the tree untested and runs the tests on CI_NEXT_DAEMON_POOL before promotion. The job keeps its name: cmux_tui_publisher_testing.py finds it by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16020-82b9-7501-8b24-24eafedababb&host=cmux16 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… read or a pending compile warns and merges (#24265) * merge-compile-guard tests: only a compile that finished red refuses (red) The guard's GitHub reads timed out under load on cosmic-gray and refused #23660 and #23371 for hours with a SHIP receipt at the head and a green compile. These tests expect a pending compile, an unreadable GitHub, a merge the guard can't name and guards it can't load to warn and let the merge through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue * merge-compile-guard: only a compile that finished red refuses; the rest warns and merges Under load on cosmic-gray (2026-10-21) the guard's GitHub reads timed out and refused #23660 and #23371 for hours with a SHIP receipt at the head and a green compile. Leo: "IF THE MERGE HOOK IS WRONG THEN JUST FUCKING MERGE". HQ's landing gate (require-cmux-guards.py, cmuxterm-hq#2976) already refuses only a missing SHIP, a failed compile, --admin and an unpinned --auto; this repo hook now agrees with it. - A compile still waiting, a GitHub read that failed or ran out of time, a merge the guard can't name and a checkout it can't read are a Hold: a systemMessage plus additionalContext, no permissionDecision, and the merge goes ahead. - A red compile still refuses, and a Hold earlier in the command never hides a red later in it. - guards-from-base.sh and the settings hook warn instead of refusing when they can't load the guards from the base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue * merge-compile-guard tests: a red fleet compile, a crashing guard and a hidden update-branch warning (red) From the Codex review of #24265: a failed build-fleet compile next to a queued Release compile read as a wait; a guard exiting 2 blocks the tool call; a merge warning short-circuited the update-branch warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue * merge-compile-guard: a red fleet compile refuses beside a queued Release compile; a guard's failure and a merge warning never block or hide From the Codex review of #24265: - fleet_red(): a build-fleet compile that finished red with no passing attempt (of it or of the Release compile) refuses before any pending state, and compile_wait() no longer reads it as a wait, so neither a queued Release compile nor a fleet pass turns it into a warning. - guards-from-base.sh runs both guards. A refusal is the answer at once; otherwise every warning is joined into one answer, so a merge's warning never hides the update-branch warning of the same command. A guard that exits non-zero is a warning, never the exit 2 that blocks a tool call. - The settings hook exits 1 (a non-blocking error) when the launcher itself fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue * guards-from-base tests: a refusal in any JSON spacing still refuses (red) Codex review of #24265, round 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue * guards-from-base: read a guard's refusal as JSON, not as text The loop matched "permissionDecision": "deny" literally, so a valid refusal written with other spacing was merged into a warning and lost (Codex review of #24265, round 2). The merger now parses each answer and passes a refusal through as it is before joining warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue * guards-from-base tests: a refusal never waits for the next guard (red) Codex review of #24265, round 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue * guards-from-base: a guard's refusal is the answer the moment it is given Both guards ran before any refusal was read, so a slow update-branch warning could run the hook past its 120 s timeout with the refusal unprinted, and the merge went through (Codex review of #24265, round 3). Each answer is read as JSON right after its guard; a refusal is printed and the loop ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
Summary
Cause 1 (a) of the 2026-10-21 CI backlog. The repo-variable check that goes with it is the companion PR into
main, linked below.Every merged PR pushes a
cmux-tui-pin-<sha12>branch. Before this change, that push'scmux-next-daemon-testsjob fell through tovars.CI_SIDE_LANE_RUNNER(glaeda-side-std-xcode-26.6), so pin daemon tests queued behind PR side lanes and filled them.The job now gets a pin clause in its
runs-on. It comes after the feat daemon and promote clauses and before the side-lane clause, and applies to a manaflow-aicmux-tui-pin-*push on every attempt:vars.CI_PIN_DAEMON_POOL, unless it equals the side lane or promotion's daemon pool (CI_NEXT_DAEMON_POOL, elseCI_NEXT_PROMOTE_POOL);vars.CI_LIGHT_LANE_RUNNER, when the owned pools are on and it is not the side lane;blacksmith-6vcpu-macos-15.The fork gate and
inputs.macos_runnerstill come first. Routing forfeat-cmux-nextand PRs is unchanged.CI_PIN_DAEMON_POOLjoins the owned-pool variables (owned_pool_sources.py, and the regex in the nightly-off-Blacksmith test).How it lands (claude-sr2's decision):
CI_PIN_DAEMON_POOLstays unset, so pins take the light minis. Until lane-host's HQ label for promotion's daemon tests is online, that is the same pool asCI_NEXT_DAEMON_POOL, and the companion check warns about it. After that,CI_NEXT_DAEMON_POOLmoves to the new label.Not covered:
macos_runnerinput, notruns-on.workflow_dispatchrun on a pin ref: it routes as before.Testing
The tests are in 945c458 and the fix is in a299616. They were run on huge-blue.
Red at 945c458: three failures in
tests/test_ci_tui_routes_avoid_nightly_label.pyand one intests/test_tui_publish_workflow_security.py'sruns-onstring test.Green at a299616:
tests/test_ci_tui_routes_avoid_nightly_label.pytest_ci_fork_runner_routingtest_ci_side_lane_placementtest_cmux_tui_publisher_testingtest_ci_nightly_off_blacksmithtest_cmux_tui_pin_duplicate_skiptest_cmux_tui_lane_daemon_testsWhat the new routes tests cover:
runs-onsends a pin push to the side label, and only the light fallback may equal the promotion pool.actionlint is clean, and
prepush-gate.pyreports no new violations.Observed / Not run
test_tui_publish_workflow_security.py: its other 16 tests need the realcmux-tui/sources, and they error the same way on red and green.test_ci_owned_pool_sources: it needs both branch refs in a real clone. CI runs both.Changelog
none
Agent session: cosmic-gray
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Stops
cmux-tui-pin-*branch pushes (one per merged pull request) from queuing their daemon tests on the PR side-lane runner, which was filling the lane and causing CI backlogs.The
cmux-next-daemon-testsjob now routes pin pushes tovars.CI_PIN_DAEMON_POOLon every attempt. If that pool equals the side lane or the promotion's daemon pool, it falls back tovars.CI_LIGHT_LANE_RUNNERand then to Blacksmith macOS 15.CI_PIN_DAEMON_POOLis registered as an owned pool variable; routing forfeat-cmux-nextand pull requests is unchanged.CI_PIN_DAEMON_POOLinitially stays unset, so pins take the light minis until promotion's daemon tests get their own label. Tests cover 1,152 variable combinations to guarantee pin pushes never land on the side label.Written for commit e370347. Summary will update on new commits.