Skip to content

ci(cmux-tui): pin pushes run their daemon tests on their own pool, never the side lane - #23660

Merged
teamleaderleo merged 3 commits into
feat-cmux-nextfrom
ci-speed/pin-daemon-pool
Oct 21, 2026
Merged

teamleaderleo merged 3 commits into
feat-cmux-nextfrom
ci-speed/pin-daemon-pool

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Cause 1 (a) of the 2026-10-21 CI backlog. The repo-variable check that goes with it is the companion PR into main, linked below.

Every merged PR pushes a cmux-tui-pin-<sha12> branch. Before this change, that push's cmux-next-daemon-tests job fell through to vars.CI_SIDE_LANE_RUNNER (glaeda-side-std-xcode-26.6), so pin daemon tests queued behind PR side lanes and filled them.

The job now gets a pin clause in its runs-on. It comes after the feat daemon and promote clauses and before the side-lane clause, and applies to a manaflow-ai cmux-tui-pin-* push on every attempt:

  1. vars.CI_PIN_DAEMON_POOL, unless it equals the side lane or promotion's daemon pool (CI_NEXT_DAEMON_POOL, else CI_NEXT_PROMOTE_POOL);
  2. otherwise vars.CI_LIGHT_LANE_RUNNER, when the owned pools are on and it is not the side lane;
  3. otherwise blacksmith-6vcpu-macos-15.

The fork gate and inputs.macos_runner still come first. Routing for feat-cmux-next and PRs is unchanged. CI_PIN_DAEMON_POOL joins the owned-pool variables (owned_pool_sources.py, and the regex in the nightly-off-Blacksmith test).

How it lands (claude-sr2's decision): CI_PIN_DAEMON_POOL stays unset, so pins take the light minis. Until lane-host's HQ label for promotion's daemon tests is online, that is the same pool as CI_NEXT_DAEMON_POOL, and the companion check warns about it. After that, CI_NEXT_DAEMON_POOL moves to the new label.

Not covered:

  • A pin push's build legs: their runner is the reusable workflow's macos_runner input, not runs-on.
  • A workflow_dispatch run on a pin ref: it routes as before.

Testing

The tests are in 945c458 and the fix is in a299616. They were run on huge-blue.

Red at 945c458: three failures in tests/test_ci_tui_routes_avoid_nightly_label.py and one in tests/test_tui_publish_workflow_security.py's runs-on string test.

Green at a299616:

Test Result
tests/test_ci_tui_routes_avoid_nightly_label.py 7 of 7
test_ci_fork_runner_routing 20 OK
test_ci_side_lane_placement 31 OK
test_cmux_tui_publisher_testing 11 OK
test_ci_nightly_off_blacksmith 9 OK
test_cmux_tui_pin_duplicate_skip pass
test_cmux_tui_lane_daemon_tests pass

What the new routes tests cover:

  • A pin push goes to the pin pool when it is set and to the light lane when it is unset, on attempts 1 to 3.
  • A guard over 1,152 variable combinations: no runs-on sends a pin push to the side label, and only the light fallback may equal the promotion pool.
  • The state this lands in.

actionlint is clean, and prepush-gate.py reports no new violations.

Observed / Not run

  • Observed: the red and green runs above.
  • Not run:
    • Most of test_tui_publish_workflow_security.py: its other 16 tests need the real cmux-tui/ sources, and they error the same way on red and green.
    • test_ci_owned_pool_sources: it needs both branch refs in a real clone. CI runs both.

Changelog

none

Agent session: cosmic-gray

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Stops cmux-tui-pin-* branch pushes (one per merged pull request) from queuing their daemon tests on the PR side-lane runner, which was filling the lane and causing CI backlogs.

The cmux-next-daemon-tests job now routes pin pushes to vars.CI_PIN_DAEMON_POOL on every attempt. If that pool equals the side lane or the promotion's daemon pool, it falls back to vars.CI_LIGHT_LANE_RUNNER and then to Blacksmith macOS 15. CI_PIN_DAEMON_POOL is registered as an owned pool variable; routing for feat-cmux-next and pull requests is unchanged.

CI_PIN_DAEMON_POOL initially stays unset, so pins take the light minis until promotion's daemon tests get their own label. Tests cover 1,152 variable combinations to guarantee pin pushes never land on the side label.

Written for commit e370347. Summary will update on new commits.

View guided diff Turn on auto-fix

teamleaderleo and others added 2 commits October 21, 2026 12:28
… lane

Red until the fix: cmux-next-daemon-tests sends a cmux-tui-pin-* push to
CI_PIN_DAEMON_POOL, else CI_LIGHT_LANE_RUNNER, on every attempt. Under no
variable values does a pin push land on the side label, and only the
light-lane fallback may share the promotion's daemon pool
(CI_NEXT_DAEMON_POOL, else CI_NEXT_PROMOTE_POOL). No nightly job reads
CI_PIN_DAEMON_POOL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
Every merged pull request pushes a cmux-tui-pin-* branch, and its
cmux_next_ daemon tests fell through to CI_SIDE_LANE_RUNNER, flooding the
pull request side lane (2026-10-21, claude-sr2).

cmux-next-daemon-tests: a manaflow-ai cmux-tui-pin-* push takes
vars.CI_PIN_DAEMON_POOL on every attempt unless it names the side lane or
the promotion's daemon pool (CI_NEXT_DAEMON_POOL, else
CI_NEXT_PROMOTE_POOL); else vars.CI_LIGHT_LANE_RUNNER (owned pools on)
unless it is the side lane; else Blacksmith macOS 15. The light minis may
serve both pins and promotion until promotion gets its own label; main's
ci-repo-variables.yml warns then. The fork gate and the dispatch input stay
first; feat-cmux-next and pull request routing are unchanged.
owned_pool_sources counts CI_PIN_DAEMON_POOL as an owned pool variable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15bf1-fdac-78a0-ac2e-a3aef81402c7&host=cosmic-gray
@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review e370347: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23660 -f ref=ci-speed/pin-daemon-pool -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base cd9b30e; does not block

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by ci-speed on cosmic-gray · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Companion (repo-variable check, into main): #23661

@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 9 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp0xroi21s/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmp7uj64mzs/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test_a_file_a_test_reads_selects_that_test (main.PullRequestTiers.test_a_file_a_tes
t_reads_selects_that_test)/
test_a_file_a_test_reads_selects_that_test (main.PullRequestTiers.test_a_file_a_tes
t_reads_selects_that_test)
View Logs
test_a_shared_module_selects_its_dependents (main.PullRequestTiers.test_a_shared_mo
dule_selects_its_dependents)/
test_a_shared_module_selects_its_dependents (main.PullRequestTiers.test_a_shared_mo
dule_selects_its_dependents)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-35mxy created for this PR by the cmux coordinator.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent, correctness first): SHIP at a299616

Reds: cmux-next checks (Tier routing, test_cmux_next_route ChiefViewsTests + ChiefWorkspaceName namespace-type) and perf gate ([relaunch] pane_snapshots 4.0 > 1 per pane) are base reds unrelated to this runs-on change; cmux app scheme compile (Debug) passed. CI_NEXT_DAEMON_POOL is now glaeda-daemon-trusted-light-xcode-26.6 (cmux19, HQ #2790), so pins on the light fallback no longer share promotion's pool.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent, correctness first): SHIP at e370347

Re-review of the update-branch merge: the PR's diff against the new merge base matches the reviewed a299616 diff line for line; the only base change to a touched file (cmux-tui-artifacts.yml: daemon_marker skip in the job's if:) does not touch runs-on, and the pinned runs-on string test passes at the merged head (huge-blue: routes 4/4, nightly-off-blacksmith 9 OK, workflow security 78 OK).

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

teamleaderleo added a commit that referenced this pull request Oct 22, 2026
…omotion stays on macOS (#23837)

* test(cmux-tui): pin pushes run the cmux_next_ daemon tests on Linux

Red until the fix. A cmux-tui-pin-* push routes cmux-next-daemon-tests to
vars.CI_PIN_DAEMON_POOL, else blacksmith-16vcpu-ubuntu-2404, on every
attempt. The job keeps its name and installs the Linux build dependencies
only on Linux, its cargo cache is keyed per OS, and only a macOS run writes
the tree's daemon-tests marker, so the feat-cmux-next promotion gate stays
on its Mac pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16020-82b9-7501-8b24-24eafedababb&host=cmux16

* ci(cmux-tui): pin pushes run the cmux_next_ daemon tests on Linux

Every merged pull request pushes a cmux-tui-pin-* branch, and its
cmux_next_ daemon tests took a Mac (the light minis since #23660, before
that the side lane). The tests are Rust and pass on Linux, so a pin push
now takes vars.CI_PIN_DAEMON_POOL, else blacksmith-16vcpu-ubuntu-2404, on
every attempt. The job installs clang and libclang on Linux and keys its
cargo cache per OS (the macOS key is unchanged).

The feat-cmux-next promotion gate stays on macOS: the job outputs its
runner OS and publish-tree-tested writes the daemon-tests marker only for
a macOS run. A pin usually publishes the tree its feat-cmux-next merge
pushes next; without a marker, that push's preflight finds the tree
untested and runs the tests on CI_NEXT_DAEMON_POOL before promotion.

The job keeps its name: cmux_tui_publisher_testing.py finds it by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16020-82b9-7501-8b24-24eafedababb&host=cmux16

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Oct 22, 2026
… read or a pending compile warns and merges (#24265)

* merge-compile-guard tests: only a compile that finished red refuses (red)

The guard's GitHub reads timed out under load on cosmic-gray and refused #23660 and #23371 for hours with
a SHIP receipt at the head and a green compile. These tests expect a pending compile, an unreadable GitHub,
a merge the guard can't name and guards it can't load to warn and let the merge through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

* merge-compile-guard: only a compile that finished red refuses; the rest warns and merges

Under load on cosmic-gray (2026-10-21) the guard's GitHub reads timed out and refused #23660 and
#23371 for hours with a SHIP receipt at the head and a green compile. Leo: "IF THE MERGE HOOK IS
WRONG THEN JUST FUCKING MERGE". HQ's landing gate (require-cmux-guards.py, cmuxterm-hq#2976)
already refuses only a missing SHIP, a failed compile, --admin and an unpinned --auto; this repo
hook now agrees with it.

- A compile still waiting, a GitHub read that failed or ran out of time, a merge the guard can't
  name and a checkout it can't read are a Hold: a systemMessage plus additionalContext, no
  permissionDecision, and the merge goes ahead.
- A red compile still refuses, and a Hold earlier in the command never hides a red later in it.
- guards-from-base.sh and the settings hook warn instead of refusing when they can't load the
  guards from the base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

* merge-compile-guard tests: a red fleet compile, a crashing guard and a hidden update-branch warning (red)

From the Codex review of #24265: a failed build-fleet compile next to a queued Release compile read as
a wait; a guard exiting 2 blocks the tool call; a merge warning short-circuited the update-branch warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

* merge-compile-guard: a red fleet compile refuses beside a queued Release compile; a guard's failure and a merge warning never block or hide

From the Codex review of #24265:
- fleet_red(): a build-fleet compile that finished red with no passing attempt (of it or of the Release
  compile) refuses before any pending state, and compile_wait() no longer reads it as a wait, so neither a
  queued Release compile nor a fleet pass turns it into a warning.
- guards-from-base.sh runs both guards. A refusal is the answer at once; otherwise every warning is joined
  into one answer, so a merge's warning never hides the update-branch warning of the same command. A guard
  that exits non-zero is a warning, never the exit 2 that blocks a tool call.
- The settings hook exits 1 (a non-blocking error) when the launcher itself fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

* guards-from-base tests: a refusal in any JSON spacing still refuses (red)

Codex review of #24265, round 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

* guards-from-base: read a guard's refusal as JSON, not as text

The loop matched "permissionDecision": "deny" literally, so a valid refusal written with other spacing was
merged into a warning and lost (Codex review of #24265, round 2). The merger now parses each answer and
passes a refusal through as it is before joining warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

* guards-from-base tests: a refusal never waits for the next guard (red)

Codex review of #24265, round 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

* guards-from-base: a guard's refusal is the answer the moment it is given

Both guards ran before any refusal was read, so a slow update-branch warning could run the hook past its
120 s timeout with the refusal unprinted, and the merge went through (Codex review of #24265, round 3).
Each answer is read as JSON right after its guard; a refusal is printed and the loop ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a1602e-bc09-7990-8c75-5c412a66b114&host=Air-Blue

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant