Skip to content

ci: route nightly tick controls through fleet picker - #30509

Closed
teamleaderleo wants to merge 2 commits into
feat-cmux-nextfrom
lane/nightly-tick-fleet-first
Closed

teamleaderleo wants to merge 2 commits into
feat-cmux-nextfrom
lane/nightly-tick-fleet-first

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

HOLD: main-source live admission repair is incomplete; the current head must not land.

The nightly-next tick request job was pinned to blacksmith-2vcpu-ubuntu-2404; run 41286823037 remained queued while that pool admitted nothing. The tick and feed-watch control jobs now use the live CI_CONTROL_RUNNER picker result first and fall back to hosted Ubuntu under the declared light Linux exception. The CI routing test rejects the old Blacksmith pin across all nightly-next control workflows.

Validation:

  • PYTHONPATH=tests python3 -m unittest tests.test_ci_nightly_off_blacksmith (11 passed, 1 skipped)
  • PyYAML parse of both changed workflows
  • git diff --check

agent-identity footer --repo manaflow-ai/cmux
agent session


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

The nightly-next tick request job stayed queued because it was pinned to blacksmith-2vcpu-ubuntu-2404, a pool that admitted nothing (run 41286823037). The tick and feed-watch control jobs now use the live CI_CONTROL_RUNNER fleet-picker result first and fall back to hosted Ubuntu as the light-linux exception.

Bug Fixes

  • The CI routing test rejects the old Blacksmith pin across all nightly-next control workflows.

Written for commit 543e915. Summary will update on new commits.

View guided diff Turn on auto-fix

@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@blacksmith-sh

blacksmith-sh Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Found 45 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp9faokg1s/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpdsgefzfx/summary-fail.json (live):/summary-fail.json (live): View Logs
3 runner guard check(s) failed:/3 runner guard check(s) failed: View Logs
3 runner guard check(s) failed:/3 runner guard check(s) failed: View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
GitHub-hosted runner label: .github/workflows/
nightly-next-feed-watch.yml:33: runs-on: ${{ github.repository_owner != 'manaflow-ai
' && 'ubuntu-24.04'
GitHub-hosted runner label: .github/workflows/
nightly-next-feed-watch.yml:33: runs-on: ${{ github.repository_owner != 'manaflow-ai
' && 'ubuntu-24.04'

...and 35 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 1abca6c: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30509 -f ref=lane/nightly-tick-fleet-first -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on c026ef2 to compare; do not merge until green; see lane-rules

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: only the sweep's base merge since, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in .github/workflows/nightly-next-feed-watch.yml, .github/workflows/nightly-next-tick.yml. Listed for a lane to resolve.

@teamleaderleo

teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Held, not resolved: the HOLD (main-source live admission for CI_CONTROL_RUNNER) must land first, and base since moved both runs-on lines to declared pool reads (#32457, fromJSON(vars.RUNNER_POOLS).linux_2 / .linux_4), so this needs rework onto that design (e.g. a control pool whose members are the fleet picker and hosted Ubuntu) rather than a conflict merge that would undo it.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Still held, left as is. State on current base: the dead-pin problem this PR fixed (the tick's literal blacksmith-2vcpu-ubuntu-2404) is now handled by #32457, which reads fromJSON(vars.RUNNER_POOLS).linux_2 (tick) and .linux_4 (feed-watch), so a size that starts nothing is left by the pool. The live-admission picker the HOLD waits on exists on base (scripts/ci/nightly_next_control_runner.py, #30657, refreshed by #31891) but no workflow on main or feat-cmux-next runs it yet. This PR's vars.CI_CONTROL_RUNNER || 'ubuntu-24.04' line would undo the pool read and trips the hosted-runner guard, so it should not be conflict-merged. The remaining work is a picker job in the tick and feed-watch workflows (main first, the schedule source) whose output feeds runs-on, with the pool read as its fallback; once that lands, this PR can be closed as superseded.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Superseded by #32457: base's tick and feed-watch jobs no longer pin a single Blacksmith label; they resolve from the declared runner pool (fromJSON(vars.RUNNER_POOLS).linux_2 / linux_4). This PR's only remaining hunk (vars.CI_CONTROL_RUNNER || 'ubuntu-24.04') and its test would undo that pool design and trip the hosted-runner guard. The HOLD's outstanding work, wiring the live-admission picker (scripts/ci/nightly_next_control_runner.py) into these workflows, is not on base yet and belongs in its own PR built on the runner-pool design.

teamleaderleo added a commit that referenced this pull request Nov 4, 2026
#35630)

#30657 added scripts/ci/nightly_next_control_runner.py and
nightly_next_control_rescue.py as a helper-only prerequisite for #30509,
which closed in favour of the runner-pool design (#32457). No workflow on
feat-cmux-next or main calls either; only their own tests, a pool test and
the test registry named them. The tick already picks its runner from
RUNNER_POOLS.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f08996319b14c7281027

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 543e915d Deployed Nov 1, 2026 by teamleaderleo via cmux-tui tree preflight #97522
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant