Skip to content

ci(nightly-next): dispatched and re-run builds take the owned-Mac pick too - #26237

Merged
teamleaderleo merged 2 commits into
feat-cmux-nextfrom
leo/nightly-picker-dispatch-rerun
Oct 27, 2026
Merged

teamleaderleo merged 2 commits into
feat-cmux-nextfrom
leo/nightly-picker-dispatch-rerun

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Brief: "Today it routes only attempt-1 push runs to owned (glaeda) labels. workflow_dispatch runs and attempt-2+ re-runs never get the routing token. They fall through to Blacksmith macOS, which is stalled, so a dispatched or re-run tip nightly can never build on an owned mini. They must go through the same picker as attempt-1 push runs: Linux or owned Macs first, Blacksmith last."

What

  • Routing token: pick-nightly-build-runner now mints it for nightly-next push and workflow_dispatch runs at any attempt. Before, it minted only for a push on attempt 1.
  • Picker: scripts/ci/nightly_build_runner.py no longer refuses an owned pick when the attempt is above 1. It now outputs attempt, the attempt the pick was made for.
  • build-nightly-app runs-on: it takes owned only when needs.pick-nightly-build-runner.outputs.attempt == github.run_attempt, meaning the pick was made fresh for this attempt.
    • A push, a dispatch, or a full re-run picks again and gets an idle owned Mac first.
    • A re-run of failed jobs (how the rescue moves a refused or stuck job) keeps the earlier attempt's pick. So it takes blacksmith, and a refusing label is never taken again.
  • Watch marker: an owned pick outside attempt 1 of a push, which the rescue's workflow_run watch does not cover, uploads the owned-pool-watch marker so the sweeper adopts the run.
    • The job's owned output is set only if that upload succeeded. An owned job is never left without a rescue (Codex review finding, fixed). A Codex gpt-6-astra xhigh review returned SHIP on the rebased head.
  • Environment: the picker job runs in the existing ci-routing-agent environment, like the other dispatchable pickers (iroh-release-gate.yml, test-ios.yml). Now that a dispatch can mint the token, tests/test_supplychain_security.py needs this. That environment has no protection rules and no branch policy. No secret, permission or environment is added.

Unchanged: no evicting concurrency group, no cancel-in-progress, and only labels the rescue watches (pr_runner_pool.persistent, so glaeda-side-; glaeda-dev- is still never offered).

The main-side rescue change, #26236, has landed (23bd715). This PR is rebased onto #26048 (dedicated builder). dedicated_route()'s attempt-1-only guard is removed too, so a dispatch, or a re-run that picks again, tries the dedicated builder first, then an idle owned Mac, then Blacksmith. Main's rescue already watches the dedicated label.

Tests (cmux-ci test, Linux)

  • tests/test_nightly_build_runner.py: 27 tests OK. New tests:
    • a workflow_dispatch run and a run_attempt=2 run each get the routing token;
    • end to end, the picker's outputs feed into build-nightly-app's runs-on, and both runs land on the free glaeda-side-std-xcode-26.6. With a stale pick (a re-run of failed jobs), the same run lands on Blacksmith;
    • the marker condition and the owned output's gate on the upload.
  • test_nightly_trusted_routing.py, test_nightly_step_coverage.py, test_ci_nightly_off_blacksmith.py, test_ci_owned_pool_rescue.py, test_nightly_sign_runner.py: OK.
  • test_supplychain_security.py: this job no longer fails it. The remaining failure, pick-nightly-sign-runner, also fails on base.

Before: a dispatch, or attempt 2 with a free owned Mac, went to blacksmith-*-macos-26. After: both go to glaeda-side-std-xcode-26.6.

Known limit: the tick's cancelled-tip re-run (gh run rerun --failed) keeps the picker's earlier output when the picker had already succeeded. Those re-runs still take Blacksmith, which is the refusal-safe path.

Agent session: agent session

🤖 Generated with Claude Code

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-5v8x4j created for this PR by the cmux coordinator.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 30a95f7: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr26237 -f ref=leo/nightly-picker-dispatch-rerun -f platform=macos -f backend_mode=local

web and size checks: green

@blacksmith-sh

blacksmith-sh Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
nightly.yml must derive the embedded key and sign every appcast through the helper/
nightly.yml must derive the embedded key and sign every appcast through the helper
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

teamleaderleo and others added 2 commits October 27, 2026 09:09
…k too

pick-nightly-build-runner minted its routing token only on attempt 1 of a
nightly-next push, and both the picker and build-nightly-app refused an owned
pick past attempt 1, so a workflow_dispatch run or a re-run never got the
dedicated builder or an owned Mac and queued on Blacksmith macOS. The token now
mints for nightly-next push and dispatch runs at any attempt, the picker reports
the attempt it picked for, and the build takes `owned` only when that attempt is
its own: a re-run of failed jobs keeps the earlier pick and takes `blacksmith`,
so a refused owned label is never taken again. An owned pick outside the
rescue's workflow_run watch uploads the owned-pool-watch marker, and `owned`
stands only once that upload succeeded. The picker reads the routing key in the
ci-routing-agent environment, like the other dispatchable pickers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
… attempt rule as the build

#26253 gave the helper pick-nightly-build-runner's owned label on run_attempt 1. With dispatches
and re-picking re-runs now picking an owned Mac, the helper takes the pick when it was made for
its own attempt, like build-nightly-app; a re-run of failed jobs keeps the older pick and takes
Blacksmith. The rescue already watches the helper's owned label (job_pool()).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
@teamleaderleo
teamleaderleo force-pushed the leo/nightly-picker-dispatch-rerun branch from 1903bc1 to 30a95f7 Compare October 27, 2026 16:32
@teamleaderleo
teamleaderleo merged commit 5562e09 into feat-cmux-next Oct 27, 2026
194 of 197 checks passed
@teamleaderleo
teamleaderleo deleted the leo/nightly-picker-dispatch-rerun branch October 27, 2026 17:00
@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

teamleaderleo added a commit that referenced this pull request Oct 28, 2026
…wned Mac too

pick-nightly-sign-runner's owned pick for the secret-free verify jobs ran
only on attempt 1 of a push. Dispatch run 40594111618 reached signing on
attempt 3: signer-dmg-image waited 51 min on blacksmith-6vcpu-macos-26 and
signer-verify queued there behind it, with every Blacksmith macOS 26 label
stalled.

The pick now runs on nightly-next push and dispatch runs at any attempt,
as pick-nightly-build-runner does since #26237: its `attempt` output names
the attempt the pick was made for, the verify jobs read `owned` only for
that attempt (a re-run of failed jobs that keeps it takes `alternate`),
and outside attempt 1 of a push it leaves the rescue sweeper's
owned-pool-watch marker, without which `owned` is empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo added a commit that referenced this pull request Oct 28, 2026
…wned Mac too (#26489)

pick-nightly-sign-runner's owned pick for the secret-free verify jobs ran
only on attempt 1 of a push. Dispatch run 40594111618 reached signing on
attempt 3: signer-dmg-image waited 51 min on blacksmith-6vcpu-macos-26 and
signer-verify queued there behind it, with every Blacksmith macOS 26 label
stalled.

The pick now runs on nightly-next push and dispatch runs at any attempt,
as pick-nightly-build-runner does since #26237: its `attempt` output names
the attempt the pick was made for, the verify jobs read `owned` only for
that attempt (a re-run of failed jobs that keeps it takes `alternate`),
and outside attempt 1 of a push it leaves the rescue sweeper's
owned-pool-watch marker, without which `owned` is empty.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 30a95f73 Deployed Oct 27, 2026 by teamleaderleo via wait for same-key tree publisher #65921
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant