Repository navigation
ci(nightly-next): dispatched and re-run builds take the owned-Mac pick too - #26237
Merged
Merged
Conversation
Collaborator
Author
|
Owned by an agent session · session |
Collaborator
Author
|
Bead |
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 15:22 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 15:22 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 15:23 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 15:24 — with
GitHub Actions
Inactive
This was referenced Oct 27, 2026
teamleaderleo
force-pushed
the
leo/nightly-picker-dispatch-rerun
branch
from
October 27, 2026 15:42
2215af0 to
1903bc1
Compare
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 15:42 — with
GitHub Actions
Inactive
Contributor
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
web and size checks: green |
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 15:42 — with
GitHub Actions
Inactive
|
Found 2 test failures on Blacksmith runners: Failures
|
…k too pick-nightly-build-runner minted its routing token only on attempt 1 of a nightly-next push, and both the picker and build-nightly-app refused an owned pick past attempt 1, so a workflow_dispatch run or a re-run never got the dedicated builder or an owned Mac and queued on Blacksmith macOS. The token now mints for nightly-next push and dispatch runs at any attempt, the picker reports the attempt it picked for, and the build takes `owned` only when that attempt is its own: a re-run of failed jobs keeps the earlier pick and takes `blacksmith`, so a refused owned label is never taken again. An owned pick outside the rescue's workflow_run watch uploads the owned-pool-watch marker, and `owned` stands only once that upload succeeded. The picker reads the routing key in the ci-routing-agent environment, like the other dispatchable pickers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
… attempt rule as the build #26253 gave the helper pick-nightly-build-runner's owned label on run_attempt 1. With dispatches and re-picking re-runs now picking an owned Mac, the helper takes the pick when it was made for its own attempt, like build-nightly-app; a re-run of failed jobs keeps the older pick and takes Blacksmith. The rescue already watches the helper's owned label (job_pool()). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo
force-pushed
the
leo/nightly-picker-dispatch-rerun
branch
from
October 27, 2026 16:32
1903bc1 to
30a95f7
Compare
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 16:33 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 16:33 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 16:53 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 16:54 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 16:54 — with
GitHub Actions
Inactive
teamleaderleo
temporarily deployed
to
ci-routing-agent
October 27, 2026 16:55 — with
GitHub Actions
Inactive
Contributor
|
Shipped in:
|
teamleaderleo
added a commit
that referenced
this pull request
Oct 28, 2026
…wned Mac too pick-nightly-sign-runner's owned pick for the secret-free verify jobs ran only on attempt 1 of a push. Dispatch run 40594111618 reached signing on attempt 3: signer-dmg-image waited 51 min on blacksmith-6vcpu-macos-26 and signer-verify queued there behind it, with every Blacksmith macOS 26 label stalled. The pick now runs on nightly-next push and dispatch runs at any attempt, as pick-nightly-build-runner does since #26237: its `attempt` output names the attempt the pick was made for, the verify jobs read `owned` only for that attempt (a re-run of failed jobs that keeps it takes `alternate`), and outside attempt 1 of a push it leaves the rescue sweeper's owned-pool-watch marker, without which `owned` is empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo
added a commit
that referenced
this pull request
Oct 28, 2026
…wned Mac too (#26489) pick-nightly-sign-runner's owned pick for the secret-free verify jobs ran only on attempt 1 of a push. Dispatch run 40594111618 reached signing on attempt 3: signer-dmg-image waited 51 min on blacksmith-6vcpu-macos-26 and signer-verify queued there behind it, with every Blacksmith macOS 26 label stalled. The pick now runs on nightly-next push and dispatch runs at any attempt, as pick-nightly-build-runner does since #26237: its `attempt` output names the attempt the pick was made for, the verify jobs read `owned` only for that attempt (a re-run of failed jobs that keeps it takes `alternate`), and outside attempt 1 of a push it leaves the rescue sweeper's owned-pool-watch marker, without which `owned` is empty. Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
Why
Brief: "Today it routes only attempt-1 push runs to owned (glaeda) labels. workflow_dispatch runs and attempt-2+ re-runs never get the routing token. They fall through to Blacksmith macOS, which is stalled, so a dispatched or re-run tip nightly can never build on an owned mini. They must go through the same picker as attempt-1 push runs: Linux or owned Macs first, Blacksmith last."
What
pick-nightly-build-runnernow mints it for nightly-next push andworkflow_dispatchruns at any attempt. Before, it minted only for a push on attempt 1.scripts/ci/nightly_build_runner.pyno longer refuses an owned pick when the attempt is above 1. It now outputsattempt, the attempt the pick was made for.build-nightly-appruns-on: it takesownedonly whenneeds.pick-nightly-build-runner.outputs.attempt == github.run_attempt, meaning the pick was made fresh for this attempt.blacksmith, and a refusing label is never taken again.owned-pool-watchmarker so the sweeper adopts the run.ownedoutput is set only if that upload succeeded. An owned job is never left without a rescue (Codex review finding, fixed). A Codex gpt-6-astra xhigh review returned SHIP on the rebased head.ci-routing-agentenvironment, like the other dispatchable pickers (iroh-release-gate.yml, test-ios.yml). Now that a dispatch can mint the token,tests/test_supplychain_security.pyneeds this. That environment has no protection rules and no branch policy. No secret, permission or environment is added.Unchanged: no evicting concurrency group, no
cancel-in-progress, and only labels the rescue watches (pr_runner_pool.persistent, so glaeda-side-; glaeda-dev- is still never offered).The main-side rescue change, #26236, has landed (23bd715). This PR is rebased onto #26048 (dedicated builder).
dedicated_route()'s attempt-1-only guard is removed too, so a dispatch, or a re-run that picks again, tries the dedicated builder first, then an idle owned Mac, then Blacksmith. Main's rescue already watches the dedicated label.Tests (
cmux-ci test, Linux)tests/test_nightly_build_runner.py: 27 tests OK. New tests:workflow_dispatchrun and arun_attempt=2run each get the routing token;build-nightly-app's runs-on, and both runs land on the freeglaeda-side-std-xcode-26.6. With a stale pick (a re-run of failed jobs), the same run lands on Blacksmith;ownedoutput's gate on the upload.test_nightly_trusted_routing.py,test_nightly_step_coverage.py,test_ci_nightly_off_blacksmith.py,test_ci_owned_pool_rescue.py,test_nightly_sign_runner.py: OK.test_supplychain_security.py: this job no longer fails it. The remaining failure,pick-nightly-sign-runner, also fails on base.Before: a dispatch, or attempt 2 with a free owned Mac, went to
blacksmith-*-macos-26. After: both go toglaeda-side-std-xcode-26.6.Known limit: the tick's cancelled-tip re-run (
gh run rerun --failed) keeps the picker's earlier output when the picker had already succeeded. Those re-runs still take Blacksmith, which is the refusal-safe path.Agent session: agent session
🤖 Generated with Claude Code