Repository navigation
ci(nightly-next): decide's light jobs take a free mini, Blacksmith last - #29859
teamleaderleo wants to merge 3 commits into
Conversation
resolve-nightly-cmux-tui-client and find-nightly-next-recovery need only decide, hold no environment and no secret, and run git, gh, curl, bash and stdlib Python. On every nightly-next run they took blacksmith-32vcpu-ubuntu-2404 with no fleet route (run 41254086065). decide's last steps now place them with scripts/ci/portable_job_placement.py (NIGHTLY=1), the way build-nightly-app places its light jobs: a mini idle now, else their Blacksmith Linux set member at once. A re-run of failed jobs keeps the Blacksmith route (light_attempt). Both leave the Blacksmith-only debt in test_ci_nightly_off_blacksmith.py and take class portable-checks in runners.json; runner_routing_table accepts decide's placement output as the nightly mini term. Leo, 2026-10-31: "FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-4fbe72147bb64b0edb5d
… janitor snapshot Review at a581f92 (gpt-6.1-sol xhigh): the two new decide steps had no nightly-step-coverage entry (release-notary guard), and decide's token lacked actions: read, so the placement could not read the queue janitor's snapshot artifact and would have kept both jobs on Blacksmith. The light-on-Linux test's silent needs now applies only to the decide-placed jobs, and the decide test pins the placement as decide's last step, after the mint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-4fbe72147bb64b0edb5d
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="scripts/ci/runner_routing_table.py">
<violation number="1" location="scripts/ci/runner_routing_table.py:85">
P3: The comment above this regex still says only build-nightly-app places the light jobs. The regex now also accepts decide's `light_mini_runner`, so update the comment to name both producers and which jobs each places.</violation>
</file>
<file name="tests/test_nightly_trusted_routing.py">
<violation number="1" location="tests/test_nightly_trusted_routing.py:519">
P2: These assertions lock in a Blacksmith fallback for two nightly jobs. AGENTS.md forbids Blacksmith routes on nightly jobs, including fallbacks. If the 'Blacksmith last' design is intended, the AGENTS.md exception should be recorded and the policy test updated to match.</violation>
</file>
<file name="tests/test_ci_nightly_off_blacksmith.py">
<violation number="1" location="tests/test_ci_nightly_off_blacksmith.py:226">
P3: The docstring and test name still say these light jobs stay on Linux, but the new mini branch lets two of them run on a Mac mini first. Update the docstring and rename the test to match the placement exception.</violation>
</file>
<file name=".github/workflows/nightly.yml">
<violation number="1" location=".github/workflows/nightly.yml:835">
P1: The placement snapshot read uses `github.token`, but decide's explicit `permissions:` block does not grant `actions: read`, so the artifact listing fails. The failure is swallowed, so no job is placed on a mini and the feature silently does nothing. Add `permission-actions: read` to the `light-route-token` mint step (as `snapshot-read-token` does) and pass that token as `SNAPSHOT_TOKEN`.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| FLEET_LINUX: "" | ||
| SIDE_LABEL: ${{ vars.CI_SIDE_LANE_RUNNER }} | ||
| ROUTE_TOKEN: ${{ steps.light-route-token.outputs.token }} | ||
| SNAPSHOT_TOKEN: ${{ github.token }} |
There was a problem hiding this comment.
P1: The placement snapshot read uses github.token, but decide's explicit permissions: block does not grant actions: read, so the artifact listing fails. The failure is swallowed, so no job is placed on a mini and the feature silently does nothing. Add permission-actions: read to the light-route-token mint step (as snapshot-read-token does) and pass that token as SNAPSHOT_TOKEN.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 834:
<comment>The placement snapshot read uses `github.token`, but decide's explicit `permissions:` block does not grant `actions: read`, so the artifact listing fails. The failure is swallowed, so no job is placed on a mini and the feature silently does nothing. Add `permission-actions: read` to the `light-route-token` mint step (as `snapshot-read-token` does) and pass that token as `SNAPSHOT_TOKEN`.</comment>
<file context>
@@ -802,6 +806,34 @@ jobs:
+ FLEET_LINUX: ""
+ SIDE_LABEL: ${{ vars.CI_SIDE_LANE_RUNNER }}
+ ROUTE_TOKEN: ${{ steps.light-route-token.outputs.token }}
+ SNAPSHOT_TOKEN: ${{ github.token }}
+ run: python3 scripts/ci/portable_job_placement.py
+
</file context>
| def test_no_placement_a_stale_one_or_a_fork_keeps_the_old_route(self): | ||
| for job in self.LIGHT: | ||
| for changes, expected in ( | ||
| ({'needs.decide.outputs.light_mini_jobs': ''}, 'blacksmith-32vcpu-ubuntu-2404'), |
There was a problem hiding this comment.
P2: These assertions lock in a Blacksmith fallback for two nightly jobs. AGENTS.md forbids Blacksmith routes on nightly jobs, including fallbacks. If the 'Blacksmith last' design is intended, the AGENTS.md exception should be recorded and the policy test updated to match.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_trusted_routing.py, line 519:
<comment>These assertions lock in a Blacksmith fallback for two nightly jobs. AGENTS.md forbids Blacksmith routes on nightly jobs, including fallbacks. If the 'Blacksmith last' design is intended, the AGENTS.md exception should be recorded and the policy test updated to match.</comment>
<file context>
@@ -482,6 +483,78 @@ def test_the_build_job_mints_only_the_routing_token_for_the_placement(self):
+ def test_no_placement_a_stale_one_or_a_fork_keeps_the_old_route(self):
+ for job in self.LIGHT:
+ for changes, expected in (
+ ({'needs.decide.outputs.light_mini_jobs': ''}, 'blacksmith-32vcpu-ubuntu-2404'),
+ ({'needs.decide.outputs.light_attempt': ''}, 'blacksmith-32vcpu-ubuntu-2404'),
+ # A re-run of failed jobs keeps decide's earlier outputs: the mini may be busy by now.
</file context>
| # nightly.yml's light jobs: build-nightly-app's last step places them (NIGHTLY=1) on a free mini, else | ||
| # Blacksmith. They have no fleet Linux step: a nightly job never takes vars.LINUX_RUNNER as it is. | ||
| NIGHTLY_MINI = re.compile(r"needs\.build-nightly-app\.outputs\.light_mini_runner$") | ||
| NIGHTLY_MINI = re.compile(r"needs\.(?:build-nightly-app|decide)\.outputs\.light_mini_runner$") |
There was a problem hiding this comment.
P3: The comment above this regex still says only build-nightly-app places the light jobs. The regex now also accepts decide's light_mini_runner, so update the comment to name both producers and which jobs each places.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/runner_routing_table.py, line 85:
<comment>The comment above this regex still says only build-nightly-app places the light jobs. The regex now also accepts decide's `light_mini_runner`, so update the comment to name both producers and which jobs each places.</comment>
<file context>
@@ -82,7 +82,7 @@
# nightly.yml's light jobs: build-nightly-app's last step places them (NIGHTLY=1) on a free mini, else
# Blacksmith. They have no fleet Linux step: a nightly job never takes vars.LINUX_RUNNER as it is.
-NIGHTLY_MINI = re.compile(r"needs\.build-nightly-app\.outputs\.light_mini_runner$")
+NIGHTLY_MINI = re.compile(r"needs\.(?:build-nightly-app|decide)\.outputs\.light_mini_runner$")
# nightly.yml's first job (decide) has no job before it to place it: attempt 1 of a nightly-next push asks for
# glaeda's nightly slot, which only nightly jobs take, and the owned-pool rescue moves it to Blacksmith when no
</file context>
| or runs_on.endswith("|| 'blacksmith-32vcpu-ubuntu-2404' }}") and "LINUX_RUNNER" not in runs_on, runs_on) | ||
| self.assertNotRegex(runs_on, r"vars\.LINUX_RUNNER \|\|") | ||
| self.assertNotIn("macos", runs_on) | ||
| # decide's light-job placement (a mini idle when decide finished) is the one fleet route ahead of |
There was a problem hiding this comment.
P3: The docstring and test name still say these light jobs stay on Linux, but the new mini branch lets two of them run on a Mac mini first. Update the docstring and rename the test to match the placement exception.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_nightly_off_blacksmith.py, line 226:
<comment>The docstring and test name still say these light jobs stay on Linux, but the new mini branch lets two of them run on a Mac mini first. Update the docstring and rename the test to match the placement exception.</comment>
<file context>
@@ -223,8 +223,14 @@ def test_light_nightly_jobs_stay_on_linux_while_the_overflow_switch_is_on(self)
or runs_on.endswith("|| 'blacksmith-32vcpu-ubuntu-2404' }}") and "LINUX_RUNNER" not in runs_on, runs_on)
self.assertNotRegex(runs_on, r"vars\.LINUX_RUNNER \|\|")
self.assertNotIn("macos", runs_on)
+ # decide's light-job placement (a mini idle when decide finished) is the one fleet route ahead of
+ # the Linux set; with no placement (needs read as '') the job takes the set as before.
+ if "needs.decide.outputs.light_mini_runner" in runs_on:
</file context>
GitHub Actions
Inactive
There was a problem hiding this comment.
1 issue found across 6 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test_nightly_trusted_routing.py">
<violation number="1" location="tests/test_nightly_trusted_routing.py:486">
P3: This class copies `LightJobsOnAFreeMini` almost verbatim: `runs_on`, the placement test, and the no-placement/stale/fork test differ only in the `needs.<job>` key. Move the shared helpers and the three test methods into a base mixin or parameterized base class, and keep only the decide-specific test in the new class.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| self.assertTrue(all(value == 'read' for key, value in mint['with'].items() if key.startswith('permission-'))) | ||
|
|
||
|
|
||
| class DecideLightJobsOnAFreeMini(unittest.TestCase): |
There was a problem hiding this comment.
P3: This class copies LightJobsOnAFreeMini almost verbatim: runs_on, the placement test, and the no-placement/stale/fork test differ only in the needs.<job> key. Move the shared helpers and the three test methods into a base mixin or parameterized base class, and keep only the decide-specific test in the new class.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_trusted_routing.py, line 486:
<comment>This class copies `LightJobsOnAFreeMini` almost verbatim: `runs_on`, the placement test, and the no-placement/stale/fork test differ only in the `needs.<job>` key. Move the shared helpers and the three test methods into a base mixin or parameterized base class, and keep only the decide-specific test in the new class.</comment>
<file context>
@@ -482,6 +483,81 @@ def test_the_build_job_mints_only_the_routing_token_for_the_placement(self):
self.assertTrue(all(value == 'read' for key, value in mint['with'].items() if key.startswith('permission-')))
+class DecideLightJobsOnAFreeMini(unittest.TestCase):
+ """decide's light dependents take a mini idle when decide finished, else Blacksmith.
+
</file context>
|
Review (codex gpt-6.1-sol xhigh, router-review): SHIP at d32c268 No correctness findings. Focused routing and placement tests pass. Also: Review (subagent opus-5.5 medium): SHIP at a581f92. Follow-ups from it: check that the minis' python3 is 3.11 or newer (chief-input-key's tomllib; it is continue-on-error), and resolve-nightly-cmux-tui-client can hold a side slot for up to its 75-minute wait. |
…y-fleet-runs-on # Conflicts: # tests/test_ci_nightly_off_blacksmith.py
GitHub Actions
Inactive
|
Review (codex gpt-6-astra high, router-review): SHIP at 70711df No correctness findings. Relevant routing, placement, workflow, resolver and recovery tests pass. |
GitHub Actions
Inactive
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
|
|
Closing as superseded by #29864 ("ci(nightly): place early controls on fleet before overflow"), which landed on feat-cmux-next while this waited for the merge. It gives |
|
Conflict wave: not resolved, because feat-cmux-next already has this change. #29864 (b85875a "ci(nightly): place early controls on fleet before overflow") gives |
HOLD: superseded by #29864; do not land.
Leo, 2026-10-31: "FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT"
What
resolve-nightly-cmux-tui-clientandfind-nightly-next-recoveryneed onlydecide, hold no environment and no secret, and run git, gh, curl, bash and stdlib Python. On every nightly-next run they tookblacksmith-32vcpu-ubuntu-2404with no fleet route (run 41254086065: both started on Blacksmith 32 vCPU at 23:53Z, right afterdecidefinished on an owned mini).decide(which on attempt 1 already runs on an owned mini) now ends with the same two stepsbuild-nightly-appuses for its light jobs (#29436): mint the read-only routing token, thenscripts/ci/portable_job_placement.pywithNIGHTLY=1. Each of the two jobs takes a mini that is idle at that moment, or its Blacksmith Linux runner-set member right away. Both steps arecontinue-on-errorand run after the decision, so a failed placement leaves the old route in place and never failsdecide. A re-run of failed jobs keeps the Blacksmith route (light_attemptmust equal the run attempt).tests/test_ci_nightly_off_blacksmith.py: both jobs leaveBLACKSMITH_LINUX_ONLY_DEBT, which goes from 32 to 30 jobs. The light-jobs-on-Linux test accepts only decide's gated placement term ahead of the Linux set and still refuses any macOS label.tests/test_nightly_trusted_routing.py: newDecideLightJobsOnAFreeMinicovers routing, a stale or missing placement, re-runs, forks, no environment or secret on the placed jobs, and the token's read-only permissions..github/runners.json: both jobs areportable-checks.scripts/ci/runner_routing_table.pyaccepts decide's placement output as the nightly mini term.Audit: NEXT nightly runs-on, feat-cmux-next at 07efced
decide(control runner or nightly slot, then Blacksmith Linux on the rescue re-run),build-nightly-appandbuild-nightly-ghostty-cli-helper(picker owned, then Blacksmith),build-sign-notarize-nightly,verify-*,signer-dmg-image(sign picker),sign-nightly-identity-legacy,verify-nightly-identity-legacyandgenerate-nightly-deltas(owned slot or side lane only, no Blacksmith).build-nightly-remote-daemonandawait-nightly-tree-marker(ci(nightly-next): the nightly's light Linux-hop jobs take a free mini, Blacksmith last #29436). This PR addsresolve-nightly-cmux-tui-clientandfind-nightly-next-recovery.build-nightly-inputs,resolve-nightly-ghostty-cli-helper, both prepare jobs, thesigner-*Linux jobs,attest-nightly-remote-daemon, the visual-gate router, verdict and report, every publish job, release notes, the reel, and the failure reporting. Most of them hold an environment secret (publish, signing, routing) or produce Linux-only output (signer-buildbuilds a Linuxrcodesign), so each needs its own design and is not a mechanical move. fix(nightly): keep DMG signer on trusted Linux route #29821 just moved the DMG signer back to Linux for this reason.visual-gate-nightlyruns only whenNIGHTLY_NEXT_VISUAL_GATEis set (ci(nightly): nightly-next runs stop skipping their own builds; no 32 vCPU pins; display visual gate opt-in #28961). The launch smoke is a separate post-publish workflow.Prior art: #29436 and #29650 (same pattern, merged), #29821 (open, DMG signer, separate jobs). #28967's failures are already gone on base; the current
test_ci_runs_on_fallbackred issigner-build-macos, which #29821 removes.Tests
Run locally:
python3 -m unittest tests/test_ci_nightly_off_blacksmith.py tests/test_nightly_trusted_routing.py tests/test_ci_portable_job_placement.py tests/test_nightly_workflow_size.py tests/test_ci_runs_on_values.py tests/test_publish_app_token_workflow.py tests/test_ci_warm_sources.py tests/test_ci_linux_single_size.py tests/test_ci_signing_owned_mac.py, all OK.tests/test_runner_routing_table.py,tests/test_ci_runs_on_fallback.pyandtests/test_ci_macos_xcode_selection.pyfail the same way on base and on this head:signer-build-macosandsigner-dmg, from #29801 and fixed by #29821.tests/test_runner_label_policy.pyfails on base and here onCI_CONTROL_RUNNER(base red).nightly.yml is now 501,561 bytes, under the 508,000 headroom line.
Risk: both jobs can now run on an owned mini (macOS). Their scripts already handle macOS (
pin-cmux-tui.shusesshasumthere, for example), andtest_ci_macos_xcode_selectionalready listsresolve-nightly-cmux-tui-clientas a git, gh and Python control job.Agent session: agent session
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Routes
resolve-nightly-cmux-tui-clientandfind-nightly-next-recoveryto a free owned mini first, keeping Blacksmith as fallback, so nightly-next jobs no longer rely on Blacksmith by default.decidemints a read-only routing token and runsportable_job_placement.py(NIGHTLY=1) as its last steps, the same patternbuild-nightly-appuses for its light jobs.decidefinishes, or its Blacksmith runner right away; failed or re-run jobs keep the old Blacksmith route.continue-on-errorand run afterdecide's decision, so a failed placement never failsdecide;decide's token hasactions: readso the placement can read the queue janitor's snapshot, and both steps are registered innightly-step-coverage.json.runners.json,runner_routing_table.py, and the placement/routing tests; Blacksmith-only debt drops from 32 to 30 jobs.resolve-nightly-cmux-tui-clientandfind-nightly-next-recoverycontinue to hold no environment and no secret.Written for commit d32c268. Summary will update on new commits.