Skip to content

ci(nightly): prefer dedicated M4 Pro fleet for early jobs and builds - #30249

Merged
teamleaderleo merged 7 commits into
feat-cmux-nextfrom
lane/nightly-dedicated-mini-routing
Nov 1, 2026
Merged

ci(nightly): prefer dedicated M4 Pro fleet for early jobs and builds#30249
teamleaderleo merged 7 commits into
feat-cmux-nextfrom
lane/nightly-dedicated-mini-routing

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Early NEXT nightly jobs can leave owned machines unused when the routing App cannot read capacity. Prefer the explicitly reserved all-nightly M4 Pro label on the first attempt for bootstrap, the four portable controls, and the build picker. The app and helper also use that reservation without waiting for quota-limited capacity reads. The exact repository variable enables the route only after managed runners are registered; unset or invalid values preserve the existing routes.

Leo: “ok just dedicate at least one m4 pro mini just completely to the nightly situation idgaf anymore”. Earlier: “FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT”.

The shared label does not name a host or assume one runner. It is the explicit reservation exception; retries retain the existing fallback. The policy accepts only the exact repository, NEXT track, first attempt and declared label, and CI tests exercise missing, stale and foreign inputs. Existing signing environments, credentials and permissions stay intact. The signer compiler retains Linux so its artifact stays ELF. This change adds no cancellation or timer.

Prior art: preserves #30085's Linux admission fallback and #30173's declared light overflow when the reservation is disabled. #30221's dependency receipts and #30201's cache fallback are separate; #30200 owns advisory publication. The reservation reuses existing persistent build caches and source-keyed web bundles; it adds no compile. External shared-cache configuration remains with fleet enrollment.

Validation: python3 tests/test_nightly_dedicated_mini.py </dev/null reproduces the missing priority on base and passes all seven tests after the fix. Picker, admission, trusted/fork routing, label policy, hosted-Mac, nightly routing and central table suites pass (242 focused Python tests total). The live-tree routing-table check, git diff --check and pre-push static checks pass.

Existing base warning: the fleet-label shell guard reports the same eight expressions on base f3bfbe9fca25 and this head; normalized findings are identical. The hosted-runner shell guard passes.

Observed / Not run

Observed: Python policy and picker tests pass, including quota-independent app/helper selection and preserving retry/other-track routes. Not run: a nightly on the new reservation. Activation remains disabled until registration and admission are verified by fleet enrollment; this PR does not claim end-to-end publication proof.

Performance Before After Delta Evidence
Early fleet-admission wait with the reservation enabled Up to 180 seconds Skipped Removes up to 180 seconds Workflow condition regression
Blacksmith minutes/day Not measured Not measured Portable first attempts prefer the dedicated fleet Policy tests; live measurement pending activation

Changelog

  • Prefer the dedicated nightly fleet for early NEXT jobs and app/helper builds.

Agent session: agent session
🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review dd76092: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30249 -f ref=lane/nightly-dedicated-mini-routing -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 5bdd4c9; does not block

Shipped in:

@teamleaderleo

teamleaderleo commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

This was referenced Nov 1, 2026
Closed
Merged
@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

teamleaderleo commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator Author

Review (subagent): SHIP at dd76092

Review (codex gpt-6-astra high): SHIP at dd76092

Independent re-review after fixing the helper workflow selector: the exact reservation branch now precedes the legacy builder exclusion and Linux reuse/resume branches. Actual picker outputs route both app and helper correctly. Retry and other-track fallbacks remain unchanged. No blocking findings. Focused dedicated routing, central policy and existing nightly routing tests pass; the fleet shell guard has the same eight findings as base, with no new violation.

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
@teamleaderleo
teamleaderleo merged commit d8f537d into feat-cmux-next Nov 1, 2026
88 of 93 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-dedicated-mini-routing branch November 1, 2026 06:04
This was referenced Nov 1, 2026
Merged
Merged
@lawrencecchen lawrencecchen mentioned this pull request Nov 1, 2026
Merged
@lawrencecchen

lawrencecchen commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Base red from this PR (seen on every feat-cmux-next PR in nightly-guards / workflow-guard-tests / release-notary). Two tests fail at the tip:

  1. bash ./tests/test_nightly_universal_build.sh: "FAIL: nightly must build and hand off the macOS 26 helper and Xcode 26.5 app before publishing". The test pins the full runs-on expression of build-nightly-ghostty-cli-helper, and this PR put the CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' route at its front.
  2. python3 tests/test_ci_warm_sources.py: 10 nightly.yml jobs now route to a Mac, and .github/warm-sources.json has no entry for them: close-nightly-failure-issue, decide, find-nightly-next-recovery, nightly-identity-legacy-published, pick-nightly-build-runner, report-nightly-failure, report-visual-gate-nightly, resolve-nightly-cmux-tui-client, resolve-nightly-ghostty-cli-helper and visual-gate-nightly-verdict. The same test also reports cmux-tui-spec.yml:clippy.

I did not fix these in #30624, because each one needs a decision from the owner:

  • Item 1: is the dedicated-mini route the intended first route for the helper? If so, the test's pinned expression must change to include it.
  • Item 2: give each job a class and an exempt reason. Most of these jobs look like controls that compile nothing, but the owner must confirm this.

#30624 fixes the other base reds.

@lawrencecchen lawrencecchen mentioned this pull request Nov 1, 2026
Merged
@lawrencecchen

lawrencecchen commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Follow-up to my comment above. The lead decided (12:06Z) not to wait. #30824 fixes both reds:

lawrencecchen added a commit that referenced this pull request Nov 1, 2026
@lawrencecchen
base red: release-notary green (helper admission gate, nightly.yml si…
53dd0c6 
…ze, universal build pin, warm sources, promotion admission, highlight) (#30824)

* base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini

#30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly
mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them
as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other
nightly.yml Mac jobs that compile nothing, with an exemption that says what the
job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache)
takes linux-rust with actions-cache, as its cmux-tui-spec peers do.

Red at 9288e32: python3 tests/test_ci_warm_sources.py   FAILED (11 unlisted jobs)
Green here: same command, OK.

* base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate

The test pins the full runs-on and if: of the nightly helper jobs. Two
merged changes moved them:
- #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER)
  first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression
  now has that route; the merged workflow is the source of truth.
- #30284 (4e81bfe) added the early admission conjunct to the resolver and
  the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the
  resolver and the helper carry it as one conjunct after their gate, and that
  build-nightly-app keeps its own gate (it is skipped with the resolver it needs).
  The build_only exclusion is checked the same way as before.

Red at 9288e32: bash ./tests/test_nightly_universal_build.sh   FAIL: nightly must build and hand off the macOS 26 helper ...
Green with the nightly.yml commit after this one: same command, PASS.

* base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom

tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper
and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe)
added the early admission conjunct to the lookup on purpose; the helper build
now has it too, so it never builds while its lookup waits for admission.

tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the
508,000 headroom. The cache save steps repeated the same three R2 env values
six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT);
each is now an anchor on its first use and an alias on the others, as
&pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the
same workflow as before except the helper's if:. nightly.yml is 507,061 bytes.
No step is removed and no limit changes. (Job-level anchors are refused by
test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide
text that test_nightly_notices_submodule.py and
test_nightly_promotion_admission.py count, so the anchors are on step env.)

Red at 9288e32:
  python3 tests/test_nightly_ghostty_helper_cache.py   FAILED (failures=1)
  python3 tests/test_nightly_workflow_size.py          508735 not less than 508000
Green here: both OK. All 123 tests that read nightly.yml: the same results as
at 9288e32 except these and test_nightly_universal_build.sh, now green.

* base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job

#30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard
from main" with promote-nightly-next-acked through the YAML anchor
&legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in
the text and test_both_promote_decisions_pass_the_train_verifier (it counted
the text) failed. It now finds the two jobs that call guard.trainVerifier in
the parsed workflow and requires each to check out the verifier; it fails when
either job's checkout drops it (checked by removing it from the anchor).

Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py   AssertionError
Green here: PASS (75 cases).

* base red: release-notary, the Cloud environment editor highlight links its pull request

#30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md
with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed.
It links #30357, as #30624 did for three others.

Red at 265c68a: python3 scripts/whats-new/test_digest.py   FAILED (failures=1)
Green here: OK.
@lawrencecchen lawrencecchen mentioned this pull request Nov 1, 2026
Merged
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
@lawrencecchen @cmux-bot
base red: nightly.yml under its 490,000-byte budget (30 run blocks mo…
fa65549 
…ve to scripts/ci/nightly-steps/) (#30912)

* base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini

#30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly
mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them
as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other
nightly.yml Mac jobs that compile nothing, with an exemption that says what the
job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache)
takes linux-rust with actions-cache, as its cmux-tui-spec peers do.

Red at 9288e32: python3 tests/test_ci_warm_sources.py   FAILED (11 unlisted jobs)
Green here: same command, OK.

* base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate

The test pins the full runs-on and if: of the nightly helper jobs. Two
merged changes moved them:
- #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER)
  first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression
  now has that route; the merged workflow is the source of truth.
- #30284 (4e81bfe) added the early admission conjunct to the resolver and
  the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the
  resolver and the helper carry it as one conjunct after their gate, and that
  build-nightly-app keeps its own gate (it is skipped with the resolver it needs).
  The build_only exclusion is checked the same way as before.

Red at 9288e32: bash ./tests/test_nightly_universal_build.sh   FAIL: nightly must build and hand off the macOS 26 helper ...
Green with the nightly.yml commit after this one: same command, PASS.

* base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom

tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper
and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe)
added the early admission conjunct to the lookup on purpose; the helper build
now has it too, so it never builds while its lookup waits for admission.

tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the
508,000 headroom. The cache save steps repeated the same three R2 env values
six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT);
each is now an anchor on its first use and an alias on the others, as
&pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the
same workflow as before except the helper's if:. nightly.yml is 507,061 bytes.
No step is removed and no limit changes. (Job-level anchors are refused by
test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide
text that test_nightly_notices_submodule.py and
test_nightly_promotion_admission.py count, so the anchors are on step env.)

Red at 9288e32:
  python3 tests/test_nightly_ghostty_helper_cache.py   FAILED (failures=1)
  python3 tests/test_nightly_workflow_size.py          508735 not less than 508000
Green here: both OK. All 123 tests that read nightly.yml: the same results as
at 9288e32 except these and test_nightly_universal_build.sh, now green.

* base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job

#30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard
from main" with promote-nightly-next-acked through the YAML anchor
&legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in
the text and test_both_promote_decisions_pass_the_train_verifier (it counted
the text) failed. It now finds the two jobs that call guard.trainVerifier in
the parsed workflow and requires each to check out the verifier; it fails when
either job's checkout drops it (checked by removing it from the anchor).

Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py   AssertionError
Green here: PASS (75 cases).

* base red: release-notary, the Cloud environment editor highlight links its pull request

#30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md
with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed.
It links #30357, as #30624 did for three others.

Red at 265c68a: python3 scripts/whats-new/test_digest.py   FAILED (failures=1)
Green here: OK.

* base red: preflight, nightly.yml back under its 490,000-byte budget: 30 run blocks move to scripts/ci/nightly-steps/

tests/test_workflow_file_size.py (ci-guards preflight) caps nightly.yml at 490,000 bytes;
it was 505,604 after #30824 (508,735 before it), red on every pull request. Lead
decision 17:37Z: move run blocks, as #25879 did.

30 `run: |` blocks move verbatim into scripts/ci/nightly-steps/<name>.sh; each step now
runs `bash scripts/ci/nightly-steps/<name>.sh` with the same env:, if:, id:, shell and
working directory. yaml.safe_load gives the same jobs and steps as before except those
30 run strings, and tests/nightly_workflow_text.py puts the scripts back inline
byte-identical to the old workflow. nightly.yml goes from 505,604 to 486,014 bytes.

Which blocks:
- Every block uses no ${{ }} expression and starts with `set -euo pipefail`, so the
  script sets the same shell options the inline block did (bash scripts/x.sh does not
  inherit GitHub's `bash -e`).
- No block has a working-directory, runs inside a checkout it performs, or is a
  security gate whose text a test pins ("Hold the whole Mac before any credential").
- Each step's script is in the tree the step runs on: steps in refresh-compilation-cache,
  refresh-test-compilation-cache and resolve-nightly-cmux-tui-client check out the
  workflow commit (decide's head_sha is context.sha); steps in build-nightly-app,
  build-nightly-ghostty-cli-helper and build-nightly-remote-daemon check out build_sha
  and are skipped by an unsigned resume, the run that builds an older saved tree.

Tests that read the raw nightly.yml text for a moved block now read it through
tests/nightly_workflow_text.py (expand), as #25879 set up; what they check is unchanged.
The new scripts pass shellcheck -S warning (the level actionlint gives shellcheck), and
actionlint reports the same 5 findings on nightly.yml as before (unused anchors).

Red at 6238504 (#30824 head): python3 tests/test_workflow_file_size.py   505604 not less than 490000
Green here: OK. All 124 tests that read nightly.yml: the same results as at 6238504
except test_workflow_file_size.py, now green.

---------

Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
@lawrencecchen lawrencecchen mentioned this pull request Nov 1, 2026
Merged
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
@lawrencecchen @cmux-bot
ci(nightly): headroom under the size budget (14 more run blocks, step…
3c0c02a 
… notes into scripts, expressions through env) (#31013)

* base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini

#30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly
mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them
as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other
nightly.yml Mac jobs that compile nothing, with an exemption that says what the
job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache)
takes linux-rust with actions-cache, as its cmux-tui-spec peers do.

Red at 9288e32: python3 tests/test_ci_warm_sources.py   FAILED (11 unlisted jobs)
Green here: same command, OK.

* base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate

The test pins the full runs-on and if: of the nightly helper jobs. Two
merged changes moved them:
- #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER)
  first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression
  now has that route; the merged workflow is the source of truth.
- #30284 (4e81bfe) added the early admission conjunct to the resolver and
  the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the
  resolver and the helper carry it as one conjunct after their gate, and that
  build-nightly-app keeps its own gate (it is skipped with the resolver it needs).
  The build_only exclusion is checked the same way as before.

Red at 9288e32: bash ./tests/test_nightly_universal_build.sh   FAIL: nightly must build and hand off the macOS 26 helper ...
Green with the nightly.yml commit after this one: same command, PASS.

* base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom

tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper
and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe)
added the early admission conjunct to the lookup on purpose; the helper build
now has it too, so it never builds while its lookup waits for admission.

tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the
508,000 headroom. The cache save steps repeated the same three R2 env values
six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT);
each is now an anchor on its first use and an alias on the others, as
&pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the
same workflow as before except the helper's if:. nightly.yml is 507,061 bytes.
No step is removed and no limit changes. (Job-level anchors are refused by
test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide
text that test_nightly_notices_submodule.py and
test_nightly_promotion_admission.py count, so the anchors are on step env.)

Red at 9288e32:
  python3 tests/test_nightly_ghostty_helper_cache.py   FAILED (failures=1)
  python3 tests/test_nightly_workflow_size.py          508735 not less than 508000
Green here: both OK. All 123 tests that read nightly.yml: the same results as
at 9288e32 except these and test_nightly_universal_build.sh, now green.

* base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job

#30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard
from main" with promote-nightly-next-acked through the YAML anchor
&legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in
the text and test_both_promote_decisions_pass_the_train_verifier (it counted
the text) failed. It now finds the two jobs that call guard.trainVerifier in
the parsed workflow and requires each to check out the verifier; it fails when
either job's checkout drops it (checked by removing it from the anchor).

Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py   AssertionError
Green here: PASS (75 cases).

* base red: release-notary, the Cloud environment editor highlight links its pull request

#30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md
with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed.
It links #30357, as #30624 did for three others.

Red at 265c68a: python3 scripts/whats-new/test_digest.py   FAILED (failures=1)
Green here: OK.

* base red: preflight, nightly.yml back under its 490,000-byte budget: 30 run blocks move to scripts/ci/nightly-steps/

tests/test_workflow_file_size.py (ci-guards preflight) caps nightly.yml at 490,000 bytes;
it was 505,604 after #30824 (508,735 before it), red on every pull request. Lead
decision 17:37Z: move run blocks, as #25879 did.

30 `run: |` blocks move verbatim into scripts/ci/nightly-steps/<name>.sh; each step now
runs `bash scripts/ci/nightly-steps/<name>.sh` with the same env:, if:, id:, shell and
working directory. yaml.safe_load gives the same jobs and steps as before except those
30 run strings, and tests/nightly_workflow_text.py puts the scripts back inline
byte-identical to the old workflow. nightly.yml goes from 505,604 to 486,014 bytes.

Which blocks:
- Every block uses no ${{ }} expression and starts with `set -euo pipefail`, so the
  script sets the same shell options the inline block did (bash scripts/x.sh does not
  inherit GitHub's `bash -e`).
- No block has a working-directory, runs inside a checkout it performs, or is a
  security gate whose text a test pins ("Hold the whole Mac before any credential").
- Each step's script is in the tree the step runs on: steps in refresh-compilation-cache,
  refresh-test-compilation-cache and resolve-nightly-cmux-tui-client check out the
  workflow commit (decide's head_sha is context.sha); steps in build-nightly-app,
  build-nightly-ghostty-cli-helper and build-nightly-remote-daemon check out build_sha
  and are skipped by an unsigned resume, the run that builds an older saved tree.

Tests that read the raw nightly.yml text for a moved block now read it through
tests/nightly_workflow_text.py (expand), as #25879 set up; what they check is unchanged.
The new scripts pass shellcheck -S warning (the level actionlint gives shellcheck), and
actionlint reports the same 5 findings on nightly.yml as before (unused anchors).

Red at 6238504 (#30824 head): python3 tests/test_workflow_file_size.py   505604 not less than 490000
Green here: OK. All 124 tests that read nightly.yml: the same results as at 6238504
except test_workflow_file_size.py, now green.

* ci(nightly): 14 more run blocks and the step notes move to scripts/ci/nightly-steps/; expressions reach scripts through env (headroom under the size budget)

After #30912 nightly.yml had 3,986 bytes left under tests/test_workflow_file_size.py's
490,000-byte cap, so the next nightly.yml change would turn preflight red on every
pull request again. Lead decision 13:2xZ: (a) the remaining safe blocks, (b) step
notes into their script's header, (c) the `${{ }}` blocks with each expression
passed through env. nightly.yml goes from 486,014 to 457,877 bytes (32,123 bytes of room).

(a) 7 blocks with no ${{ }} move verbatim as in #30912. Four of them (Fetch the
Chromium engine, Upload dSYMs to Sentry, Finish the dSYM upload to Sentry, Report a
missing daemon attestation) do not start with `set -euo pipefail`; they ran under
GitHub's default `bash -e {0}`, so they are called with `bash -e`, and
tests/nightly_workflow_text.py accepts that call form.

(c) 7 blocks used ${{ }} expressions. Each expression now sits in a new `env:` entry of
the same step and the script reads that variable in the same place, GitHub's pattern
against script injection. Each name is new to the workflow, read only by its script,
and never assigned there. `--allow-non-descendant`, which was spliced in unquoted
when non-empty and vanished when empty, is `${ALLOW_NON_DESCENDANT_FLAG:+"$ALLOW_NON_DESCENDANT_FLAG"}`:
one word or none, as before. Republish deltas and revised appcasts stays inline:
tests/test_nightly_republish_delta_guard.py runs its exact text, and its job is a
resume job.

(b) The comment lines above and inside 46 script-backed steps (only those whose script
is called once) move into the header of their script, under "The step's notes,
moved from nightly.yml". Comments of steps that keep inline code stay. yaml.safe_load
of nightly.yml is unchanged by this part.

Not moved: the run blocks of the 7 jobs an unsigned resume runs over an older
build_sha (build-sign-notarize-nightly, prepare-nightly-variant, prepare-nightly-variant-primary,
verify-signed-nightly, sign-release-notes, published-in, publish-nightly-reel). A resume
of a source compiled before this change would not find a new script. They move 8 days
after #30912 merged, when no unsigned copy of a pre-merge commit is left.

Tests: 8 tests that read the inline text of a moved block read the expanded view
(tests/nightly_workflow_text.py) or copy the script into their scratch tree;
test_nightly_trusted_routing and test_ci_fleet_compilation_cache check the expression
in the step's env and the variable in its script. What they check is unchanged.

Red at the size budget: none at #30912's head (486,014 < 490,000); the gain is headroom.
All 124 tests that read nightly.yml: the same results as at 17bf155 (all green).
shellcheck -S warning: clean on scripts/ci/nightly-steps/*.sh. actionlint 1.7.12-manaflow.1:
the same 5 findings on nightly.yml as before (unused anchors).

* ci(nightly): Generate deltas and revised appcast stays inline (its .publish-tooling checkout lists single files)

Review of #31013 at 8a2494b: generate-nightly-deltas checks out .publish-tooling
with a non-cone sparse list of five named files (anchor &checkout-the-gh-token-wrapper),
so scripts/ci/nightly-steps/deltas-generate-appcast.sh would not be in it and the step
would fail on every run, hidden by the job's continue-on-error. My selection matched
the directory name as a substring of the list. The step goes back exactly as on the
feat tip (block and its notes), the script is removed, and test_nightly_sparkle_key.sh
is back to the feat tip's text.

An exact check of all 140 step-script calls (the last checkout before each call that
provides its path, root or .publish-tooling, must contain the file: no sparse set,
a cone-mode directory or a non-cone pattern) finds no gap at this head; it found
exactly this one before the fix, and none in the feat tip's 127 calls.

nightly.yml: 458,313 bytes (31,687 under 490,000). All 124 tests that read nightly.yml:
the same results as at the feat tip 49aa291. shellcheck -S warning clean; actionlint:
the same 5 findings as before.

---------

Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
@teamleaderleo teamleaderleo mentioned this pull request Nov 2, 2026
Closed

This branch was previously deployed

1 inactive deployment
ci-routing-agent — dd760928 Deployed Nov 1, 2026 by teamleaderleo via wait for same-key tree publisher #94086
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lawrencecchen lawrencecchen Awaiting requested review from lawrencecchen lawrencecchen is a code owner

Assignees

No one assigned

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@teamleaderleo @lawrencecchen