ci(nightly): headroom under the size budget (14 more run blocks, step notes into scripts, expressions through env) by lawrencecchen · Pull Request #31013 · manaflow-ai/cmux

lawrencecchen · 2026-11-01T14:16:29Z

After #30912, nightly.yml had only 3,986 bytes left under the 490,000-byte cap of tests/test_workflow_file_size.py (ci-guards preflight). The next nightly.yml change would turn preflight red on every PR again. Lead decision (13:2xZ): this PR does three things, (a), (b) and (c) below. nightly.yml goes from 486,014 to 458,313 bytes, which leaves 31,687 bytes of room.

Nightly owners (#30284, #30249, #30471, #30278, #30113):

13 more steps now call scripts/ci/nightly-steps/<name>.sh.
The notes that described a script-backed step now sit in that script's header, under "The step's notes, moved from nightly.yml".
Seven steps now pass their ${{ }} values to the script through env: (table below).
tests/nightly_workflow_text.py still shows the workflow with every script inline.

(a) The remaining safe blocks (6)

These move word for word, by #30912's rules:

no ${{ }};
no shell: and no working directory;
the script exists in the tree the step runs on;
each build_sha step is skipped by an unsigned resume.

Four of them do not start with set -euo pipefail: "Fetch the Chromium engine", "Upload dSYMs to Sentry", "Finish the dSYM upload to Sentry" and "Report a missing daemon attestation". They ran under GitHub's default bash -e {0}, so they are now called with bash -e scripts/..., and the expansion helper accepts that call form.

"Generate deltas and revised appcast" (generate-nightly-deltas) stays inline. The first review found that its .publish-tooling checkout is a non-cone sparse checkout that lists single files, so a moved script would not be there. The step would fail on every run, and continue-on-error would hide it. My own selection matched the directory name as a substring and missed this. That job is now byte-identical to the feat tip.

Every step script must exist in the checkout its step runs from. I checked this for all 140 step-script calls in nightly.yml: for each call, the last checkout before it that provides the path (root, or path: .publish-tooling) must contain the file, either with no sparse set, a cone-mode directory or a non-cone pattern. The head has no gap (the feat tip has none in 127 calls either). Before the fix, the same check reported exactly this one step.

(c) Blocks with `${{ }}`: each expression is passed through `env:` (7 blocks, 11 expressions)

GitHub recommends this pattern against script injection. Each expression moves to a new env: entry of the same step, and the script reads that variable in the same place.

Job :: step	Expression	Env name	Script	Reads	Writes	Other scripts	Uses in nightly.yml
build-nightly-app :: Start Release DerivedData empty	`${{ needs.decide.outputs.track }}`	`DECIDE_TRACK`	`app-start-release-derived-data-empty.sh`	1	0	0	1
build-nightly-app :: Start Release DerivedData empty	`${{ needs.decide.outputs.channel }}`	`DECIDE_CHANNEL`	`app-start-release-derived-data-empty.sh`	2	0	0	1
build-nightly-app :: Compute Xcode compilation cache key	`${{ needs.decide.outputs.track }}`	`CACHE_KEY_TRACK`	`app-compilation-cache-key.sh`	1	0	0	1
build-nightly-app :: Build nightly app (Release)	`${{ needs.decide.outputs.cold_cache == 'true' && '1'		'' }}`	`DECIDE_COLD_CACHE_DISABLE`	`app-build-release.sh`	1	0
build-nightly-app :: Assemble the Computer Use helper	`${{ contains(needs.decide.outputs.variants, 'x86_64') && 'arm64 x86_64'		'arm64' }}`	`HELPER_ARCHS`	`app-assemble-computer-use-helper.sh`	1	0
build-nightly-app :: Resolve immutable SSH daemon metadata	`${{ needs.resolve-nightly-cmux-tui-client.outputs.daemon_build }}`	`RESOLVED_DAEMON_BUILD`	`app-resolve-daemon-metadata.sh`	1	0	0	1
build-nightly-app :: Resolve immutable SSH daemon metadata	`${{ needs.decide.outputs.track }}`	`DAEMON_DECIDE_TRACK`	`app-resolve-daemon-metadata.sh`	1	0	0	1
build-nightly-app :: Resolve immutable SSH daemon metadata	`${{ needs.decide.outputs.should_publish }}`	`DECIDE_SHOULD_PUBLISH`	`app-resolve-daemon-metadata.sh`	1	0	0	1
publish-nightly :: Move channel release tag to built commit	`${{ steps.source.outputs.build_sha }}`	`SOURCE_BUILD_SHA`	`publish-move-release-tag.sh`	1	0	0	1
publish-nightly :: Move channel release tag to built commit	`${{ needs.decide.outputs.release_tag != 'nightly' && '--allow-non-descendant'		'' }}`	`ALLOW_NON_DESCENDANT_FLAG`	`publish-move-release-tag.sh`	2	0
verify-nightly-identity-legacy :: Hand the tested app's Info.plist to publish	`${{ steps.verify.outputs.tested_sha256 }}`	`VERIFIED_DMG_SHA256`	`legacy-identity-verify-hand-info-plist.sh`	1	0	0	1

What the table proves:

Reads: the script reads the variable that often. That is the same number of places as the old block had ${{ }}.
Writes = 0: the script never assigns the variable.
Other scripts = 0: no other step script uses the name.
Uses in nightly.yml = 1: the name appears only on its env: line, so no job or workflow env can shadow it.

For every one of the 13 changed steps, the script body equals the old run block with each ${{ expr }} replaced by ${NAME}. I checked this programmatically against the feat tip. Two cases need a note:

--allow-non-descendant: the old text spliced it in unquoted when set and nothing when empty. It is now ${ALLOW_NON_DESCENDANT_FLAG:+"$ALLOW_NON_DESCENDANT_FLAG"}, which gives the same one word or no word.
DECIDE_COLD_CACHE_DISABLE: it was already inside double quotes, so the replacement keeps the same quoting.

"Republish deltas and revised appcasts" stays inline. test_nightly_republish_delta_guard.py runs its exact text, and its job checks out build_sha.

(b) Step notes move into their script's header (46 steps, 13.6 KB)

Only for steps whose script is called once. Comments of steps that keep inline code stay in nightly.yml. Moving the notes does not change yaml.safe_load of nightly.yml, and the non-comment lines of the 46 scripts are unchanged.

Why the 7 resume jobs are left out, and when they move

These 7 jobs run during an unsigned resume (nightly_next_unsigned_run) over the source run's build_sha: build-sign-notarize-nightly, prepare-nightly-variant, prepare-nightly-variant-primary, verify-signed-nightly, sign-release-notes, published-in and publish-nightly-reel.

resume-unsigned-nightly.py only requires that build_sha is an ancestor of the dispatch commit. So a resume of a source compiled before the scripts exist would not find them. Unsigned apps are kept for 1 day, and resumed copies for 7 days. These blocks therefore move 8 days after #30912 merged (2026-11-01 13:56Z), on or after 2026-11-09 14:00Z. That task is recorded in the lead's STATUS.

Tests

7 tests now read the expanded view, or copy the moved script into their scratch tree: test_nightly_background_steps, test_upload_sentry_dsyms.sh, test_nightly_next_computer_use_helper and test_nightly_sparkle_key.sh.
test_nightly_trusted_routing checks env.RESOLVED_DAEMON_BUILD equals the resolver output, and that the script reads ${RESOLVED_DAEMON_BUILD}.
test_ci_fleet_compilation_cache checks env.DECIDE_COLD_CACHE_DISABLE equals the old expression, and that the command reads it.
The helper's regex accepts bash -e.
What each test checks is unchanged.

No new CI job, step or gate. No limit changes, and the "Hold the whole Mac" gate is untouched. Risk tier: high.

Evidence

python3 tests/test_workflow_file_size.py: green at base (486,014, with 3,986 bytes of room) and green at head (458,313, with 31,687 bytes of room). This PR adds headroom; base red: nightly.yml under its 490,000-byte budget (30 run blocks move to scripts/ci/nightly-steps/) #30912 fixed the red.
All 124 tests that read nightly.yml: the same results at this head as at base.
- At base red: nightly.yml under its 490,000-byte budget (30 run blocks move to scripts/ci/nightly-steps/) #30912's head (17bf1554652): all green on both sides.
- At the current feat tip (49aa2917ea9): the same 9 tests are red on both sides. These are base reds from later merges, and this PR changes none of them: test_check_ghostty_zig_workflows, test_ci_change_areas, test_ci_cloud_probe_rescue, test_ci_macos_xcode_selection, test_ci_self_hosted_guard.sh, test_ci_signing_keychain_is_deleted_before_bundle_code, test_ci_workflow_guards_are_wired, test_cmux_next_visual_gate and test_seed_derived_data.
shellcheck -S warning scripts/ci/nightly-steps/*.sh: clean.
actionlint 1.7.12-manaflow.1: the same 5 findings on nightly.yml as at base, all unused anchors that were already there.

Observed / Not run

Observed: the above, plus prepush-gate.py.
Not run: no nightly run (that needs the nightly-next push path). The fleet compile on this PR is the compile gate.

Changelog

none

#30249 routed to the dedicated mini #30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other nightly.yml Mac jobs that compile nothing, with an exemption that says what the job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache) takes linux-rust with actions-cache, as its cmux-tui-spec peers do. Red at 9288e32: python3 tests/test_ci_warm_sources.py FAILED (11 unlisted jobs) Green here: same command, OK.

…rged helper route and admission gate The test pins the full runs-on and if: of the nightly helper jobs. Two merged changes moved them: - #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER) first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression now has that route; the merged workflow is the source of truth. - #30284 (4e81bfe) added the early admission conjunct to the resolver and the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the resolver and the helper carry it as one conjunct after their gate, and that build-nightly-app keeps its own gate (it is skipped with the resolver it needs). The build_only exclusion is checked the same way as before. Red at 9288e32: bash ./tests/test_nightly_universal_build.sh FAIL: nightly must build and hand off the macOS 26 helper ... Green with the nightly.yml commit after this one: same command, PASS.

… admission gate; nightly.yml back under its headroom tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe) added the early admission conjunct to the lookup on purpose; the helper build now has it too, so it never builds while its lookup waits for admission. tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the 508,000 headroom. The cache save steps repeated the same three R2 env values six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT); each is now an anchor on its first use and an alias on the others, as &pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the same workflow as before except the helper's if:. nightly.yml is 507,061 bytes. No step is removed and no limit changes. (Job-level anchors are refused by test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide text that test_nightly_notices_submodule.py and test_nightly_promotion_admission.py count, so the anchors are on step env.) Red at 9288e32: python3 tests/test_nightly_ghostty_helper_cache.py FAILED (failures=1) python3 tests/test_nightly_workflow_size.py 508735 not less than 508000 Green here: both OK. All 123 tests that read nightly.yml: the same results as at 9288e32 except these and test_nightly_universal_build.sh, now green.

…d-nightly-notary

…ed promote-guard checkout per job #30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard from main" with promote-nightly-next-acked through the YAML anchor &legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in the text and test_both_promote_decisions_pass_the_train_verifier (it counted the text) failed. It now finds the two jobs that call guard.trainVerifier in the parsed workflow and requires each to check out the verifier; it fails when either job's checkout drops it (checked by removing it from the anchor). Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py AssertionError Green here: PASS (75 cases).

…s its pull request #30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed. It links #30357, as #30624 did for three others. Red at 265c68a: python3 scripts/whats-new/test_digest.py FAILED (failures=1) Green here: OK.

…d-nightly-notary

…30 run blocks move to scripts/ci/nightly-steps/ tests/test_workflow_file_size.py (ci-guards preflight) caps nightly.yml at 490,000 bytes; it was 505,604 after #30824 (508,735 before it), red on every pull request. Lead decision 17:37Z: move run blocks, as #25879 did. 30 `run: |` blocks move verbatim into scripts/ci/nightly-steps/<name>.sh; each step now runs `bash scripts/ci/nightly-steps/<name>.sh` with the same env:, if:, id:, shell and working directory. yaml.safe_load gives the same jobs and steps as before except those 30 run strings, and tests/nightly_workflow_text.py puts the scripts back inline byte-identical to the old workflow. nightly.yml goes from 505,604 to 486,014 bytes. Which blocks: - Every block uses no ${{ }} expression and starts with `set -euo pipefail`, so the script sets the same shell options the inline block did (bash scripts/x.sh does not inherit GitHub's `bash -e`). - No block has a working-directory, runs inside a checkout it performs, or is a security gate whose text a test pins ("Hold the whole Mac before any credential"). - Each step's script is in the tree the step runs on: steps in refresh-compilation-cache, refresh-test-compilation-cache and resolve-nightly-cmux-tui-client check out the workflow commit (decide's head_sha is context.sha); steps in build-nightly-app, build-nightly-ghostty-cli-helper and build-nightly-remote-daemon check out build_sha and are skipped by an unsigned resume, the run that builds an older saved tree. Tests that read the raw nightly.yml text for a moved block now read it through tests/nightly_workflow_text.py (expand), as #25879 set up; what they check is unchanged. The new scripts pass shellcheck -S warning (the level actionlint gives shellcheck), and actionlint reports the same 5 findings on nightly.yml as before (unused anchors). Red at 6238504 (#30824 head): python3 tests/test_workflow_file_size.py 505604 not less than 490000 Green here: OK. All 124 tests that read nightly.yml: the same results as at 6238504 except test_workflow_file_size.py, now green.

…d-nightly-size

…/nightly-steps/; expressions reach scripts through env (headroom under the size budget) After #30912 nightly.yml had 3,986 bytes left under tests/test_workflow_file_size.py's 490,000-byte cap, so the next nightly.yml change would turn preflight red on every pull request again. Lead decision 13:2xZ: (a) the remaining safe blocks, (b) step notes into their script's header, (c) the `${{ }}` blocks with each expression passed through env. nightly.yml goes from 486,014 to 457,877 bytes (32,123 bytes of room). (a) 7 blocks with no ${{ }} move verbatim as in #30912. Four of them (Fetch the Chromium engine, Upload dSYMs to Sentry, Finish the dSYM upload to Sentry, Report a missing daemon attestation) do not start with `set -euo pipefail`; they ran under GitHub's default `bash -e {0}`, so they are called with `bash -e`, and tests/nightly_workflow_text.py accepts that call form. (c) 7 blocks used ${{ }} expressions. Each expression now sits in a new `env:` entry of the same step and the script reads that variable in the same place, GitHub's pattern against script injection. Each name is new to the workflow, read only by its script, and never assigned there. `--allow-non-descendant`, which was spliced in unquoted when non-empty and vanished when empty, is `${ALLOW_NON_DESCENDANT_FLAG:+"$ALLOW_NON_DESCENDANT_FLAG"}`: one word or none, as before. Republish deltas and revised appcasts stays inline: tests/test_nightly_republish_delta_guard.py runs its exact text, and its job is a resume job. (b) The comment lines above and inside 46 script-backed steps (only those whose script is called once) move into the header of their script, under "The step's notes, moved from nightly.yml". Comments of steps that keep inline code stay. yaml.safe_load of nightly.yml is unchanged by this part. Not moved: the run blocks of the 7 jobs an unsigned resume runs over an older build_sha (build-sign-notarize-nightly, prepare-nightly-variant, prepare-nightly-variant-primary, verify-signed-nightly, sign-release-notes, published-in, publish-nightly-reel). A resume of a source compiled before this change would not find a new script. They move 8 days after #30912 merged, when no unsigned copy of a pre-merge commit is left. Tests: 8 tests that read the inline text of a moved block read the expanded view (tests/nightly_workflow_text.py) or copy the script into their scratch tree; test_nightly_trusted_routing and test_ci_fleet_compilation_cache check the expression in the step's env and the variable in its script. What they check is unchanged. Red at the size budget: none at #30912's head (486,014 < 490,000); the gain is headroom. All 124 tests that read nightly.yml: the same results as at 17bf155 (all green). shellcheck -S warning: clean on scripts/ci/nightly-steps/*.sh. actionlint 1.7.12-manaflow.1: the same 5 findings on nightly.yml as before (unused anchors).

…d-nightly-size-2 # Conflicts: # scripts/ci/nightly-steps/app-bound-rust-targets.sh # scripts/ci/nightly-steps/app-build-mobile-host.sh # scripts/ci/nightly-steps/app-bundle-chief-and-mobile-host.sh # scripts/ci/nightly-steps/app-bundle-cmux-tui-client.sh # scripts/ci/nightly-steps/app-bundle-fast-cmux-tui-client.sh # scripts/ci/nightly-steps/app-drop-iroh-stub.sh # scripts/ci/nightly-steps/app-fetch-tree-companions.sh # scripts/ci/nightly-steps/helper-cache-key.sh # scripts/ci/nightly-steps/helper-source-archive.sh # scripts/ci/nightly-steps/helper-zig-packages-notices.sh # scripts/ci/nightly-steps/refresh-test-collect-swift-packages.sh # scripts/ci/nightly-steps/resolve-cmux-tui-client-commit.sh

github-actions · 2026-11-01T14:16:44Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 8a2494b: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr31013 -f ref=fix/base-red-nightly-size-2 -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4132492786501 (3c0c02a5ae)
cmux NEXT nightly 1.0.0-nightly.4132569907101 (641e45405d)
cmux NEXT nightly 1.0.0-nightly.4132675672201 (1d9d0c83e6)
cmux NEXT nightly 1.0.0-nightly.4132727143001 (fe90490c5d)

blacksmith-sh · 2026-11-01T14:22:19Z

Found 37 test failures on Blacksmith runners:

Failures

Test	View Logs
`2 runner guard check(s) failed:/2 runner guard check(s) failed:`	View Logs
`2 runner guard check(s) failed:/2 runner guard check(s) failed:`	View Logs
`main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/` `test_download_prints_the_active_profile_only`	View Logs
`nightly.yml job build-sign-notarize-nightly never runs scripts/` `select-ci-xcode.sh, so it builds with the image's default Xcode`	View Logs
`nightly.yml job generate-nightly-deltas never runs scripts/` `select-ci-xcode.sh, so it builds with the image's default Xcode`	View Logs
`nightly.yml job recover-nightly-next-notarization never runs scripts/` `select-ci-xcode.sh, so it builds with the image's default Xcode`	View Logs
`nightly.yml job refresh-test-compilation-cache never runs scripts/` `select-ci-xcode.sh, so it builds with the image's default Xcode`	View Logs
`nightly.yml job signer-dmg-image never runs scripts/` `select-ci-xcode.sh, so it builds with the image's default Xcode`	View Logs
`nightly.yml job verify-final-dmg-nightly never runs scripts/` `select-ci-xcode.sh, so it builds with the image's default Xcode`	View Logs
`nightly.yml job verify-nightly-identity-legacy never runs scripts/` `select-ci-xcode.sh, so it builds with the image's default Xcode`	View Logs

...and 27 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

cubic-dev-ai

11 issues found across 61 files

_{You’re at about 99% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/ci/nightly-steps/github-names-install-still-newest.sh">

<violation number="1" location="scripts/ci/nightly-steps/github-names-install-still-newest.sh:6">
P3: The new header note says `still-newest` exits 0 while no build above this one is on the feed, but the script actually exits 0 only when the feed serves exactly this build. It exits 1 for an older feed build too, so every guarded write is skipped in that case. Reword the note to match the equality check.</violation>
</file>

<file name="scripts/ci/nightly-steps/attest-report-missing.sh">

<violation number="1" location="scripts/ci/nightly-steps/attest-report-missing.sh:3">
P3: This header comment is inaccurate. The workflow step has no `env:` block and the script reads no inputs, so drop the claim or describe what the script actually needs.</violation>
</file>

<file name="scripts/ci/nightly-steps/legacy-identity-verify-hand-info-plist.sh">

<violation number="1" location="scripts/ci/nightly-steps/legacy-identity-verify-hand-info-plist.sh:16">
P3: If `cp` or `mkdir` fails after the attach, the script exits without detaching the image. Register an EXIT trap right after the attach, as the sibling DMG scripts do, so the mount is always released.</violation>
</file>

<file name="scripts/ci/nightly-steps/app-finish-dsym-upload.sh">

<violation number="1" location="scripts/ci/nightly-steps/app-finish-dsym-upload.sh:3">
P3: This header claims the step's `env:` still sets every input, but the 'Finish the dSYM upload to Sentry' step has no `env:` block. The script only reads `$RUNNER_TEMP`, which the runner sets. Reword the comment so future maintainers don't look for an `env:` binding that isn't there.</violation>
</file>

<file name="scripts/ci/nightly-steps/publish-release-metadata.sh">

<violation number="1" location="scripts/ci/nightly-steps/publish-release-metadata.sh:9">
P3: This note says other channels keep the Actions token, but the step passes `GH_APP_TOKEN` for every non-nightly-next track, and `finalize-release.py` uses that App token first. The step's `if:` also skips nightly-next, so this script never publishes that track. Rewrite the note to describe the actual token order.</violation>
</file>

<file name="scripts/ci/nightly-steps/github-names-move-tag.sh">

<violation number="1" location="scripts/ci/nightly-steps/github-names-move-tag.sh:3">
P3: This header comment is inaccurate: the step's `env:` sets only `GH_TOKEN`, `GITHUB_TOKEN` and `BUILD_SHA`. `CHANNEL_RELEASE_TAG` and `PUBLISH_TOOLING` come from the job-level `env:`. Reword it so anyone moving this script knows the job env is also required.</violation>
</file>

<file name="scripts/ci/nightly-steps/publish-move-release-tag.sh">

<violation number="1" location="scripts/ci/nightly-steps/publish-move-release-tag.sh:11">
P3: The moved note's "see above" pointed to the workflow context it was lifted from, and that context is gone from this file. Drop the back-reference, or name the step it means.</violation>
</file>

<file name="scripts/ci/nightly-steps/publish-prepare-release-notes.sh">

<violation number="1" location="scripts/ci/nightly-steps/publish-prepare-release-notes.sh:11">
P3: The timing note doesn't match the workflow. The notes step's `timeout-minutes` is 70 in nightly.yml, so a 60-minute rate-limit wait leaves 10 minutes, not 20. Correct the figure, and say the bound is set in nightly.yml rather than "here".</violation>
</file>

<file name="scripts/ci/nightly-steps/app-build-release.sh">

<violation number="1" location="scripts/ci/nightly-steps/app-build-release.sh:3">
P3: The header says the step's `env:` sets every input, but `NIGHTLY_COMPILE_ARCHS` and `CHANNEL_ICON_NAME` come from the job-level `env:`. Reword the note to name both levels so it stays accurate.</violation>
</file>

<file name="scripts/ci/nightly-steps/app-upload-dsyms.sh">

<violation number="1" location="scripts/ci/nightly-steps/app-upload-dsyms.sh:1">
P2: This script exists only in commits from this PR onward. Non-resume runs can check out an older build_sha, as the header's run 37568319518 reference shows, so `bash -e` exits 127 and the dSYM upload is silently skipped. The inline step did not have this problem. The PR's inline-jobs exception covers only unsigned resumes. Consider keeping the upload body inline in this job, or making the step fall back to the inline logic when the script is absent.</violation>
</file>

<file name="scripts/ci/nightly-steps/deltas-generate-appcast.sh">

<violation number="1" location="scripts/ci/nightly-steps/deltas-generate-appcast.sh:1">
P2: The generate-deltas job checks out `.publish-tooling` sparsely without this new script, then invokes it from that checkout. Add the script to the sparse-checkout list; otherwise every delta generation fails before producing output.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai · 2026-11-01T14:25:08Z

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash


P2: This script exists only in commits from this PR onward. Non-resume runs can check out an older build_sha, as the header's run 37568319518 reference shows, so bash -e exits 127 and the dSYM upload is silently skipped. The inline step did not have this problem. The PR's inline-jobs exception covers only unsigned resumes. Consider keeping the upload body inline in this job, or making the step fall back to the inline logic when the script is absent.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/app-upload-dsyms.sh, line 1: <comment>This script exists only in commits from this PR onward. Non-resume runs can check out an older build_sha, as the header's run 37568319518 reference shows, so `bash -e` exits 127 and the dSYM upload is silently skipped. The inline step did not have this problem. The PR's inline-jobs exception covers only unsigned resumes. Consider keeping the upload body inline in this job, or making the step fall back to the inline logic when the script is absent.</comment> <file context> @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# nightly.yml build-nightly-app: "Upload dSYMs to Sentry". Moved out of the workflow to keep nightly.yml under +# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input. </file context>

cubic-dev-ai · 2026-11-01T14:25:08Z

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash


P2: The generate-deltas job checks out .publish-tooling sparsely without this new script, then invokes it from that checkout. Add the script to the sparse-checkout list; otherwise every delta generation fails before producing output.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/deltas-generate-appcast.sh, line 1: <comment>The generate-deltas job checks out `.publish-tooling` sparsely without this new script, then invokes it from that checkout. Add the script to the sparse-checkout list; otherwise every delta generation fails before producing output.</comment> <file context> @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# nightly.yml generate-nightly-deltas: "Generate deltas and revised appcast". Moved out of the workflow to keep nightly.yml under +# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input. </file context>

cubic-dev-ai · 2026-11-01T14:25:08Z

 # nightly.yml under GitHub's workflow file size limit; the job's and step's env: still set every input.
+#
+# The step's notes, moved from nightly.yml:
+# still-newest: exit 0 while no build above this one is on the live feed. Every mutable


P3: The new header note says still-newest exits 0 while no build above this one is on the feed, but the script actually exits 0 only when the feed serves exactly this build. It exits 1 for an older feed build too, so every guarded write is skipped in that case. Reword the note to match the equality check.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/github-names-install-still-newest.sh, line 6: <comment>The new header note says `still-newest` exits 0 while no build above this one is on the feed, but the script actually exits 0 only when the feed serves exactly this build. It exits 1 for an older feed build too, so every guarded write is skipped in that case. Reword the note to match the equality check.</comment> <file context> @@ -1,6 +1,11 @@ # nightly.yml under GitHub's workflow file size limit; the job's and step's env: still set every input. +# +# The step's notes, moved from nightly.yml: +# still-newest: exit 0 while no build above this one is on the live feed. Every mutable +# write below (rolling name, feed copy, body, tag) runs it just before writing, so an +# older run whose uploads waited out a rate limit never writes over a newer build's. </file context>

Suggested change

# still-newest: exit 0 while no build above this one is on the live feed. Every mutable

# still-newest: exit 0 only while this build is the newest on the live feed; it exits 1 when the feed serves any other build, older or newer. Every mutable

cubic-dev-ai · 2026-11-01T14:25:08Z

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+# nightly.yml attest-nightly-remote-daemon: "Report a missing daemon attestation". Moved out of the workflow to keep nightly.yml under
+# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input.


P3: This header comment is inaccurate. The workflow step has no env: block and the script reads no inputs, so drop the claim or describe what the script actually needs.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/attest-report-missing.sh, line 3: <comment>This header comment is inaccurate. The workflow step has no `env:` block and the script reads no inputs, so drop the claim or describe what the script actually needs.</comment> <file context> @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +# nightly.yml attest-nightly-remote-daemon: "Report a missing daemon attestation". Moved out of the workflow to keep nightly.yml under +# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input. +echo "::warning title=SSH daemon assets shipped without provenance::actions/attest-build-provenance failed twice (usually the installation API rate limit). The nightly continues; gh attestation verify will find no attestation for this build's cmuxd-remote assets." </file context>

cubic-dev-ai · 2026-11-01T14:25:08Z

+hdiutil attach "$dmg" -nobrowse -readonly -mountpoint "$mnt" >/dev/null
+mkdir -p tested-plist
+cp "$mnt/cmux NIGHTLY.app/Contents/Info.plist" tested-plist/Info.plist
+hdiutil detach "$mnt" >/dev/null || hdiutil detach -force "$mnt" >/dev/null


P3: If cp or mkdir fails after the attach, the script exits without detaching the image. Register an EXIT trap right after the attach, as the sibling DMG scripts do, so the mount is always released.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/legacy-identity-verify-hand-info-plist.sh, line 16: <comment>If `cp` or `mkdir` fails after the attach, the script exits without detaching the image. Register an EXIT trap right after the attach, as the sibling DMG scripts do, so the mount is always released.</comment> <file context> @@ -0,0 +1,17 @@ +hdiutil attach "$dmg" -nobrowse -readonly -mountpoint "$mnt" >/dev/null +mkdir -p tested-plist +cp "$mnt/cmux NIGHTLY.app/Contents/Info.plist" tested-plist/Info.plist +hdiutil detach "$mnt" >/dev/null || hdiutil detach -force "$mnt" >/dev/null +/usr/libexec/PlistBuddy -c "Print :CFBundleVersion" tested-plist/Info.plist | grep -Fxq "$NIGHTLY_BUILD" </file context>

cubic-dev-ai · 2026-11-01T14:25:09Z

+# nightly-next commits change .github/workflows relative to main, and
+# GitHub refuses the Actions token any release create/update that targets
+# such a commit. nightly-next therefore publishes its release metadata
+# with the release App token; other channels keep the Actions token.


P3: This note says other channels keep the Actions token, but the step passes GH_APP_TOKEN for every non-nightly-next track, and finalize-release.py uses that App token first. The step's if: also skips nightly-next, so this script never publishes that track. Rewrite the note to describe the actual token order.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/publish-release-metadata.sh, line 9: <comment>This note says other channels keep the Actions token, but the step passes `GH_APP_TOKEN` for every non-nightly-next track, and `finalize-release.py` uses that App token first. The step's `if:` also skips nightly-next, so this script never publishes that track. Rewrite the note to describe the actual token order.</comment> <file context> @@ -1,6 +1,12 @@ +# nightly-next commits change .github/workflows relative to main, and +# GitHub refuses the Actions token any release create/update that targets +# such a commit. nightly-next therefore publishes its release metadata +# with the release App token; other channels keep the Actions token. set -euo pipefail { </file context>

Suggested change

# with the release App token; other channels keep the Actions token.

# with the release App token; other channels pass the release App token first (GH_APP_TOKEN) with the Actions token as fallback.

cubic-dev-ai · 2026-11-01T14:25:09Z

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+# nightly.yml publish-nightly-next-github-names: "Move the nightly-next tag to the published build". Moved out of the workflow to keep nightly.yml under
+# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input.


P3: This header comment is inaccurate: the step's env: sets only GH_TOKEN, GITHUB_TOKEN and BUILD_SHA. CHANNEL_RELEASE_TAG and PUBLISH_TOOLING come from the job-level env:. Reword it so anyone moving this script knows the job env is also required.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/github-names-move-tag.sh, line 3: <comment>This header comment is inaccurate: the step's `env:` sets only `GH_TOKEN`, `GITHUB_TOKEN` and `BUILD_SHA`. `CHANNEL_RELEASE_TAG` and `PUBLISH_TOOLING` come from the job-level `env:`. Reword it so anyone moving this script knows the job env is also required.</comment> <file context> @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# nightly.yml publish-nightly-next-github-names: "Move the nightly-next tag to the published build". Moved out of the workflow to keep nightly.yml under +# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input. +set -euo pipefail +"$RUNNER_TEMP/still-newest" || exit 0 </file context>

cubic-dev-ai · 2026-11-01T14:25:09Z

+# after every publication step succeeds. A newer commit becomes the next
+# run's candidate. The tag is deliberately the only operation using the
+# elevated app token on classic channels (nightly-next also needs it for
+# release metadata, see above); assets and feeds use the Actions token and


P3: The moved note's "see above" pointed to the workflow context it was lifted from, and that context is gone from this file. Drop the back-reference, or name the step it means.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/publish-move-release-tag.sh, line 11: <comment>The moved note's "see above" pointed to the workflow context it was lifted from, and that context is gone from this file. Drop the back-reference, or name the step it means.</comment> <file context> @@ -0,0 +1,23 @@ +# after every publication step succeeds. A newer commit becomes the next +# run's candidate. The tag is deliberately the only operation using the +# elevated app token on classic channels (nightly-next also needs it for +# release metadata, see above); assets and feeds use the Actions token and +# remain the publication record if this marker is stale. +set -euo pipefail </file context>

cubic-dev-ai · 2026-11-01T14:25:09Z

+# any asset upload instead of using up the job's time and being
+# cancelled mid-upload (run 37611677848 left partial assets that way).
+# 60 of these minutes are the most its gh calls may wait for a rate-limit
+# reset (gh_budget_wait.sh); the job's timeout leaves 20 after them.


P3: The timing note doesn't match the workflow. The notes step's timeout-minutes is 70 in nightly.yml, so a 60-minute rate-limit wait leaves 10 minutes, not 20. Correct the figure, and say the bound is set in nightly.yml rather than "here".

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/publish-prepare-release-notes.sh, line 11: <comment>The timing note doesn't match the workflow. The notes step's `timeout-minutes` is 70 in nightly.yml, so a 60-minute rate-limit wait leaves 10 minutes, not 20. Correct the figure, and say the bound is set in nightly.yml rather than "here".</comment> <file context> @@ -2,6 +2,14 @@ +# any asset upload instead of using up the job's time and being +# cancelled mid-upload (run 37611677848 left partial assets that way). +# 60 of these minutes are the most its gh calls may wait for a rate-limit +# reset (gh_budget_wait.sh); the job's timeout leaves 20 after them. +# nightly-next runs are pushes to nightly-next; their PRs merge into feat-cmux-next. set -euo pipefail </file context>

Suggested change

# reset (gh_budget_wait.sh); the job's timeout leaves 20 after them.

# reset (gh_budget_wait.sh); the notes step's timeout-minutes (70 in nightly.yml) leaves 10 after them.

cubic-dev-ai · 2026-11-01T14:25:09Z

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# nightly.yml build-nightly-app: "Build nightly app (Release)". Moved out of the workflow to keep nightly.yml under
+# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input.


P3: The header says the step's env: sets every input, but NIGHTLY_COMPILE_ARCHS and CHANNEL_ICON_NAME come from the job-level env:. Reword the note to name both levels so it stays accurate.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly-steps/app-build-release.sh, line 3: <comment>The header says the step's `env:` sets every input, but `NIGHTLY_COMPILE_ARCHS` and `CHANNEL_ICON_NAME` come from the job-level `env:`. Reword the note to name both levels so it stays accurate.</comment> <file context> @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# nightly.yml build-nightly-app: "Build nightly app (Release)". Moved out of the workflow to keep nightly.yml under +# its size budget (tests/test_workflow_file_size.py); the step's env: still sets every input. +set -euo pipefail +# cx-f58x: compile out browser-import code as `decide` chose (the </file context>

…ublish-tooling checkout lists single files) Review of #31013 at 8a2494b: generate-nightly-deltas checks out .publish-tooling with a non-cone sparse list of five named files (anchor &checkout-the-gh-token-wrapper), so scripts/ci/nightly-steps/deltas-generate-appcast.sh would not be in it and the step would fail on every run, hidden by the job's continue-on-error. My selection matched the directory name as a substring of the list. The step goes back exactly as on the feat tip (block and its notes), the script is removed, and test_nightly_sparkle_key.sh is back to the feat tip's text. An exact check of all 140 step-script calls (the last checkout before each call that provides its path, root or .publish-tooling, must contain the file: no sparse set, a cone-mode directory or a non-cone pattern) finds no gap at this head; it found exactly this one before the fix, and none in the feat tip's 127 calls. nightly.yml: 458,313 bytes (31,687 under 490,000). All 124 tests that read nightly.yml: the same results as at the feat tip 49aa291. shellcheck -S warning clean; actionlint: the same 5 findings as before.

…d-nightly-size-2

lawrencecchen · 2026-11-01T15:25:54Z

Review (subagent, correctness first): SHIP at ee31939

The first review, at 8a2494b, blocked one defect. generate-nightly-deltas checks out .publish-tooling with a non-cone sparse list of five named files, so the moved script would not exist there. The step would fail on every run, and continue-on-error would hide it. That step now stays inline, and its job is identical to the feat tip in yaml.safe_load.

The re-review at this head confirmed:

Two independent checkers find 0 gaps in 140 step-script calls: each script exists in the checkout its step runs from. Both checkers flag the gap on the old head.
13 changed steps. Only run and new env keys differ, and no existing env value changed.
The same 11 env entries over 7 steps. Each body equals the old block with ${{ }} replaced by ${NAME}, and ALLOW_NON_DESCENDANT_FLAG uses the intended :+ form.
bash -e is used exactly on the 4 blocks that had no top-level set -e.
Comment move: header-only edits, and no shellcheck directive moved.
Mutation map: emptying each moved script gives the same test failures at head as at base. Four of them had no test coverage at base either.
These pass: the size test, the nightly size test, the universal build test, the sparkle key test and the edited tests. shellcheck is clean.

Follow-up, out of scope here: add scripts/ci/nightly-steps to CMUX_TUI_TREE_SAME_PATHS, so a build behind the tip cannot run an older copy of a moved script.

lawrencecchen · 2026-11-01T15:27:40Z

The tooling-tests red on this PR is a base red, not caused by this PR. tests/test_ci_overflow_pick_reads_pool.py (from #30878) is registered as linux-guard, but no workflow runs it, so the registry check fails on every PR. #31065 fixes it.

lawrencecchen · 2026-11-01T15:39:23Z

Review (subagent, correctness first): SHIP at 0d005de

git show --remerge-diff 0d005de23a39 shows no own changes (parents ee31939 and feat 74ab26dca2eb).
Patch-id of the PR patch is equal: 2ba003d6a975 at ee31939 (base 79e0181) and at the new head (base 74ab26dca2eb).
Feat changed 65 files between the two merge-bases, none in nightly.yml, scripts/ci/nightly-steps/, or any test/script that reads nightly.yml. No moved block has a lost change.
nightly.yml at the new head is 458313 bytes (< 490000); every scripts/ci/nightly-steps/*.sh it calls exists at the new head.

… registered as linux-guard becomes manual (#31065) validate_test_execution_registry.py --pr-lanes feat-cmux-next failed on every pull request (seen on #31013): tests/test_ci_overflow_pick_reads_pool.py: linux-guard lane is not run by any workflow #30878 (da249ca) registered it as linux-guard and added no ci-guards.yml step. CLAUDE.md forbids new CI tests (Lawrence 2026-10-23), so it becomes a manual entry with its run command, as #30624 and #30872 did. The test passes (about 2 s). Red at 9dee26d: python3 scripts/ci/validate_test_execution_registry.py --pr-lanes feat-cmux-next --base-sha HEAD exit 1 Green here: same command, exit 0.

…nightly-steps (#31121) * test(nightly): the resolver refuses a published commit whose nightly step scripts differ from the tip's (red) Since #30912 the nightly runs step scripts from scripts/ci/nightly-steps/ in the resolved commit's checkout, under the tip's workflow. A published commit behind the tip whose step scripts differ would run older copies of steps the tip's workflow names. The case runs nightly.yml's own CMUX_TUI_TREE_SAME_PATHS policy. Red here: bash scripts/cmux-next/tests/pin-cmux-tui-resolve-newest.test.sh FAIL: nightly must not resolve a published revision with an older nightly step script * fix(nightly): the resolved build commit carries the tip's scripts/ci/nightly-steps resolve-nightly-cmux-tui-client's CMUX_TUI_TREE_SAME_PATHS gains scripts/ci/nightly-steps. pin-cmux-tui.sh compares `git rev-parse <rev>:<path>`, which for a directory is its tree, so a candidate whose step scripts differ in any file from the tip's is skipped with "scripts/ci/nightly-steps differs from the tip", as for nightly.yml and the Chief lockfile. Follow-up from the #31013 review. Cost: a nightly no longer builds a published commit behind the tip when the tip changed a step script after it; it takes a newer published commit, or builds the tip while its tree publishes (resolve-tree-pending.sh), as for a nightly.yml change today. Green here: bash scripts/cmux-next/tests/pin-cmux-tui-resolve-newest.test.sh PASS

* ci guards: depth-limited fetches only in shallow checkouts (base red) scripts/measure/fresh-home-launch.sh and workers/cmux-vm/scripts/test-mesh-label-compat.ts passed --depth to a fetch in the caller's checkout, which turns a full clone shallow. Both now pass it only when git rev-parse --is-shallow-repository says the checkout is already shallow. Regressed by #30495 (fresh-home-launch.sh) and #30501 (test-mesh-label-compat.ts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: map the transcript's TurnRows.tsx in place of the deleted MessageGroupRow.tsx (base red) #29800 removed MessageGroupRow.tsx (every sent message is its own card); TurnRows.tsx now draws each sent message row in order (SentMessageLine.tsx is already mapped), so chat-transcript maps it. Regressed by #29800 (deleted the file #28698 had mapped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: scan cmux-next-ios/verify/ for errno-after-assertion (base red) cmux-next-ios/verify/imsg-replies/GestureServerUITests.swift is an XCTest source outside every TEST_ROOTS entry. Scan cmux-next-ios/verify/ (it has no errno use, so nothing else changes). Regressed by 201101a (added the verify/imsg-replies rig; landed by a safe-push train, no PR). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: list the four manual-lane workflow readers in UNWIRED (base red) tests/test_ci_blacksmith_linux_sizes.py, test_ci_overflow_pick_reads_pool.py, test_nightly_next_control_rescue.py and test_ci_cloud_probe_rescue.py read .github/workflows/ but are lane = "manual" in tests/test-execution.toml (no new CI tests; wiring them as linux-guard reddened tooling-tests on every PR). They join UNWIRED with the same reason as the other manual-lane entries, and each says when to run it by hand. Regressed by #30360, #30878, #30657 and #30796 (each added its test without an UNWIRED entry). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: fleet rule lints pass again (base red) "ci: Validate fleet rule lints" failed on feat-cmux-next in three rules. big-red: the personal machine's name left product code and tests. Doc comments say "one lane VM" / "<host>"; the webview tests use my-vm (its machine hue still differs from gpu-box's). Regressed by #28105 (peer_handshake_tests.rs, list_reply.rs, list_reply_tests.rs), #29495 (host_update_tests.rs), #29126 (chatOpenPacedList.test.ts), #31108 (project-chooser-machine-tag). pkill: the iOS verify rigs and the tool-evals remote daemon stop only what they started. imsg-replies records the conversation-sim, the recorders (simctl itself, not the xcrun shim) and the gesture servers in pidfiles and signals those PIDs after checking the command line still matches; pq_browser_events records its two site servers' PIDs; the tool-evals remote stop kills the setsid group whose PID the launcher recorded beside the socket. Regressed by 201101a (imsg-replies, first in #22328), #25915 (pq_browser_events.py), #29080 (tool-evals/session.mjs). Allowlisted as safe, with reasons: scripts/ci/macos-stubs/Security.tbd (a linker stub listing symbol names, no keychain read; #27603) and tests/tool-evals/cases/hard.ndjson (an eval case forbidding pkill; #28738). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: determinize python tests flagged by test-determinism (base red) Regressed by the commits introducing each assertion (see report). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: nightly Xcode selection, unused anchors, daemon retention key (base red) - Validate macOS jobs select a pinned Xcode: the nine flagged nightly.yml jobs already run select-ci-xcode.sh through the `- *select-xcode` step alias; the guard did not expand step aliases. It now resolves `- *name` to the `- &name` step, with self-tests for both an alias to the selector and an alias to another step. Regressed by #30471 (steps turned into aliases to keep nightly.yml under its size limit). - Lint every workflow: drop the five sign-nightly-identity-legacy-* anchors left after the Blacksmith twin that aliased them was removed (a7b6a61, "legacy sign and deltas run on owned Macs only"; the twin came from #27357). No behaviour change. - upload-artifact retention: the 14-day cmux-nightly-remote-daemon upload moved from build-nightly-app to build-nightly-remote-daemon in #25935 with its retention unchanged; move the allowlist key with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: media tours and visual-gate state match their validators (base red) Drop the unknown step key "record" (re-added after #26888 removed it by #27933 and f283a2e), the unimplemented top-level frameInterval in home-chief-shimmer (c022070), the focusLeft right after closePane in palette-chat-toggles (#25405), split pane-polish (31 steps) into pane-polish and pane-polish-themes, and give the app-store-consent-deploy state a synthetic host that is not an email shape (4a05792). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: allowlist remaining test-determinism findings with reasons (base red) Swift/web/tool-evals findings cannot be determinized without Mac or webview runs; each entry names its reason and adding PR. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: Linux pool for the push kicks; record pool readers as unrescued (base red) - cmux-next-base-tier-kick, cmux-next-dev-kick and ios-internal-nightly-trigger waited on one fixed Blacksmith Linux size (!startsWith(LINUX_RUNNER) || overflow 8 || 2 vCPU). They now read fromJSON(vars.RUNNER_POOLS).linux after the fork branches. Their pinning tests (base tier, DEV kick, iOS internal builder) assert the pool read instead. Regressed by #30837. - owned_pool_rescue.py: feat-cmux-next's UNRESCUED_WORKFLOW_PATHS records the secretless Linux pool readers the owned-pool sources test found unlisted (fast gates, gallery health, PR checks, cloud backend check, web validation, and the three kicks above), with the existing _POOL_READ_WHY. Regressed by #29844 and #31138. - test_ci_linux_single_size.py: drop cmux-next-fast-gates.yml from FIXED_PAIR (moved by #31138); test_ci_overflow_pick_reads_pool.py: drop the two kicks from PICKS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: signing-runner and retry-attempt guards follow the owned-Mac signing rule (base red) Signing keychain guard (rule 7): accept owned-pool runner variables only where safety is proven in the job itself: - 7a: a job in glaeda's exclusive signing class whose first step is the verbatim claim gate may fall back to CI_NIGHTLY_NEXT_BUILD_RUNNER / CI_SIDE_LANE_RUNNER (nightly.yml sign-nightly-identity-legacy and generate-nightly-deltas, owned-only by design, Lawrence a7b6a61; deltas' Sparkle key added in #29261). test_ci_signing_owned_mac.py holds the claim script, always() keychain delete and twin/picker rules for them. - 7b: stable-next.yml sign (#27898) reads every signing secret only under env.SIGN == 'developer-id', whose condition picks a literal Blacksmith label before any variable; its owned dry-run holds no key. Proven structurally. Retry guard: cmux-next.yml swift-test/daemon-test/generated-files' side-lane fallback (#29814) is now gated at run_attempt <= 3 for the bot, so the bot's attempt 4 takes blacksmith-6vcpu-macos-26 like every other cmux-next fleet fallback (#22565). The three owned-only nightly jobs are exempt while they keep exactly their owned route. The textual check now forbids naming attempts after 2 (== 3+), as its comment says, instead of also == 2 (#29847 nightly.yml, #29130 cmux-next-guards.yml name the bot's attempt 2, which may stay owned). Regressed by #29814, #27898, #29261, #29847, #29130 and a7b6a61. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: owned Mac pools through the picker/vars pattern; follow nightly step scripts for sentry-cli (base red) check_no_self_hosted_fleet_runners The guard told every workflow to "use a cloud label so required jobs never land on a mini". That is outdated policy: owned fleet Macs now come first for macOS jobs (Linux, then owned Macs through the picker, Blacksmith last), release signing may run on owned Macs, display tests gate nothing, and the NEXT nightly runs on the fleet with the nightly-only minis dedicated to it. The guard kept up only through exact per-workflow selector strings, which drifted with every routing change and turned the guard red on base: - cmux-next.yml swift-test/daemon-test/generated-files: the glaeda-side deferred fallback (regressed by #29814) and the attempt-1 gui selector (#28605, #27853) - cmux-next-base-autofix.yml / cmux-next-pr-autofix.yml: the gui selector gained a re-run predicate (#27853) - nightly.yml sign/prepare jobs: owned pool as the final fleet fallback (#29154) Fix the guard instead of moving those jobs to Blacksmith: an owned POOL label (glaeda-[<lane>-]<class>-xcode-<version>, served by many minis, never one host) is accepted as a quoted literal in a ${{ }} runs-on that routes through the declared picker/vars pattern (reads needs.<job>.outputs.* or vars.*). In a workflow a pull request can trigger, the fork pull request branch (fork heads to Blacksmith) must still come first on the line, so fork code never reaches an owned Mac (runner trust unchanged). Still refused: bare self-hosted/macOS/ARM64, single-host labels (cmux-macos, mac-mini, slot-N, tart-*, ...), macos-26 outside the fork branch, and an owned label named directly with no picker or variable route. The exact gui selector strings are replaced by this rule; self-tests cover each accept and refuse case. check_sentry_cli_install_portability #31013 moved nightly.yml's "Upload dSYMs to Sentry" body into scripts/ci/nightly-steps/app-upload-dsyms.sh (size budget), so the guard no longer saw the ./scripts/upload-sentry-dsyms.sh call. The guard now reads the step's step script too. Regressed by #31013. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: change-area tests follow the RUNNER_POOLS and devEngines routing; helper walk skips non-running namers (base red) - release-admission and remote-daemon-admission read fromJSON(vars.RUNNER_POOLS).linux by design since #30878; the two tests now assert that instead of vars.LINUX_RUNNER. - fast-checks.yml's setup-bun reads the bun pin from webviews/package.json devEngines at run time (#29757); the setup-bun test accepts a steps.<id>.outputs.version whose step reads devEngines, and still requires the literal pin everywhere else. - indexnow.yml pinned bun 1.3.14 when #30697 backported it; it now installs devEngines' 1.4.4. - ci_helper_areas() failed open on files that name a helper without running it, so owned_pool_rescue.py, owned_build_state.py and e2e_warm_derived_data.py stopped routing by where they run. Regressed by #28528 (test_cmux_next_route.py names owned_pool_rescue.py as a routing input and itself), #28513 (docs/ci/cmux-next-tiers.md names that test), #30657 (nightly_next_control_rescue.py's only test is manual-lane), #28568 (.github/warm-sources.json prose) and #30467 (base_red_excuse.py's guard-test table and its test's fixtures). The walk now ignores a test naming itself, docs, a Python test naming it only in a docstring, the listed path tables and warm-sources.json, and treats a test the registry declares in an inert lane that nothing imports as a dead end. An unregistered test nothing names still fails open. - cache_restore_receipt.py now routes precisely (macOS + Release, via ci-macos.yml's cache-restore action) instead of failing open; its assertion says so. python3 tests/test_ci_change_areas.py: 4 of 234 failed before, 0 after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: review fixes for #31366 (daemon pgid, fork-first owned pools, stale pidfiles) - tests/tool-evals/session.mjs: the remote launcher recorded $! of the backgrounded && chain (the wrapping subshell), so the stop signalled a group that did not exist. setsid now starts a shell that writes its own PID and execs the daemon, so the recorded PID is the daemon's process group. Checked locally: the recorded PID equals the daemon's PGID and SID, and kill -TERM -- -PID stops it. - tests/test_ci_self_hosted_guard.sh (security): the owned-pool exemption in pull request workflows only required the fork branch somewhere before the owned literal, so `needs.x.outputs.runner || <fork branch> || 'glaeda-...'` passed. The fork branch must now be the first selector of the expression (after only the hosted repository_owner clause). New negative self-tests for reordered expressions; red with the old match, green with the new. - cmux-next-ios/verify/parity/pq_browser_events.py: stop_site() now stops only servers this run started (serve() reports each pidfile it wrote), kills a PID only while its command line is still that server's, and removes the pidfile after. Checked with a local stub harness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: universal nightly helper runs-on match works under mawk (base red) The helper runs-on expected string passed to awk -v carried `\&\&`, which gawk reads as `&&` (with a warning) but mawk, the default awk on Ubuntu runners, keeps as `\&\&`, so the whole expression never matched and "nightly must build and hand off the macOS 26 helper" failed. Plain `&&` means the same in both. Regressed by #30824. bash tests/test_nightly_universal_build.sh: FAIL before, PASS after (mawk 1.3.4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…d clean (#31755) The workflow's Release step now only runs app-build-release.sh (#31013). The step-text hash therefore no longer sees the xcodebuild flags in that script or in compilation-cache.sh, so a flag change could reuse objects built under the old flags. Both scripts now count toward the input key. Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

lawrencecchen added 11 commits November 1, 2026 09:39

Merge remote-tracking branch 'origin/feat-cmux-next' into fix/base-re…

3f10db8

…d-nightly-notary

Merge remote-tracking branch 'origin/feat-cmux-next' into fix/base-re…

6238504

…d-nightly-notary

Merge remote-tracking branch 'origin/feat-cmux-next' into fix/base-re…

17bf155

…d-nightly-size

lawrencecchen added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026

lawrencecchen temporarily deployed to ci-routing-agent November 1, 2026 14:16 — with GitHub Actions Inactive

cubic-dev-ai Bot reviewed Nov 1, 2026

View reviewed changes

lawrencecchen added 2 commits November 1, 2026 15:21

Merge remote-tracking branch 'origin/feat-cmux-next' into fix/base-re…

ee31939

…d-nightly-size-2

lawrencecchen temporarily deployed to ci-routing-agent November 1, 2026 15:22 — with GitHub Actions Inactive

lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Nov 1, 2026

lawrencecchen mentioned this pull request Nov 1, 2026

base red: tooling-tests registry, test_ci_overflow_pick_reads_pool.py (#30878) becomes manual #31065

Merged

Merge branch 'feat-cmux-next' into fix/base-red-nightly-size-2

0d005de

cmux-bot Bot temporarily deployed to ci-routing-agent November 1, 2026 15:37 Inactive

cmux-bot Bot temporarily deployed to ci-routing-agent November 1, 2026 15:38 Inactive

lawrencecchen added ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds and removed ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds labels Nov 1, 2026

lawrencecchen merged commit 3c0c02a into feat-cmux-next Nov 1, 2026
188 of 196 checks passed

lawrencecchen deleted the fix/base-red-nightly-size-2 branch November 1, 2026 16:04

This was referenced Nov 1, 2026

fix(nightly): the resolved build commit carries the tip's scripts/ci/nightly-steps #31121

Merged

nightly runs scripts from the build commit's checkout, so older copies of scripts/ci/*.py and scripts/cmux-next/*.sh can run #31127

Open

github-actions Bot mentioned this pull request Nov 1, 2026

feat-cmux-next: Swift test targets do not compile #31122

Closed

teamleaderleo mentioned this pull request Nov 1, 2026

base red: CI guard profile fails on base content #31366

Merged

teamleaderleo mentioned this pull request Nov 2, 2026

ci(nightly): a change to the Release build script makes the next build clean #31755

Merged

teamleaderleo mentioned this pull request Nov 2, 2026

ci(nightly): the nightly-next Mac compile records the Linux build's REF #25812

Open

	# still-newest: exit 0 while no build above this one is on the live feed. Every mutable
	# still-newest: exit 0 only while this build is the newest on the live feed; it exits 1 when the feed serves any other build, older or newer. Every mutable

	# with the release App token; other channels keep the Actions token.
	# with the release App token; other channels pass the release App token first (GH_APP_TOKEN) with the Actions token as fallback.

	# reset (gh_budget_wait.sh); the job's timeout leaves 20 after them.
	# reset (gh_budget_wait.sh); the notes step's timeout-minutes (70 in nightly.yml) leaves 10 after them.

Conversation

lawrencecchen commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

(a) The remaining safe blocks (6)

(c) Blocks with ${{ }}: each expression is passed through env: (7 blocks, 11 expressions)

(b) Step notes move into their script's header (46 steps, 13.6 KB)

Why the 7 resume jobs are left out, and when they move

Tests

Evidence

Observed / Not run

Changelog

Uh oh!

github-actions Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blacksmith-sh Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

lawrencecchen commented Nov 1, 2026

Uh oh!

lawrencecchen commented Nov 1, 2026

Uh oh!

lawrencecchen commented Nov 1, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lawrencecchen commented Nov 1, 2026 •

edited

Loading

(c) Blocks with `${{ }}`: each expression is passed through `env:` (7 blocks, 11 expressions)

github-actions Bot commented Nov 1, 2026 •

edited

Loading

blacksmith-sh Bot commented Nov 1, 2026 •

edited

Loading