[teamleaderleo]

Summary

Leo's ask, after a direct push broke every nightly-next at a step only the nightly ran: "Every step that only nightly runs gets a PR-time dry run."

.github/nightly-step-coverage.json now lists every step of nightly.yml, plus the reusable workflows it calls (today nightly-tag-permission-probe.yml), keyed by job id and step name. Each entry is one of:

• {"pr": "<workflow>:<job>:<step>"}: a step in a workflow that runs on pull requests runs the same script, on fixtures or as a dry run.
• {"plumbing": true}: setup and transfer steps with no check of their own.
• {"nightly_only": "<reason>"}: steps that need signing or publishing secrets, the Release app, or a Mac. The reason is written out.

Steps that use an external checkout, artifact, cache or setup action are counted as plumbing automatically (PLUMBING_ACTIONS) and have no entry. Local actions (./.github/actions/cache-restore, cache-save) always need a reviewed entry, because their steps can change in this repo. The nightly may call only local reusable workflows; an external uses: is refused.

tests/test_nightly_step_coverage.py enforces this in the release-notary guard group, which runs on every feat-cmux-next pull request through cmux-next-guards.yml. The test fails when:

• a nightly step has no entry, so a new nightly-only step can't land without a counterpart or a reviewed reason;
• an entry names a step that is gone or renamed;
• an entry has more than one kind, or a nightly_only reason too short to review;
• a pr value isn't a nonempty <workflow>:<job>:<step>, or the step doesn't exist;
• the counterpart can't run for a feat-cmux-next pull request:
  • Trigger: the workflow's pull_request filters must admit the branch. branches/branches-ignore patterns are evaluated in order like GitHub (a later ! pattern negates). Path filters are refused only when they provably admit no path: after the last match-all closing pattern (!** in paths, ** in paths-ignore) nothing reopens a path. Otherwise a caller that runs must call it.
  • Conditions: no caller job, job, needed job or step if: may be false for that PR. Conditions are read with GitHub's precedence (!, then &&, then ||) in three-valued logic, and string comparisons ignore case, as GitHub's do. Only comparisons of github.event_name, the base ref, needs.<job>.result for known-skipped jobs, a matrix value or a literal caller input decide; anything else, or anything unreadable, counts as possibly true.
  • Matrix: the job's real rows are expanded (cross product, multi-key exclude, include extending the original rows or adding its own, a caller's fromJSON input), and the step must be able to run in some row.
  • Guards route: cmux-next-guards.yml ci-guard-group runs ci-guards.yml's groups through guards-local.sh --feat-routes, not uses:. That route counts only when the job (its if and needs) can run and the invoking step can run in one of the job's matrix rows. Its groups come from detect_linux_guard_changes.py.

Fixture tests (CoverageFixtureTests, mostly one table, COUNTERPART_CASES) show each rejection, and a few pass cases: a null or malformed pr; triggers limited to main, negated by a later ! pattern, or ignoring every path (and a pass case for paths: ["src/**", "!src/x/**"]); caller, job, needed-job and step conditions that exclude the PR; a step that needs a skipped job's success; precedence; a case-insensitive github.event_name != 'PULL_REQUEST'; matrix rows removed by single- or multi-key excludes or absent from an include-only matrix; the guards route with its job or step switched off, or its step limited to a matrix value the job never has; an external reusable workflow; and a local action without an entry. To check the tests, I reverted each fix in a copy of the validator; each of those reversions fails at least one case.

The counterparts are derived by following uses: ./.github/workflows/... from nightly.yml, so a newly called reusable workflow is pulled in too. The test caught mistakes in the hand-made list it started from. "Build fast arm64 cmux-tui client" was mapped to cmux-tui-build-package.yml, which no pull request runs. "Build Ghostty CLI helper" and "Verify signed diff sidecar runtime contract" pointed at ci-macos.yml, which runs only for PRs into main. All three are nightly_only with that reason.

Totals: 234 entries (55 pr, 97 plumbing, 82 nightly_only), plus the derived external-action plumbing steps. Consecutive plumbing steps of a job share a registry line, to keep the PR under 800 lines.

The 20 gaps (steps with no PR-time counterpart before this PR)

Job / step	Disposition
publish-nightly / Require every DMG to match a tested digest	nightly_only for now. Follow-up PR (written, cut from this one for the 800-line cap): tests/test_nightly_tested_dmg_gate.py runs this step's own run: block from nightly.yml on fixture DMGs.
nightly-next-macos15-launch / Check every bundled Mach-O loads on macOS 14	nightly_only for now. Follow-up PR (written): tests/test_macho_minos.py runs the same macho_minos.py scan --floor 14.0 on synthetic Mach-O fixtures.
publish-nightly-identity-legacy / Refuse a DMG that is not the tested one	Now runs on PRs. tests/test_nightly_next_legacy_tested_digest.sh already existed, but only test_nightly_universal_build.sh ran it, after earlier checks that are red on base (below). It now runs directly in the new release-notary step "Dry-run the nightly's tested-DMG gate for the old feed".
prepare-nightly-variant / Inject the Ghostty dependency licenses	New dry run. The fixture tests of verify-ghostty-license-bundle.py and its collector (test_ghostty_licenses.py, test_check_bundle_notices.py) used to run only in the workflow_dispatch cmux-next-notices.yml. They now run in cmux-next-guards.yml vt-link-graph (Linux, every feat-cmux-next PR).
build-nightly-ghostty-cli-helper / libghostty-vt Zig packages are covered by the notices	New dry run. The --license-manifest mode's fixture test (test_check_ghostty_vt_notices.py) now runs in the same vt-link-graph step. PR CI used to run only --check-link-graph.
build-nightly-ghostty-cli-helper / Build the cmux-next source archive	nightly_only: zig fetches every Ghostty package on the macOS runner. Its collector and verifier logic is covered by the fixture step above.
build-nightly-app / Build nightly app (Release)	nightly_only: xcodebuild Release of the cmux scheme on a Mac. release-compile and cmux-scheme-compile cover parts of it.
build-nightly-app / Build the OptChat Chief, Build the cmux-next iPhone host	nightly_only: cargo builds for macOS targets need the macOS SDK.
build-nightly-app / Drop the empty Iroh.framework stub; Assemble the Computer Use helper	nightly_only: need the Release app, or a Mac compile of cmux-cua.
prepare-nightly-variant / Theme picker regression, Thin bundle, Verify nightly binary architectures	nightly_only: need the Release app plus lipo/file.
build-sign-notarize-nightly / Inject nightly identities and metadata; Start and Finish the signed-app checks	nightly_only: PlistBuddy and otool on the Release app.
verify-signed-nightly / Smoke launch signed app; Smoke bundled CLI	nightly_only: need the signed Release app running.
nightly-next-macos15-launch / Launch on macOS 15	nightly_only: needs the published DMG and a macOS 15 guest.

The two "Visual smoke" steps added since the list was made point at nightly-next-canary.yml canary / Visual smoke, which runs the same gate.sh.

Follow-ups (not in this PR, to stay under the size budget)

• The tested-DMG gate and macOS 14 Mach-O dry runs above (about 230 lines, already written).
• Add a path-filtered pull_request trigger to cmux-next-source-archive.yml (ghostty gitlinks, pinned licenses), so a new Zig package without a pinned license fails on the PR. That's a macOS job, so it isn't done here.
• Move "Verify nightly binary architectures" and "Inject nightly identities and metadata" into scripts, with fake-lipo and plistlib fixture tests on Linux.
• Run the signed-app checks (verify-bundle-load-commands.sh, verify-app-bundle-licenses.sh, macho_minos.py) on the release-compile app. They need no signature.

Base red found on the way

tests/test_nightly_universal_build.sh fails on base at line 1265: it expects the step "Skip a build older than the old feed's", which #24019 renamed to "Skip a build the old feed already has". The script exits at the first failure, so none of the checks after line 1265 run in "Validate universal nightly workflow". Several pr entries name that step. A separate PR (ci-speed/release-notary-red) fixes it; this PR doesn't touch it.

Testing

• Red, then green, four times. At 3346dcf the test has no registry and errors with .github/nightly-step-coverage.json is missing. At eab59d9 eight fixture tests fail on the first-review rules; 1787d9e passes them. At a840f3f five fixture tests for the second review fail; e02ba6a passes them. At 8ad6e8a three fixture cases for the third review fail; at 2fad46c all cases pass, the live registry included.
• I added a scratch step to publish-nightly in nightly.yml (reverted, not committed). The test then failed with nightly.yml job 'publish-nightly' step 'A new nightly-only step'.
• bash tests/test_nightly_next_legacy_tested_digest.sh and python3 scripts/cmux-next/notices/test_check_ghostty_vt_notices.py (21 tests) pass. test_check_bundle_notices.py and test_ghostty_licenses.py pass with a local shim for Path.stat(follow_symlinks=). The local python3 is 3.9, and the verifier needs 3.10+. The Ubuntu runners have 3.12.
• scripts/ci/validate_test_execution_registry.py reports the registry is valid. tests/test_cmux_next_guards_route.py, test_ci_linux_guard_routing.py, test_feat_ci_guard_group.py and test_ci_run_guards.py pass.
• python3 scripts/cmux-next/prepush-gate.py reports no new violations.

Observed / Not run

• Observed: everything listed above, run locally with Python on macOS.
• Not run: actionlint and shellcheck, which aren't installed here; CI's guard jobs lint the workflow edits. The new steps haven't run on a Linux runner yet; this PR's CI is their first run. No Swift or Rust build is involved.

Size: 799 added lines. 257 of them are the registry; the validator and its fixture tests are about 515.

Changelog

none

Proof

CI only, no UI.

Agent session: cmux16

🤖 Generated with Claude Code

[teamleaderleo]

Owned by ci-speed on cmux16 · session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 16774b3: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr24261 -f ref=ci-speed/nightly-step-dry-runs -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 35c4e3d to compare; do not merge until green; see lane-rules

[teamleaderleo]

Bead cx-dquw2 created for this PR by the cmux coordinator.

[blacksmith-sh]

Found 11 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmpmtc2m95a/summary-fail.json (live):/summary-fail.json (live):	View Logs
/tmp/tmptdyfkb95/summary-missing.json (live):/summary-missing.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs
test_base_passes (main.NoticesPairTests.test_base_passes)/ test_base_passes (main.NoticesPairTests.test_base_passes)	View Logs
test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)/ test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)	View Logs
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo dules)/ test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo dules)	View Logs

...and 1 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 15 of 15 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

Review (independent Codex xhigh via acpmux, correctness first): SHIP at da33041

Seven rounds; each finding (fail-open validator cases: null pr, trigger filters, conditions, matrices, reusable workflows, local actions, precedence, case, empty matrices) has a fixture committed red before its fix.

[teamleaderleo]

Review (independent Codex xhigh via acpmux, correctness first): SHIP at 16774b3

The only change since the SHIP at da33041 is a merge of feat-cmux-next; the reviewer re-read the full diff at this head.

[teamleaderleo]

base-red-watch: cmux-next guards / nightly-guards / workflow-guard-tests / release-notary went red on feat-cmux-next in https://github.com/manaflow-ai/cmux/actions/runs/39758434195 (first red at 64049b4, last green at ecc390d). Suspect: #24261 (commit 64049b4, which touches .github/workflows/ci-guards.yml). If it is yours, fix forward on feat-cmux-next.

[github-actions]

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/39766181620 at 81640d1 failed: cmux-next web bundles (source-keyed cache).
Those jobs last passed at ecc390d. This pull request is one of 4 merged since: #24069, #24261, #24351, #24051.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

[github-actions]

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/39767073305 at aa8d973 failed: cmux-next web bundles (source-keyed cache).
Those jobs last passed at ecc390d. This pull request is one of 5 merged since: #24069, #24261, #24351, #24051, #24325.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.