Skip to content

ci(nightly): authenticate controller admission check receipts - #30284

Merged
teamleaderleo merged 6 commits into
feat-cmux-nextfrom
lane/nightly-check-receipt-consumer
Nov 1, 2026
Merged

teamleaderleo merged 6 commits into
feat-cmux-nextfrom
lane/nightly-check-receipt-consumer

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

NEXT control jobs cannot authenticate the private controller from a runner outside its network. Consume its authenticated GitHub Check Run instead: require the existing publishing App, exact run, attempt, source, jobs and canonical lease identity, plus the controller's terminal decision and clocks. Missing or invalid publication fails placement; it never becomes a no-start receipt or a paid runner choice.

Leo: “FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT”.

CI_NIGHTLY_ADMISSION_CHECKS=1 opts ordinary NEXT builds in through a run/source-bound workflow title. The controller uses that marker and the actual current-attempt bootstrap job to admit demand before a runner starts. Dedicated-fleet jobs retain first priority and skip this reader. Unsigned-resume titles and routing remain compatible with the existing retry tracker. Downstream controls require a verified receipt from their own attempt, so a failed-jobs rerun cannot reuse stale placement and fall through to paid capacity.

The only permission addition is checks: read on the existing metadata job, needed to authenticate the Check Run's App identity. No App permission, credential, runner trust or release-environment change is made. Read-only permissions from #30248 are retained. Existing routes remain when the opt-in is disabled. Activation stays off until the controller demand and publisher changes are deployed.

The reader makes at most two authenticated GitHub requests, with one delayed retry only when publication is absent. Its 210-second publication allowance covers the controller's 180-second lease and delivery margin; the job timeout also bounds the operation. API calls have a 15-second timeout and 1 MiB response limit, and reject redirects. There is no polling loop, background timer, dispatch or cancellation. Existing warm sources and compile caches are unchanged; this adds no build.

Validation: the same focused missing-receipt regression fails on base f818cf72352 (it selects a fallback despite opt-in) and passes at this head. New receipt/workflow tests (18), legacy admission (26), trusted routing (37), dedicated routing (7), routing policy (33), nightly routing (20), fork routing (23) and promotion-queue cases (9) pass. The live routing-table check, diff validation and pre-push checks pass. Job-scoped permissions (5), reusable permissions, workflow-size (2) and source-archive policy checks also pass. Two identical license steps now use YAML anchors to retain workflow-size headroom; the parsed workflow is unchanged by that deduplication. Coverage includes wrong App, identity/attempt/source mismatch, ambiguous or truncated publication, malformed JSON, nanosecond clock boundaries, missing/stale downstream outputs, dedicated priority, and anchored unsigned-resume titles.

Observed / Not run

Observed: 173 focused tests and cases pass; the missing-receipt policy has a base/head regression. Not run: an enabled production nightly consuming the new publication. That proof follows controller deployment and explicit opt-in activation. Prior art: preserves #30085's identity-bound leases, #30173's disabled-mode fallback, #30249's dedicated priority and #30248's read-only metadata permissions.

Performance Before After Delta Evidence
GitHub admission reads per opted-in placement No public receipt reader At most two Bounded authenticated reads Reader tests
Blacksmith minutes/day Not measured Not measured Paid overflow requires controller no-start proof when enabled Routing tests; live activation pending

Changelog

  • Authenticate fleet admission receipts before routing opted-in NEXT nightly controls.

Agent session: agent session
🤖 Generated with Claude Code

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 1, 2026
…y-check-receipt-consumer

# Conflicts:
#	.github/workflows/nightly.yml
@cursor

cursor Bot commented Nov 1, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/ci/nightly_admission_receipt.py">

<violation number="1" location="scripts/ci/nightly_admission_receipt.py:54">
P1: `validate` accepts a well-formed lease ID for a different request, so the private controller path can route from a stale or cross-request receipt. Compare the receipt ID with `lease_id(request)` before using its runner map or overflow decision.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

if type(receipt.get(key)) is not type(request[key]) or receipt[key] != request[key]:
raise ValueError(f"admission receipt mismatches {key}")
lease = receipt.get("lease_id")
if not isinstance(lease, str) or not re.fullmatch(r"[0-9a-f]{24}", lease):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: validate accepts a well-formed lease ID for a different request, so the private controller path can route from a stale or cross-request receipt. Compare the receipt ID with lease_id(request) before using its runner map or overflow decision.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly_admission_receipt.py, line 54:

<comment>`validate` accepts a well-formed lease ID for a different request, so the private controller path can route from a stale or cross-request receipt. Compare the receipt ID with `lease_id(request)` before using its runner map or overflow decision.</comment>

<file context>
@@ -0,0 +1,169 @@
+        if type(receipt.get(key)) is not type(request[key]) or receipt[key] != request[key]:
+            raise ValueError(f"admission receipt mismatches {key}")
+    lease = receipt.get("lease_id")
+    if not isinstance(lease, str) or not re.fullmatch(r"[0-9a-f]{24}", lease):
+        raise ValueError("invalid lease identity")
+    created, expires = timestamp_ns(receipt.get("created_at")), timestamp_ns(receipt.get("expires_at"))
</file context>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_nightly_admission_receipt.py">

<violation number="1" location="tests/test_nightly_admission_receipt.py:144">
P2: This test does not verify that `get()` installs `NoRedirect`; removing that handler would still pass because the mocked opener accepts every request. Capture the factory mock and assert its first argument is a `NoRedirect` instance.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

def test_get_is_authenticated_bounded_and_cannot_follow_redirects(self):
opener = mock.Mock()
opener.open.return_value = io.BytesIO(json.dumps(payload(check())).encode())
with mock.patch.object(reader.urllib.request, "build_opener", return_value=opener):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This test does not verify that get() installs NoRedirect; removing that handler would still pass because the mocked opener accepts every request. Capture the factory mock and assert its first argument is a NoRedirect instance.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_admission_receipt.py, line 144:

<comment>This test does not verify that `get()` installs `NoRedirect`; removing that handler would still pass because the mocked opener accepts every request. Capture the factory mock and assert its first argument is a `NoRedirect` instance.</comment>

<file context>
@@ -0,0 +1,243 @@
+    def test_get_is_authenticated_bounded_and_cannot_follow_redirects(self):
+        opener = mock.Mock()
+        opener.open.return_value = io.BytesIO(json.dumps(payload(check())).encode())
+        with mock.patch.object(reader.urllib.request, "build_opener", return_value=opener):
+            reader.get(REQUEST, "token")
+        call = opener.open.call_args.args[0]
</file context>

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high): SHIP at 1155460. Independent subagent reviewed the exact clean merged head. Authenticated receipt authority, identity, schema, terminal clocks and bounded API reads pass; strict same-attempt gates, dedicated priority, legacy routes and unsigned resumes are preserved. Focused receipt, routing, permission, size and source-archive checks pass. No blocking findings.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_nightly_admission_receipt.py">

<violation number="1" location="tests/test_nightly_admission_receipt.py:1">
P2: This new test file is missing from `tests/test-execution.toml`, so the preflight registry validator fails the PR even though `ci-guards.yml` invokes it directly. Add a `linux-guard` registry entry for this file.</violation>
</file>

<file name=".github/workflows/ci-guards.yml">

<violation number="1" location=".github/workflows/ci-guards.yml:925">
P2: This invocation is unreachable: `workflow-guard-tests` excludes `group: ci`, while this step runs only for `matrix.group == 'ci'`. The new receipt regression therefore never runs; remove that exclusion for this lane or move the command to the job that owns the `ci` group.</violation>
</file>

<file name="scripts/ci/nightly_linux_admission.py">

<violation number="1" location="scripts/ci/nightly_linux_admission.py:27">
P2: This import replaces the legacy validator for opt-out controller receipts too, but the new validator rejects finite float `no_start_seconds` values that the existing path accepted. Keep the published-receipt validator separate, or preserve the legacy validation rules when `CHECK_RECEIPTS` is disabled.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@@ -0,0 +1,243 @@
#!/usr/bin/env python3

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This new test file is missing from tests/test-execution.toml, so the preflight registry validator fails the PR even though ci-guards.yml invokes it directly. Add a linux-guard registry entry for this file.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_admission_receipt.py, line 1:

<comment>This new test file is missing from `tests/test-execution.toml`, so the preflight registry validator fails the PR even though `ci-guards.yml` invokes it directly. Add a `linux-guard` registry entry for this file.</comment>

<file context>
@@ -0,0 +1,243 @@
+#!/usr/bin/env python3
+"""Only the controller App's exact terminal lease can route a nightly job."""
+import copy
</file context>

run: |
python3 tests/test_ci_portable_job_placement.py
python3 tests/test_nightly_linux_admission.py
python3 tests/test_nightly_admission_receipt.py

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This invocation is unreachable: workflow-guard-tests excludes group: ci, while this step runs only for matrix.group == 'ci'. The new receipt regression therefore never runs; remove that exclusion for this lane or move the command to the job that owns the ci group.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/ci-guards.yml, line 925:

<comment>This invocation is unreachable: `workflow-guard-tests` excludes `group: ci`, while this step runs only for `matrix.group == 'ci'`. The new receipt regression therefore never runs; remove that exclusion for this lane or move the command to the job that owns the `ci` group.</comment>

<file context>
@@ -922,6 +922,7 @@ jobs:
         run: |
           python3 tests/test_ci_portable_job_placement.py
           python3 tests/test_nightly_linux_admission.py
+          python3 tests/test_nightly_admission_receipt.py
           python3 tests/test_nightly_dedicated_mini.py
 
</file context>

import portable_job_placement as portable
from fleet_admission_transport import exchange
import nightly_admission_receipt as published
from nightly_admission_receipt import validate

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This import replaces the legacy validator for opt-out controller receipts too, but the new validator rejects finite float no_start_seconds values that the existing path accepted. Keep the published-receipt validator separate, or preserve the legacy validation rules when CHECK_RECEIPTS is disabled.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly_linux_admission.py, line 27:

<comment>This import replaces the legacy validator for opt-out controller receipts too, but the new validator rejects finite float `no_start_seconds` values that the existing path accepted. Keep the published-receipt validator separate, or preserve the legacy validation rules when `CHECK_RECEIPTS` is disabled.</comment>

<file context>
@@ -22,6 +23,8 @@
 import portable_job_placement as portable
 from fleet_admission_transport import exchange
+import nightly_admission_receipt as published
+from nightly_admission_receipt import validate
 
 FLEET_LINUX = "cmux-fleet-linux-x64"
</file context>

@blacksmith-sh

blacksmith-sh Bot commented Nov 1, 2026

Copy link
Copy Markdown

Found 4 test failures on Blacksmith runners:

Failures

Test View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/
test_download_prints_the_active_profile_only
View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/
Security has no stub
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/
CoreFoundation, missing 1: ['_CFRetain']
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/
CoreFoundation, missing 1: ['_log2']
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high): SHIP at 558dd1f. Independent subagent delta review confirms the only change since reviewed 1155460 registers the receipt policy test under linux-guard. Test execution registry validation, receipt 18, admission 26 and diff checks pass. No production behavior or activation changes; no blocking findings.

@teamleaderleo
teamleaderleo merged commit 4e81bfe into feat-cmux-next Nov 1, 2026
59 of 61 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-check-receipt-consumer branch November 1, 2026 06:38
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 558dd1f: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30284 -f ref=lane/nightly-check-receipt-consumer -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 783c672; does not block

Shipped in:

@lawrencecchen

Copy link
Copy Markdown
Contributor

Base red from this PR (seen on every feat-cmux-next PR in nightly-guards / workflow-guard-tests / release-notary). Two tests fail at the tip:

  1. python3 tests/test_nightly_ghostty_helper_cache.py: test_helper_never_takes_a_hosted_mac_and_skips_compile_on_hit requires resolve-nightly-ghostty-cli-helper.if == build-nightly-ghostty-cli-helper.if. This PR added the admission clause && (github.repository != 'manaflow-ai/cmux' || ... early_admission_ready == 'true') to the lookup job only.
  2. python3 tests/test_nightly_workflow_size.py: nightly.yml is 508,735 bytes. The test allows 508,000 (GitHub's limit is 512,000).

I did not fix these in #30624, because each one needs a decision from the owner:

  • Item 1: should the helper build job get the same admission clause, or should the test allow the lookup to be stricter?
  • Item 2: which repeated block should become a YAML anchor or a script to cut about 1 KB? The other alternative is to change the headroom.

#30624 fixes the other base reds.

@lawrencecchen

Copy link
Copy Markdown
Contributor

Follow-up to my comment above. The lead decided (12:06Z) not to wait. #30824 fixes both reds:

lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…ze, universal build pin, warm sources, promotion admission, highlight) (#30824)

* base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini

#30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly
mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them
as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other
nightly.yml Mac jobs that compile nothing, with an exemption that says what the
job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache)
takes linux-rust with actions-cache, as its cmux-tui-spec peers do.

Red at 9288e32: python3 tests/test_ci_warm_sources.py   FAILED (11 unlisted jobs)
Green here: same command, OK.

* base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate

The test pins the full runs-on and if: of the nightly helper jobs. Two
merged changes moved them:
- #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER)
  first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression
  now has that route; the merged workflow is the source of truth.
- #30284 (4e81bfe) added the early admission conjunct to the resolver and
  the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the
  resolver and the helper carry it as one conjunct after their gate, and that
  build-nightly-app keeps its own gate (it is skipped with the resolver it needs).
  The build_only exclusion is checked the same way as before.

Red at 9288e32: bash ./tests/test_nightly_universal_build.sh   FAIL: nightly must build and hand off the macOS 26 helper ...
Green with the nightly.yml commit after this one: same command, PASS.

* base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom

tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper
and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe)
added the early admission conjunct to the lookup on purpose; the helper build
now has it too, so it never builds while its lookup waits for admission.

tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the
508,000 headroom. The cache save steps repeated the same three R2 env values
six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT);
each is now an anchor on its first use and an alias on the others, as
&pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the
same workflow as before except the helper's if:. nightly.yml is 507,061 bytes.
No step is removed and no limit changes. (Job-level anchors are refused by
test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide
text that test_nightly_notices_submodule.py and
test_nightly_promotion_admission.py count, so the anchors are on step env.)

Red at 9288e32:
  python3 tests/test_nightly_ghostty_helper_cache.py   FAILED (failures=1)
  python3 tests/test_nightly_workflow_size.py          508735 not less than 508000
Green here: both OK. All 123 tests that read nightly.yml: the same results as
at 9288e32 except these and test_nightly_universal_build.sh, now green.

* base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job

#30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard
from main" with promote-nightly-next-acked through the YAML anchor
&legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in
the text and test_both_promote_decisions_pass_the_train_verifier (it counted
the text) failed. It now finds the two jobs that call guard.trainVerifier in
the parsed workflow and requires each to check out the verifier; it fails when
either job's checkout drops it (checked by removing it from the anchor).

Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py   AssertionError
Green here: PASS (75 cases).

* base red: release-notary, the Cloud environment editor highlight links its pull request

#30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md
with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed.
It links #30357, as #30624 did for three others.

Red at 265c68a: python3 scripts/whats-new/test_digest.py   FAILED (failures=1)
Green here: OK.
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
…ve to scripts/ci/nightly-steps/) (#30912)

* base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini

#30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly
mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them
as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other
nightly.yml Mac jobs that compile nothing, with an exemption that says what the
job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache)
takes linux-rust with actions-cache, as its cmux-tui-spec peers do.

Red at 9288e32: python3 tests/test_ci_warm_sources.py   FAILED (11 unlisted jobs)
Green here: same command, OK.

* base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate

The test pins the full runs-on and if: of the nightly helper jobs. Two
merged changes moved them:
- #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER)
  first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression
  now has that route; the merged workflow is the source of truth.
- #30284 (4e81bfe) added the early admission conjunct to the resolver and
  the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the
  resolver and the helper carry it as one conjunct after their gate, and that
  build-nightly-app keeps its own gate (it is skipped with the resolver it needs).
  The build_only exclusion is checked the same way as before.

Red at 9288e32: bash ./tests/test_nightly_universal_build.sh   FAIL: nightly must build and hand off the macOS 26 helper ...
Green with the nightly.yml commit after this one: same command, PASS.

* base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom

tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper
and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe)
added the early admission conjunct to the lookup on purpose; the helper build
now has it too, so it never builds while its lookup waits for admission.

tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the
508,000 headroom. The cache save steps repeated the same three R2 env values
six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT);
each is now an anchor on its first use and an alias on the others, as
&pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the
same workflow as before except the helper's if:. nightly.yml is 507,061 bytes.
No step is removed and no limit changes. (Job-level anchors are refused by
test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide
text that test_nightly_notices_submodule.py and
test_nightly_promotion_admission.py count, so the anchors are on step env.)

Red at 9288e32:
  python3 tests/test_nightly_ghostty_helper_cache.py   FAILED (failures=1)
  python3 tests/test_nightly_workflow_size.py          508735 not less than 508000
Green here: both OK. All 123 tests that read nightly.yml: the same results as
at 9288e32 except these and test_nightly_universal_build.sh, now green.

* base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job

#30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard
from main" with promote-nightly-next-acked through the YAML anchor
&legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in
the text and test_both_promote_decisions_pass_the_train_verifier (it counted
the text) failed. It now finds the two jobs that call guard.trainVerifier in
the parsed workflow and requires each to check out the verifier; it fails when
either job's checkout drops it (checked by removing it from the anchor).

Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py   AssertionError
Green here: PASS (75 cases).

* base red: release-notary, the Cloud environment editor highlight links its pull request

#30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md
with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed.
It links #30357, as #30624 did for three others.

Red at 265c68a: python3 scripts/whats-new/test_digest.py   FAILED (failures=1)
Green here: OK.

* base red: preflight, nightly.yml back under its 490,000-byte budget: 30 run blocks move to scripts/ci/nightly-steps/

tests/test_workflow_file_size.py (ci-guards preflight) caps nightly.yml at 490,000 bytes;
it was 505,604 after #30824 (508,735 before it), red on every pull request. Lead
decision 17:37Z: move run blocks, as #25879 did.

30 `run: |` blocks move verbatim into scripts/ci/nightly-steps/<name>.sh; each step now
runs `bash scripts/ci/nightly-steps/<name>.sh` with the same env:, if:, id:, shell and
working directory. yaml.safe_load gives the same jobs and steps as before except those
30 run strings, and tests/nightly_workflow_text.py puts the scripts back inline
byte-identical to the old workflow. nightly.yml goes from 505,604 to 486,014 bytes.

Which blocks:
- Every block uses no ${{ }} expression and starts with `set -euo pipefail`, so the
  script sets the same shell options the inline block did (bash scripts/x.sh does not
  inherit GitHub's `bash -e`).
- No block has a working-directory, runs inside a checkout it performs, or is a
  security gate whose text a test pins ("Hold the whole Mac before any credential").
- Each step's script is in the tree the step runs on: steps in refresh-compilation-cache,
  refresh-test-compilation-cache and resolve-nightly-cmux-tui-client check out the
  workflow commit (decide's head_sha is context.sha); steps in build-nightly-app,
  build-nightly-ghostty-cli-helper and build-nightly-remote-daemon check out build_sha
  and are skipped by an unsigned resume, the run that builds an older saved tree.

Tests that read the raw nightly.yml text for a moved block now read it through
tests/nightly_workflow_text.py (expand), as #25879 set up; what they check is unchanged.
The new scripts pass shellcheck -S warning (the level actionlint gives shellcheck), and
actionlint reports the same 5 findings on nightly.yml as before (unused anchors).

Red at 6238504 (#30824 head): python3 tests/test_workflow_file_size.py   505604 not less than 490000
Green here: OK. All 124 tests that read nightly.yml: the same results as at 6238504
except test_workflow_file_size.py, now green.

---------

Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
lawrencecchen added a commit that referenced this pull request Nov 1, 2026
… notes into scripts, expressions through env) (#31013)

* base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini

#30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly
mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them
as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other
nightly.yml Mac jobs that compile nothing, with an exemption that says what the
job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache)
takes linux-rust with actions-cache, as its cmux-tui-spec peers do.

Red at 9288e32: python3 tests/test_ci_warm_sources.py   FAILED (11 unlisted jobs)
Green here: same command, OK.

* base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate

The test pins the full runs-on and if: of the nightly helper jobs. Two
merged changes moved them:
- #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER)
  first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression
  now has that route; the merged workflow is the source of truth.
- #30284 (4e81bfe) added the early admission conjunct to the resolver and
  the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the
  resolver and the helper carry it as one conjunct after their gate, and that
  build-nightly-app keeps its own gate (it is skipped with the resolver it needs).
  The build_only exclusion is checked the same way as before.

Red at 9288e32: bash ./tests/test_nightly_universal_build.sh   FAIL: nightly must build and hand off the macOS 26 helper ...
Green with the nightly.yml commit after this one: same command, PASS.

* base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom

tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper
and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe)
added the early admission conjunct to the lookup on purpose; the helper build
now has it too, so it never builds while its lookup waits for admission.

tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the
508,000 headroom. The cache save steps repeated the same three R2 env values
six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT);
each is now an anchor on its first use and an alias on the others, as
&pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the
same workflow as before except the helper's if:. nightly.yml is 507,061 bytes.
No step is removed and no limit changes. (Job-level anchors are refused by
test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide
text that test_nightly_notices_submodule.py and
test_nightly_promotion_admission.py count, so the anchors are on step env.)

Red at 9288e32:
  python3 tests/test_nightly_ghostty_helper_cache.py   FAILED (failures=1)
  python3 tests/test_nightly_workflow_size.py          508735 not less than 508000
Green here: both OK. All 123 tests that read nightly.yml: the same results as
at 9288e32 except these and test_nightly_universal_build.sh, now green.

* base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job

#30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard
from main" with promote-nightly-next-acked through the YAML anchor
&legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in
the text and test_both_promote_decisions_pass_the_train_verifier (it counted
the text) failed. It now finds the two jobs that call guard.trainVerifier in
the parsed workflow and requires each to check out the verifier; it fails when
either job's checkout drops it (checked by removing it from the anchor).

Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py   AssertionError
Green here: PASS (75 cases).

* base red: release-notary, the Cloud environment editor highlight links its pull request

#30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md
with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed.
It links #30357, as #30624 did for three others.

Red at 265c68a: python3 scripts/whats-new/test_digest.py   FAILED (failures=1)
Green here: OK.

* base red: preflight, nightly.yml back under its 490,000-byte budget: 30 run blocks move to scripts/ci/nightly-steps/

tests/test_workflow_file_size.py (ci-guards preflight) caps nightly.yml at 490,000 bytes;
it was 505,604 after #30824 (508,735 before it), red on every pull request. Lead
decision 17:37Z: move run blocks, as #25879 did.

30 `run: |` blocks move verbatim into scripts/ci/nightly-steps/<name>.sh; each step now
runs `bash scripts/ci/nightly-steps/<name>.sh` with the same env:, if:, id:, shell and
working directory. yaml.safe_load gives the same jobs and steps as before except those
30 run strings, and tests/nightly_workflow_text.py puts the scripts back inline
byte-identical to the old workflow. nightly.yml goes from 505,604 to 486,014 bytes.

Which blocks:
- Every block uses no ${{ }} expression and starts with `set -euo pipefail`, so the
  script sets the same shell options the inline block did (bash scripts/x.sh does not
  inherit GitHub's `bash -e`).
- No block has a working-directory, runs inside a checkout it performs, or is a
  security gate whose text a test pins ("Hold the whole Mac before any credential").
- Each step's script is in the tree the step runs on: steps in refresh-compilation-cache,
  refresh-test-compilation-cache and resolve-nightly-cmux-tui-client check out the
  workflow commit (decide's head_sha is context.sha); steps in build-nightly-app,
  build-nightly-ghostty-cli-helper and build-nightly-remote-daemon check out build_sha
  and are skipped by an unsigned resume, the run that builds an older saved tree.

Tests that read the raw nightly.yml text for a moved block now read it through
tests/nightly_workflow_text.py (expand), as #25879 set up; what they check is unchanged.
The new scripts pass shellcheck -S warning (the level actionlint gives shellcheck), and
actionlint reports the same 5 findings on nightly.yml as before (unused anchors).

Red at 6238504 (#30824 head): python3 tests/test_workflow_file_size.py   505604 not less than 490000
Green here: OK. All 124 tests that read nightly.yml: the same results as at 6238504
except test_workflow_file_size.py, now green.

* ci(nightly): 14 more run blocks and the step notes move to scripts/ci/nightly-steps/; expressions reach scripts through env (headroom under the size budget)

After #30912 nightly.yml had 3,986 bytes left under tests/test_workflow_file_size.py's
490,000-byte cap, so the next nightly.yml change would turn preflight red on every
pull request again. Lead decision 13:2xZ: (a) the remaining safe blocks, (b) step
notes into their script's header, (c) the `${{ }}` blocks with each expression
passed through env. nightly.yml goes from 486,014 to 457,877 bytes (32,123 bytes of room).

(a) 7 blocks with no ${{ }} move verbatim as in #30912. Four of them (Fetch the
Chromium engine, Upload dSYMs to Sentry, Finish the dSYM upload to Sentry, Report a
missing daemon attestation) do not start with `set -euo pipefail`; they ran under
GitHub's default `bash -e {0}`, so they are called with `bash -e`, and
tests/nightly_workflow_text.py accepts that call form.

(c) 7 blocks used ${{ }} expressions. Each expression now sits in a new `env:` entry of
the same step and the script reads that variable in the same place, GitHub's pattern
against script injection. Each name is new to the workflow, read only by its script,
and never assigned there. `--allow-non-descendant`, which was spliced in unquoted
when non-empty and vanished when empty, is `${ALLOW_NON_DESCENDANT_FLAG:+"$ALLOW_NON_DESCENDANT_FLAG"}`:
one word or none, as before. Republish deltas and revised appcasts stays inline:
tests/test_nightly_republish_delta_guard.py runs its exact text, and its job is a
resume job.

(b) The comment lines above and inside 46 script-backed steps (only those whose script
is called once) move into the header of their script, under "The step's notes,
moved from nightly.yml". Comments of steps that keep inline code stay. yaml.safe_load
of nightly.yml is unchanged by this part.

Not moved: the run blocks of the 7 jobs an unsigned resume runs over an older
build_sha (build-sign-notarize-nightly, prepare-nightly-variant, prepare-nightly-variant-primary,
verify-signed-nightly, sign-release-notes, published-in, publish-nightly-reel). A resume
of a source compiled before this change would not find a new script. They move 8 days
after #30912 merged, when no unsigned copy of a pre-merge commit is left.

Tests: 8 tests that read the inline text of a moved block read the expanded view
(tests/nightly_workflow_text.py) or copy the script into their scratch tree;
test_nightly_trusted_routing and test_ci_fleet_compilation_cache check the expression
in the step's env and the variable in its script. What they check is unchanged.

Red at the size budget: none at #30912's head (486,014 < 490,000); the gain is headroom.
All 124 tests that read nightly.yml: the same results as at 17bf155 (all green).
shellcheck -S warning: clean on scripts/ci/nightly-steps/*.sh. actionlint 1.7.12-manaflow.1:
the same 5 findings on nightly.yml as before (unused anchors).

* ci(nightly): Generate deltas and revised appcast stays inline (its .publish-tooling checkout lists single files)

Review of #31013 at 8a2494b: generate-nightly-deltas checks out .publish-tooling
with a non-cone sparse list of five named files (anchor &checkout-the-gh-token-wrapper),
so scripts/ci/nightly-steps/deltas-generate-appcast.sh would not be in it and the step
would fail on every run, hidden by the job's continue-on-error. My selection matched
the directory name as a substring of the list. The step goes back exactly as on the
feat tip (block and its notes), the script is removed, and test_nightly_sparkle_key.sh
is back to the feat tip's text.

An exact check of all 140 step-script calls (the last checkout before each call that
provides its path, root or .publish-tooling, must contain the file: no sparse set,
a cone-mode directory or a non-cone pattern) finds no gap at this head; it found
exactly this one before the fix, and none in the feat tip's 127 calls.

nightly.yml: 458,313 bytes (31,687 under 490,000). All 124 tests that read nightly.yml:
the same results as at the feat tip 49aa291. shellcheck -S warning clean; actionlint:
the same 5 findings as before.

---------

Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 558dd1f8 Deployed Nov 1, 2026 by teamleaderleo via wait for same-key tree publisher #94310
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants