Repository navigation
ci(nightly): authenticate controller admission check receipts - #30284
Conversation
|
Owned by an agent session · session |
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
…y-check-receipt-consumer # Conflicts: # .github/workflows/nightly.yml
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
There was a problem hiding this comment.
1 issue found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="scripts/ci/nightly_admission_receipt.py">
<violation number="1" location="scripts/ci/nightly_admission_receipt.py:54">
P1: `validate` accepts a well-formed lease ID for a different request, so the private controller path can route from a stale or cross-request receipt. Compare the receipt ID with `lease_id(request)` before using its runner map or overflow decision.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| if type(receipt.get(key)) is not type(request[key]) or receipt[key] != request[key]: | ||
| raise ValueError(f"admission receipt mismatches {key}") | ||
| lease = receipt.get("lease_id") | ||
| if not isinstance(lease, str) or not re.fullmatch(r"[0-9a-f]{24}", lease): |
There was a problem hiding this comment.
P1: validate accepts a well-formed lease ID for a different request, so the private controller path can route from a stale or cross-request receipt. Compare the receipt ID with lease_id(request) before using its runner map or overflow decision.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly_admission_receipt.py, line 54:
<comment>`validate` accepts a well-formed lease ID for a different request, so the private controller path can route from a stale or cross-request receipt. Compare the receipt ID with `lease_id(request)` before using its runner map or overflow decision.</comment>
<file context>
@@ -0,0 +1,169 @@
+ if type(receipt.get(key)) is not type(request[key]) or receipt[key] != request[key]:
+ raise ValueError(f"admission receipt mismatches {key}")
+ lease = receipt.get("lease_id")
+ if not isinstance(lease, str) or not re.fullmatch(r"[0-9a-f]{24}", lease):
+ raise ValueError("invalid lease identity")
+ created, expires = timestamp_ns(receipt.get("created_at")), timestamp_ns(receipt.get("expires_at"))
</file context>
There was a problem hiding this comment.
1 issue found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test_nightly_admission_receipt.py">
<violation number="1" location="tests/test_nightly_admission_receipt.py:144">
P2: This test does not verify that `get()` installs `NoRedirect`; removing that handler would still pass because the mocked opener accepts every request. Capture the factory mock and assert its first argument is a `NoRedirect` instance.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| def test_get_is_authenticated_bounded_and_cannot_follow_redirects(self): | ||
| opener = mock.Mock() | ||
| opener.open.return_value = io.BytesIO(json.dumps(payload(check())).encode()) | ||
| with mock.patch.object(reader.urllib.request, "build_opener", return_value=opener): |
There was a problem hiding this comment.
P2: This test does not verify that get() installs NoRedirect; removing that handler would still pass because the mocked opener accepts every request. Capture the factory mock and assert its first argument is a NoRedirect instance.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_admission_receipt.py, line 144:
<comment>This test does not verify that `get()` installs `NoRedirect`; removing that handler would still pass because the mocked opener accepts every request. Capture the factory mock and assert its first argument is a `NoRedirect` instance.</comment>
<file context>
@@ -0,0 +1,243 @@
+ def test_get_is_authenticated_bounded_and_cannot_follow_redirects(self):
+ opener = mock.Mock()
+ opener.open.return_value = io.BytesIO(json.dumps(payload(check())).encode())
+ with mock.patch.object(reader.urllib.request, "build_opener", return_value=opener):
+ reader.get(REQUEST, "token")
+ call = opener.open.call_args.args[0]
</file context>
|
Review (codex gpt-6-astra high): SHIP at 1155460. Independent subagent reviewed the exact clean merged head. Authenticated receipt authority, identity, schema, terminal clocks and bounded API reads pass; strict same-attempt gates, dedicated priority, legacy routes and unsigned resumes are preserved. Focused receipt, routing, permission, size and source-archive checks pass. No blocking findings. |
There was a problem hiding this comment.
3 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test_nightly_admission_receipt.py">
<violation number="1" location="tests/test_nightly_admission_receipt.py:1">
P2: This new test file is missing from `tests/test-execution.toml`, so the preflight registry validator fails the PR even though `ci-guards.yml` invokes it directly. Add a `linux-guard` registry entry for this file.</violation>
</file>
<file name=".github/workflows/ci-guards.yml">
<violation number="1" location=".github/workflows/ci-guards.yml:925">
P2: This invocation is unreachable: `workflow-guard-tests` excludes `group: ci`, while this step runs only for `matrix.group == 'ci'`. The new receipt regression therefore never runs; remove that exclusion for this lane or move the command to the job that owns the `ci` group.</violation>
</file>
<file name="scripts/ci/nightly_linux_admission.py">
<violation number="1" location="scripts/ci/nightly_linux_admission.py:27">
P2: This import replaces the legacy validator for opt-out controller receipts too, but the new validator rejects finite float `no_start_seconds` values that the existing path accepted. Keep the published-receipt validator separate, or preserve the legacy validation rules when `CHECK_RECEIPTS` is disabled.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| @@ -0,0 +1,243 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
P2: This new test file is missing from tests/test-execution.toml, so the preflight registry validator fails the PR even though ci-guards.yml invokes it directly. Add a linux-guard registry entry for this file.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_admission_receipt.py, line 1:
<comment>This new test file is missing from `tests/test-execution.toml`, so the preflight registry validator fails the PR even though `ci-guards.yml` invokes it directly. Add a `linux-guard` registry entry for this file.</comment>
<file context>
@@ -0,0 +1,243 @@
+#!/usr/bin/env python3
+"""Only the controller App's exact terminal lease can route a nightly job."""
+import copy
</file context>
| run: | | ||
| python3 tests/test_ci_portable_job_placement.py | ||
| python3 tests/test_nightly_linux_admission.py | ||
| python3 tests/test_nightly_admission_receipt.py |
There was a problem hiding this comment.
P2: This invocation is unreachable: workflow-guard-tests excludes group: ci, while this step runs only for matrix.group == 'ci'. The new receipt regression therefore never runs; remove that exclusion for this lane or move the command to the job that owns the ci group.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/ci-guards.yml, line 925:
<comment>This invocation is unreachable: `workflow-guard-tests` excludes `group: ci`, while this step runs only for `matrix.group == 'ci'`. The new receipt regression therefore never runs; remove that exclusion for this lane or move the command to the job that owns the `ci` group.</comment>
<file context>
@@ -922,6 +922,7 @@ jobs:
run: |
python3 tests/test_ci_portable_job_placement.py
python3 tests/test_nightly_linux_admission.py
+ python3 tests/test_nightly_admission_receipt.py
python3 tests/test_nightly_dedicated_mini.py
</file context>
| import portable_job_placement as portable | ||
| from fleet_admission_transport import exchange | ||
| import nightly_admission_receipt as published | ||
| from nightly_admission_receipt import validate |
There was a problem hiding this comment.
P2: This import replaces the legacy validator for opt-out controller receipts too, but the new validator rejects finite float no_start_seconds values that the existing path accepted. Keep the published-receipt validator separate, or preserve the legacy validation rules when CHECK_RECEIPTS is disabled.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/ci/nightly_linux_admission.py, line 27:
<comment>This import replaces the legacy validator for opt-out controller receipts too, but the new validator rejects finite float `no_start_seconds` values that the existing path accepted. Keep the published-receipt validator separate, or preserve the legacy validation rules when `CHECK_RECEIPTS` is disabled.</comment>
<file context>
@@ -22,6 +23,8 @@
import portable_job_placement as portable
from fleet_admission_transport import exchange
+import nightly_admission_receipt as published
+from nightly_admission_receipt import validate
FLEET_LINUX = "cmux-fleet-linux-x64"
</file context>
|
Found 4 test failures on Blacksmith runners: Failures
|
|
Review (codex gpt-6-astra high): SHIP at 558dd1f. Independent subagent delta review confirms the only change since reviewed 1155460 registers the receipt policy test under linux-guard. Test execution registry validation, receipt 18, admission 26 and diff checks pass. No production behavior or activation changes; no blocking findings. |
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 783c672; does not block Shipped in:
|
|
Base red from this PR (seen on every feat-cmux-next PR in
I did not fix these in #30624, because each one needs a decision from the owner:
#30624 fixes the other base reds. |
|
Follow-up to my comment above. The lead decided (12:06Z) not to wait. #30824 fixes both reds:
|
…ze, universal build pin, warm sources, promotion admission, highlight) (#30824) * base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini #30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other nightly.yml Mac jobs that compile nothing, with an exemption that says what the job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache) takes linux-rust with actions-cache, as its cmux-tui-spec peers do. Red at 9288e32: python3 tests/test_ci_warm_sources.py FAILED (11 unlisted jobs) Green here: same command, OK. * base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate The test pins the full runs-on and if: of the nightly helper jobs. Two merged changes moved them: - #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER) first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression now has that route; the merged workflow is the source of truth. - #30284 (4e81bfe) added the early admission conjunct to the resolver and the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the resolver and the helper carry it as one conjunct after their gate, and that build-nightly-app keeps its own gate (it is skipped with the resolver it needs). The build_only exclusion is checked the same way as before. Red at 9288e32: bash ./tests/test_nightly_universal_build.sh FAIL: nightly must build and hand off the macOS 26 helper ... Green with the nightly.yml commit after this one: same command, PASS. * base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe) added the early admission conjunct to the lookup on purpose; the helper build now has it too, so it never builds while its lookup waits for admission. tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the 508,000 headroom. The cache save steps repeated the same three R2 env values six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT); each is now an anchor on its first use and an alias on the others, as &pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the same workflow as before except the helper's if:. nightly.yml is 507,061 bytes. No step is removed and no limit changes. (Job-level anchors are refused by test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide text that test_nightly_notices_submodule.py and test_nightly_promotion_admission.py count, so the anchors are on step env.) Red at 9288e32: python3 tests/test_nightly_ghostty_helper_cache.py FAILED (failures=1) python3 tests/test_nightly_workflow_size.py 508735 not less than 508000 Green here: both OK. All 123 tests that read nightly.yml: the same results as at 9288e32 except these and test_nightly_universal_build.sh, now green. * base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job #30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard from main" with promote-nightly-next-acked through the YAML anchor &legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in the text and test_both_promote_decisions_pass_the_train_verifier (it counted the text) failed. It now finds the two jobs that call guard.trainVerifier in the parsed workflow and requires each to check out the verifier; it fails when either job's checkout drops it (checked by removing it from the anchor). Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py AssertionError Green here: PASS (75 cases). * base red: release-notary, the Cloud environment editor highlight links its pull request #30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed. It links #30357, as #30624 did for three others. Red at 265c68a: python3 scripts/whats-new/test_digest.py FAILED (failures=1) Green here: OK.
…ve to scripts/ci/nightly-steps/) (#30912) * base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini #30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other nightly.yml Mac jobs that compile nothing, with an exemption that says what the job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache) takes linux-rust with actions-cache, as its cmux-tui-spec peers do. Red at 9288e32: python3 tests/test_ci_warm_sources.py FAILED (11 unlisted jobs) Green here: same command, OK. * base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate The test pins the full runs-on and if: of the nightly helper jobs. Two merged changes moved them: - #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER) first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression now has that route; the merged workflow is the source of truth. - #30284 (4e81bfe) added the early admission conjunct to the resolver and the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the resolver and the helper carry it as one conjunct after their gate, and that build-nightly-app keeps its own gate (it is skipped with the resolver it needs). The build_only exclusion is checked the same way as before. Red at 9288e32: bash ./tests/test_nightly_universal_build.sh FAIL: nightly must build and hand off the macOS 26 helper ... Green with the nightly.yml commit after this one: same command, PASS. * base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe) added the early admission conjunct to the lookup on purpose; the helper build now has it too, so it never builds while its lookup waits for admission. tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the 508,000 headroom. The cache save steps repeated the same three R2 env values six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT); each is now an anchor on its first use and an alias on the others, as &pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the same workflow as before except the helper's if:. nightly.yml is 507,061 bytes. No step is removed and no limit changes. (Job-level anchors are refused by test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide text that test_nightly_notices_submodule.py and test_nightly_promotion_admission.py count, so the anchors are on step env.) Red at 9288e32: python3 tests/test_nightly_ghostty_helper_cache.py FAILED (failures=1) python3 tests/test_nightly_workflow_size.py 508735 not less than 508000 Green here: both OK. All 123 tests that read nightly.yml: the same results as at 9288e32 except these and test_nightly_universal_build.sh, now green. * base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job #30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard from main" with promote-nightly-next-acked through the YAML anchor &legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in the text and test_both_promote_decisions_pass_the_train_verifier (it counted the text) failed. It now finds the two jobs that call guard.trainVerifier in the parsed workflow and requires each to check out the verifier; it fails when either job's checkout drops it (checked by removing it from the anchor). Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py AssertionError Green here: PASS (75 cases). * base red: release-notary, the Cloud environment editor highlight links its pull request #30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed. It links #30357, as #30624 did for three others. Red at 265c68a: python3 scripts/whats-new/test_digest.py FAILED (failures=1) Green here: OK. * base red: preflight, nightly.yml back under its 490,000-byte budget: 30 run blocks move to scripts/ci/nightly-steps/ tests/test_workflow_file_size.py (ci-guards preflight) caps nightly.yml at 490,000 bytes; it was 505,604 after #30824 (508,735 before it), red on every pull request. Lead decision 17:37Z: move run blocks, as #25879 did. 30 `run: |` blocks move verbatim into scripts/ci/nightly-steps/<name>.sh; each step now runs `bash scripts/ci/nightly-steps/<name>.sh` with the same env:, if:, id:, shell and working directory. yaml.safe_load gives the same jobs and steps as before except those 30 run strings, and tests/nightly_workflow_text.py puts the scripts back inline byte-identical to the old workflow. nightly.yml goes from 505,604 to 486,014 bytes. Which blocks: - Every block uses no ${{ }} expression and starts with `set -euo pipefail`, so the script sets the same shell options the inline block did (bash scripts/x.sh does not inherit GitHub's `bash -e`). - No block has a working-directory, runs inside a checkout it performs, or is a security gate whose text a test pins ("Hold the whole Mac before any credential"). - Each step's script is in the tree the step runs on: steps in refresh-compilation-cache, refresh-test-compilation-cache and resolve-nightly-cmux-tui-client check out the workflow commit (decide's head_sha is context.sha); steps in build-nightly-app, build-nightly-ghostty-cli-helper and build-nightly-remote-daemon check out build_sha and are skipped by an unsigned resume, the run that builds an older saved tree. Tests that read the raw nightly.yml text for a moved block now read it through tests/nightly_workflow_text.py (expand), as #25879 set up; what they check is unchanged. The new scripts pass shellcheck -S warning (the level actionlint gives shellcheck), and actionlint reports the same 5 findings on nightly.yml as before (unused anchors). Red at 6238504 (#30824 head): python3 tests/test_workflow_file_size.py 505604 not less than 490000 Green here: OK. All 124 tests that read nightly.yml: the same results as at 6238504 except test_workflow_file_size.py, now green. --------- Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
… notes into scripts, expressions through env) (#31013) * base red: release-notary, warm-sources entries for the nightly controls #30249 routed to the dedicated mini #30249 (d8f537d) let ten nightly.yml control jobs take the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER), so tests/test_ci_warm_sources.py counts them as Mac jobs, and they had no entry. Each takes mac-sign, the class of the other nightly.yml Mac jobs that compile nothing, with an exemption that says what the job does. cmux-tui-spec.yml:clippy (also unlisted; it wires Swatinem/rust-cache) takes linux-rust with actions-cache, as its cmux-tui-spec peers do. Red at 9288e32: python3 tests/test_ci_warm_sources.py FAILED (11 unlisted jobs) Green here: same command, OK. * base red: release-notary, test_nightly_universal_build follows the merged helper route and admission gate The test pins the full runs-on and if: of the nightly helper jobs. Two merged changes moved them: - #30249 (d8f537d) put the dedicated nightly mini (vars.CI_NIGHTLY_ONLY_RUNNER) first in build-nightly-ghostty-cli-helper's runs-on. The pinned expression now has that route; the merged workflow is the source of truth. - #30284 (4e81bfe) added the early admission conjunct to the resolver and the helper lookup. The test now names it (EARLY_ADMISSION) and checks that the resolver and the helper carry it as one conjunct after their gate, and that build-nightly-app keeps its own gate (it is skipped with the resolver it needs). The build_only exclusion is checked the same way as before. Red at 9288e32: bash ./tests/test_nightly_universal_build.sh FAIL: nightly must build and hand off the macOS 26 helper ... Green with the nightly.yml commit after this one: same command, PASS. * base red: release-notary, the Ghostty helper build takes its lookup's admission gate; nightly.yml back under its headroom tests/test_nightly_ghostty_helper_cache.py requires build-nightly-ghostty-cli-helper and resolve-nightly-ghostty-cli-helper to share one if:. #30284 (4e81bfe) added the early admission conjunct to the lookup on purpose; the helper build now has it too, so it never builds while its lookup waits for admission. tests/test_nightly_workflow_size.py: nightly.yml was 508,735 bytes, over the 508,000 headroom. The cache save steps repeated the same three R2 env values six times each (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, CI_CACHE_R2_ENDPOINT); each is now an anchor on its first use and an alias on the others, as &pub-feeds, &pub-dmg and &pub-tooling already are. yaml.safe_load gives the same workflow as before except the helper's if:. nightly.yml is 507,061 bytes. No step is removed and no limit changes. (Job-level anchors are refused by test_nightly_signer_jobs.py and runner_routing_table.py, and step aliases hide text that test_nightly_notices_submodule.py and test_nightly_promotion_admission.py count, so the anchors are on step env.) Red at 9288e32: python3 tests/test_nightly_ghostty_helper_cache.py FAILED (failures=1) python3 tests/test_nightly_workflow_size.py 508735 not less than 508000 Green here: both OK. All 123 tests that read nightly.yml: the same results as at 9288e32 except these and test_nightly_universal_build.sh, now green. * base red: release-notary, the promotion admission test reads the shared promote-guard checkout per job #30471 (e27f116) shares promote-nightly-next's "Checkout the promote guard from main" with promote-nightly-next-acked through the YAML anchor &legacy-route-shared-4, so `scripts/ci/nightly_next_train.py` appears once in the text and test_both_promote_decisions_pass_the_train_verifier (it counted the text) failed. It now finds the two jobs that call guard.trainVerifier in the parsed workflow and requires each to check out the verifier; it fails when either job's checkout drops it (checked by removing it from the anchor). Red at origin/feat-cmux-next 265c68a: python3 tests/test_nightly_promotion_admission.py AssertionError Green here: PASS (75 cases). * base red: release-notary, the Cloud environment editor highlight links its pull request #30357 (7d5f3ae) added release-notes/next/highlights/cloud-environment-editor.md with no try-it action, docs or pr link, so scripts/whats-new/test_digest.py failed. It links #30357, as #30624 did for three others. Red at 265c68a: python3 scripts/whats-new/test_digest.py FAILED (failures=1) Green here: OK. * base red: preflight, nightly.yml back under its 490,000-byte budget: 30 run blocks move to scripts/ci/nightly-steps/ tests/test_workflow_file_size.py (ci-guards preflight) caps nightly.yml at 490,000 bytes; it was 505,604 after #30824 (508,735 before it), red on every pull request. Lead decision 17:37Z: move run blocks, as #25879 did. 30 `run: |` blocks move verbatim into scripts/ci/nightly-steps/<name>.sh; each step now runs `bash scripts/ci/nightly-steps/<name>.sh` with the same env:, if:, id:, shell and working directory. yaml.safe_load gives the same jobs and steps as before except those 30 run strings, and tests/nightly_workflow_text.py puts the scripts back inline byte-identical to the old workflow. nightly.yml goes from 505,604 to 486,014 bytes. Which blocks: - Every block uses no ${{ }} expression and starts with `set -euo pipefail`, so the script sets the same shell options the inline block did (bash scripts/x.sh does not inherit GitHub's `bash -e`). - No block has a working-directory, runs inside a checkout it performs, or is a security gate whose text a test pins ("Hold the whole Mac before any credential"). - Each step's script is in the tree the step runs on: steps in refresh-compilation-cache, refresh-test-compilation-cache and resolve-nightly-cmux-tui-client check out the workflow commit (decide's head_sha is context.sha); steps in build-nightly-app, build-nightly-ghostty-cli-helper and build-nightly-remote-daemon check out build_sha and are skipped by an unsigned resume, the run that builds an older saved tree. Tests that read the raw nightly.yml text for a moved block now read it through tests/nightly_workflow_text.py (expand), as #25879 set up; what they check is unchanged. The new scripts pass shellcheck -S warning (the level actionlint gives shellcheck), and actionlint reports the same 5 findings on nightly.yml as before (unused anchors). Red at 6238504 (#30824 head): python3 tests/test_workflow_file_size.py 505604 not less than 490000 Green here: OK. All 124 tests that read nightly.yml: the same results as at 6238504 except test_workflow_file_size.py, now green. * ci(nightly): 14 more run blocks and the step notes move to scripts/ci/nightly-steps/; expressions reach scripts through env (headroom under the size budget) After #30912 nightly.yml had 3,986 bytes left under tests/test_workflow_file_size.py's 490,000-byte cap, so the next nightly.yml change would turn preflight red on every pull request again. Lead decision 13:2xZ: (a) the remaining safe blocks, (b) step notes into their script's header, (c) the `${{ }}` blocks with each expression passed through env. nightly.yml goes from 486,014 to 457,877 bytes (32,123 bytes of room). (a) 7 blocks with no ${{ }} move verbatim as in #30912. Four of them (Fetch the Chromium engine, Upload dSYMs to Sentry, Finish the dSYM upload to Sentry, Report a missing daemon attestation) do not start with `set -euo pipefail`; they ran under GitHub's default `bash -e {0}`, so they are called with `bash -e`, and tests/nightly_workflow_text.py accepts that call form. (c) 7 blocks used ${{ }} expressions. Each expression now sits in a new `env:` entry of the same step and the script reads that variable in the same place, GitHub's pattern against script injection. Each name is new to the workflow, read only by its script, and never assigned there. `--allow-non-descendant`, which was spliced in unquoted when non-empty and vanished when empty, is `${ALLOW_NON_DESCENDANT_FLAG:+"$ALLOW_NON_DESCENDANT_FLAG"}`: one word or none, as before. Republish deltas and revised appcasts stays inline: tests/test_nightly_republish_delta_guard.py runs its exact text, and its job is a resume job. (b) The comment lines above and inside 46 script-backed steps (only those whose script is called once) move into the header of their script, under "The step's notes, moved from nightly.yml". Comments of steps that keep inline code stay. yaml.safe_load of nightly.yml is unchanged by this part. Not moved: the run blocks of the 7 jobs an unsigned resume runs over an older build_sha (build-sign-notarize-nightly, prepare-nightly-variant, prepare-nightly-variant-primary, verify-signed-nightly, sign-release-notes, published-in, publish-nightly-reel). A resume of a source compiled before this change would not find a new script. They move 8 days after #30912 merged, when no unsigned copy of a pre-merge commit is left. Tests: 8 tests that read the inline text of a moved block read the expanded view (tests/nightly_workflow_text.py) or copy the script into their scratch tree; test_nightly_trusted_routing and test_ci_fleet_compilation_cache check the expression in the step's env and the variable in its script. What they check is unchanged. Red at the size budget: none at #30912's head (486,014 < 490,000); the gain is headroom. All 124 tests that read nightly.yml: the same results as at 17bf155 (all green). shellcheck -S warning: clean on scripts/ci/nightly-steps/*.sh. actionlint 1.7.12-manaflow.1: the same 5 findings on nightly.yml as before (unused anchors). * ci(nightly): Generate deltas and revised appcast stays inline (its .publish-tooling checkout lists single files) Review of #31013 at 8a2494b: generate-nightly-deltas checks out .publish-tooling with a non-cone sparse list of five named files (anchor &checkout-the-gh-token-wrapper), so scripts/ci/nightly-steps/deltas-generate-appcast.sh would not be in it and the step would fail on every run, hidden by the job's continue-on-error. My selection matched the directory name as a substring of the list. The step goes back exactly as on the feat tip (block and its notes), the script is removed, and test_nightly_sparkle_key.sh is back to the feat tip's text. An exact check of all 140 step-script calls (the last checkout before each call that provides its path, root or .publish-tooling, must contain the file: no sparse set, a cone-mode directory or a non-cone pattern) finds no gap at this head; it found exactly this one before the fix, and none in the feat tip's 127 calls. nightly.yml: 458,313 bytes (31,687 under 490,000). All 124 tests that read nightly.yml: the same results as at the feat tip 49aa291. shellcheck -S warning clean; actionlint: the same 5 findings as before. --------- Co-authored-by: cmux-bot[bot] <348531757+cmux-bot[bot]@users.noreply.github.com>
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
NEXT control jobs cannot authenticate the private controller from a runner outside its network. Consume its authenticated GitHub Check Run instead: require the existing publishing App, exact run, attempt, source, jobs and canonical lease identity, plus the controller's terminal decision and clocks. Missing or invalid publication fails placement; it never becomes a no-start receipt or a paid runner choice.
Leo: “FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT”.
CI_NIGHTLY_ADMISSION_CHECKS=1opts ordinary NEXT builds in through a run/source-bound workflow title. The controller uses that marker and the actual current-attempt bootstrap job to admit demand before a runner starts. Dedicated-fleet jobs retain first priority and skip this reader. Unsigned-resume titles and routing remain compatible with the existing retry tracker. Downstream controls require a verified receipt from their own attempt, so a failed-jobs rerun cannot reuse stale placement and fall through to paid capacity.The only permission addition is
checks: readon the existing metadata job, needed to authenticate the Check Run's App identity. No App permission, credential, runner trust or release-environment change is made. Read-only permissions from #30248 are retained. Existing routes remain when the opt-in is disabled. Activation stays off until the controller demand and publisher changes are deployed.The reader makes at most two authenticated GitHub requests, with one delayed retry only when publication is absent. Its 210-second publication allowance covers the controller's 180-second lease and delivery margin; the job timeout also bounds the operation. API calls have a 15-second timeout and 1 MiB response limit, and reject redirects. There is no polling loop, background timer, dispatch or cancellation. Existing warm sources and compile caches are unchanged; this adds no build.
Validation: the same focused missing-receipt regression fails on base
f818cf72352(it selects a fallback despite opt-in) and passes at this head. New receipt/workflow tests (18), legacy admission (26), trusted routing (37), dedicated routing (7), routing policy (33), nightly routing (20), fork routing (23) and promotion-queue cases (9) pass. The live routing-table check, diff validation and pre-push checks pass. Job-scoped permissions (5), reusable permissions, workflow-size (2) and source-archive policy checks also pass. Two identical license steps now use YAML anchors to retain workflow-size headroom; the parsed workflow is unchanged by that deduplication. Coverage includes wrong App, identity/attempt/source mismatch, ambiguous or truncated publication, malformed JSON, nanosecond clock boundaries, missing/stale downstream outputs, dedicated priority, and anchored unsigned-resume titles.Observed / Not run
Observed: 173 focused tests and cases pass; the missing-receipt policy has a base/head regression. Not run: an enabled production nightly consuming the new publication. That proof follows controller deployment and explicit opt-in activation. Prior art: preserves #30085's identity-bound leases, #30173's disabled-mode fallback, #30249's dedicated priority and #30248's read-only metadata permissions.
Changelog
Agent session: agent session
🤖 Generated with Claude Code