Repository navigation
ci(nightly): prefer reserved mini for secret-free signing checks - #30283
Conversation
GitHub Actions
Inactive
|
Review (subagent credentialed-controls-review): BLOCK at 94f901b Blocking finding: Focused checks otherwise pass: |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".github/workflows/nightly.yml">
<violation number="1" location=".github/workflows/nightly.yml:4065">
P2: Use the full branch ref here so a manually dispatched tag named `nightly-next` cannot receive the reserved NEXT-only runner. Otherwise this regular nightly run also selects the reserved label and PR Xcode configuration, contrary to the fallback behavior for other tracks.</violation>
</file>
<file name="tests/test_nightly_sign_runner.py">
<violation number="1" location="tests/test_nightly_sign_runner.py:584">
P2: This weakens the regression test enough to allow retries to route to the reserved or legacy owned runner. Restore an exact assertion, or assert each attempt/event gate and its grouping, so a future routing change cannot bypass the attempt-1 and marker safeguards unnoticed.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| # an unset, stale or foreign value falls through to the normal owned picker. | ||
| # Release-pool signing jobs consume sign_owned instead and keep their | ||
| # existing keychain and claim contract. | ||
| owned: ${{ github.repository == 'manaflow-ai/cmux' && github.ref_name == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || (github.event_name == 'push' && github.run_attempt == 1 || steps.upload-marker.outcome == 'success') && steps.owned.outputs.owned || '' }} |
There was a problem hiding this comment.
P2: Use the full branch ref here so a manually dispatched tag named nightly-next cannot receive the reserved NEXT-only runner. Otherwise this regular nightly run also selects the reserved label and PR Xcode configuration, contrary to the fallback behavior for other tracks.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 4065:
<comment>Use the full branch ref here so a manually dispatched tag named `nightly-next` cannot receive the reserved NEXT-only runner. Otherwise this regular nightly run also selects the reserved label and PR Xcode configuration, contrary to the fallback behavior for other tracks.</comment>
<file context>
@@ -4057,7 +4057,12 @@ jobs:
+ # an unset, stale or foreign value falls through to the normal owned picker.
+ # Release-pool signing jobs consume sign_owned instead and keep their
+ # existing keychain and claim contract.
+ owned: ${{ github.repository == 'manaflow-ai/cmux' && github.ref_name == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || (github.event_name == 'push' && github.run_attempt == 1 || steps.upload-marker.outcome == 'success') && steps.owned.outputs.owned || '' }}
# A drained (or draining) owned Mac for the release signing jobs (sign-nightly-identity-legacy), as the JSON runs-on
# array scripts/ci/signing_runner.py prints, or empty, under the same marker and `attempt` rules as `owned`.
</file context>
| owned: ${{ github.repository == 'manaflow-ai/cmux' && github.ref_name == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || (github.event_name == 'push' && github.run_attempt == 1 || steps.upload-marker.outcome == 'success') && steps.owned.outputs.owned || '' }} | |
| owned: ${{ github.repository == 'manaflow-ai/cmux' && github.ref == 'refs/heads/nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || (github.event_name == 'push' && github.run_attempt == 1 || steps.upload-marker.outcome == 'success') && steps.owned.outputs.owned || '' }} |
| self.assertEqual(job["outputs"]["owned"], | ||
| "${{ (github.event_name == 'push' && github.run_attempt == 1 || " | ||
| "steps.upload-marker.outcome == 'success') && steps.owned.outputs.owned || '' }}") | ||
| owned_output = job["outputs"]["owned"] |
There was a problem hiding this comment.
P2: This weakens the regression test enough to allow retries to route to the reserved or legacy owned runner. Restore an exact assertion, or assert each attempt/event gate and its grouping, so a future routing change cannot bypass the attempt-1 and marker safeguards unnoticed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_nightly_sign_runner.py, line 584:
<comment>This weakens the regression test enough to allow retries to route to the reserved or legacy owned runner. Restore an exact assertion, or assert each attempt/event gate and its grouping, so a future routing change cannot bypass the attempt-1 and marker safeguards unnoticed.</comment>
<file context>
@@ -581,9 +581,11 @@ def test_no_job_that_holds_a_secret_reads_the_owned_pick(self) -> None:
- self.assertEqual(job["outputs"]["owned"],
- "${{ (github.event_name == 'push' && github.run_attempt == 1 || "
- "steps.upload-marker.outcome == 'success') && steps.owned.outputs.owned || '' }}")
+ owned_output = job["outputs"]["owned"]
+ self.assertIn("vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro'", owned_output)
+ self.assertIn("github.repository == 'manaflow-ai/cmux'", owned_output)
</file context>
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
GitHub Actions
Inactive
|
Review (subagent credentialed-controls-review): BLOCK at 8d9d896 The prior signing-owned-Mac invariant is fixed, but the exact head still has an own red: Other focused checks pass: |
GitHub Actions
Inactive
GitHub Actions
Inactive
There was a problem hiding this comment.
2 issues found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test_ci_signing_owned_mac.py">
<violation number="1" location="tests/test_ci_signing_owned_mac.py:482">
P2: These substring checks no longer enforce the marker predicates described by the test. Assert each output against its complete expected expression so retries cannot claim stale picks and the reservation remains restricted to its valid scope.</violation>
</file>
<file name=".github/workflows/nightly.yml">
<violation number="1" location=".github/workflows/nightly.yml:4037">
P3: This reservation branch makes the picker run on `glaeda-nightly-only-m4pro`, but the preceding comment still says it always runs on Blacksmith Linux. Update the comment to document the reserved-Mac exception and its fallback.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| # `sign_owned` output. Both outputs still stand on their marker rules. | ||
| self.assertIn("steps.sign-owned.outputs.owned", str(outputs["sign_owned"])) | ||
| self.assertNotIn("CI_NIGHTLY_ONLY_RUNNER", str(outputs["sign_owned"])) | ||
| self.assertIn("CI_NIGHTLY_ONLY_RUNNER", str(outputs["owned"])) |
There was a problem hiding this comment.
P2: These substring checks no longer enforce the marker predicates described by the test. Assert each output against its complete expected expression so retries cannot claim stale picks and the reservation remains restricted to its valid scope.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_signing_owned_mac.py, line 482:
<comment>These substring checks no longer enforce the marker predicates described by the test. Assert each output against its complete expected expression so retries cannot claim stale picks and the reservation remains restricted to its valid scope.</comment>
<file context>
@@ -474,8 +474,12 @@ def test_the_picker_runs_when_only_the_legacy_sign_does(self) -> None:
+ # `sign_owned` output. Both outputs still stand on their marker rules.
+ self.assertIn("steps.sign-owned.outputs.owned", str(outputs["sign_owned"]))
+ self.assertNotIn("CI_NIGHTLY_ONLY_RUNNER", str(outputs["sign_owned"]))
+ self.assertIn("CI_NIGHTLY_ONLY_RUNNER", str(outputs["owned"]))
</file context>
| # allowlisted Blacksmith Linux set even when RUNNER_POOLS is stale or | ||
| # points at a non-Blacksmith fleet member. | ||
| runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-4vcpu-ubuntu-2404' || contains(fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]'), fromJSON(vars.RUNNER_POOLS).linux) && fromJSON(vars.RUNNER_POOLS).linux || fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]')[0] }} | ||
| runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.repository == 'manaflow-ai/cmux' && github.ref_name == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-4vcpu-ubuntu-2404' || contains(fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]'), fromJSON(vars.RUNNER_POOLS).linux) && fromJSON(vars.RUNNER_POOLS).linux || fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]')[0] }} |
There was a problem hiding this comment.
P3: This reservation branch makes the picker run on glaeda-nightly-only-m4pro, but the preceding comment still says it always runs on Blacksmith Linux. Update the comment to document the reserved-Mac exception and its fallback.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 4037:
<comment>This reservation branch makes the picker run on `glaeda-nightly-only-m4pro`, but the preceding comment still says it always runs on Blacksmith Linux. Update the comment to document the reserved-Mac exception and its fallback.</comment>
<file context>
@@ -4034,7 +4034,7 @@ jobs:
# allowlisted Blacksmith Linux set even when RUNNER_POOLS is stale or
# points at a non-Blacksmith fleet member.
- runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-4vcpu-ubuntu-2404' || contains(fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]'), fromJSON(vars.RUNNER_POOLS).linux) && fromJSON(vars.RUNNER_POOLS).linux || fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]')[0] }}
+ runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || github.repository == 'manaflow-ai/cmux' && github.ref_name == 'nightly-next' && github.run_attempt == 1 && vars.CI_NIGHTLY_ONLY_RUNNER == 'glaeda-nightly-only-m4pro' && vars.CI_NIGHTLY_ONLY_RUNNER || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && 'blacksmith-4vcpu-ubuntu-2404' || contains(fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]'), fromJSON(vars.RUNNER_POOLS).linux) && fromJSON(vars.RUNNER_POOLS).linux || fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404","blacksmith-32vcpu-ubuntu-2404"]')[0] }}
# The owned pick mints the routing token (GLAEDA_ROUTE_APP_KEY), so the key is read in the
# routing environment, as pick-nightly-build-runner does (tests/test_supplychain_security.py).
</file context>
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base f22cb91; does not block Shipped in:
|
|
Review (independent Codex gpt-6.1-sol, high, via the review router; full diff from merge base 0771403 to this head): BLOCK at 2ec02f1. [P1] [blocking] |
|
Orphaned: no activity for 14 h, and no lane holds a lease on it. The orphan sweep could not merge |
|
Owned by an agent session · session |
…tialed-nightly-signer-prepare-next # Conflicts: # .github/workflows/nightly.yml # tests/test_ci_nightly_off_blacksmith.py # tests/test_ci_signing_owned_mac.py
…tialed-nightly-signer-prepare-next
GitHub Actions
Inactive
|
Review (subagent claude-opus-5-5 medium): SHIP at fc06cd1 Checked every consumer of |
|
Found 6 test failures on Blacksmith runners: Failures
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
![cmux-bot[bot]](https://avatars.githubusercontent.com/u/171392238?s=60&v=4){kind=small}
A newer verdict blocks b900ccc (no independent SHIP at the head).
|
Review (subagent claude-opus-5-5 medium): SHIP at b900ccc Codex BLOCK from 2ec02f1 is resolved: pick-nightly-sign-runner's runs-on is now byte-identical to base's (#31642, Follow-ups: [follow-up] the reservation path sets CMUX_CI_XCODE_APP to vars.CMUX_CI_XCODE_APP_PR, so the nightly-only reserved minis need that Xcode installed (worth one check once the label is advertised). |
The secret-free signed-app verification and DMG-image jobs now consume the exact all-nightly M4 Pro reservation on same-repository NEXT attempt 1. The picker output prefers
CI_NIGHTLY_ONLY_RUNNER=glaeda-nightly-only-m4pro, while malformed, unset, fork, other-track and retry contexts retain the existing owned-fleet and Blacksmith fallback.This keeps release credentials, signing claims and temporary-keychain cleanup unchanged. The Linux ELF signer compiler and credentialed signer/notary jobs remain on the existing Linux picker; no release secret is routed to the reservation. The change is gated by the existing activation variable, which remains off until fleet registration and the exclusive claim hook are proven.
Leo asked: “For our nightlies, are they end to end on our fleet? We can't be relying on Blacksmith that cuts out.”
Validation:
python3 tests/test_nightly_dedicated_mini.py </dev/null(8 passed)python3 tests/test_nightly_sign_runner.py </dev/null(46 passed)python3 tests/test_nightly_trusted_routing.py </dev/null(35 passed)python3 tests/test_ci_nightly_off_blacksmith.py </dev/null(20 passed)git diff --checkpass.Follow-up: preparation jobs are covered by #30113 and its dedicated-preference follow-up; credentialed signer Mac support still requires the per-job keychain/notary-profile design tracked in HQ research #5742.
Agent session: agent session
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Makes the secret-free nightly verify and DMG-image jobs prefer the reserved
glaeda-nightly-only-m4prorunner on same-reponightly-nextattempt 1.CI_NIGHTLY_ONLY_RUNNERis exactly set; unset, stale, or foreign values fall through to the existing owned-fleet and Blacksmith selection.sign_ownedcontract, and no release secret routes to the reservation.CI_NIGHTLY_ONLY_RUNNER, which stays off until fleet registration and the exclusive claim hook are proven.Written for commit b900ccc. Summary will update on new commits.