Repository navigation
ci(overflow): the watch that holds the variable-write key keeps a literal ephemeral label - #27540
Conversation
…phemeral label Since #27022 the overflow watch takes its runner from CI_CONTROL_RUNNER, and it also mints the GLAEDA_ROUTE_APP_KEY token that writes repository variables. These tests require every job of the workflow that holds a secret to run on a literal blacksmith-* label, and every job CI_CONTROL_RUNNER moves (in any workflow) to hold no secret. On this commit they fail (3 failures in python3 tests/test_ci_cloud_overflow_switch.py).
…iteral label #27022 (Leo) let CI_CONTROL_RUNNER move the overflow watch, and its comment suggests an owned Mac. The watch mints the GLAEDA_ROUTE_APP_KEY token that writes repository variables. So a variable could put that key on a reused machine, where it can outlive the job, and the job could write the variable that picks its own runner. The watch goes back to the literal blacksmith-6vcpu-macos-26, as promote-nightly-next keeps its literal label for the same reason. The nightly tick and feed watch keep CI_CONTROL_RUNNER: they hold no secret, only their job-scoped GITHUB_TOKEN. Splitting the watch (a read-only movable watch, a key-holding writer on a literal label) gives nothing here: every literal ephemeral label the repo allows is Blacksmith (check_no_github_hosted_runners refuses GitHub-hosted), so the writer would wait on Blacksmith macOS exactly as this watch does. The 2026-10-28 case #27022 fixed (Blacksmith Linux down, macOS up) still works. When all of Blacksmith stops, the watch waits and the values are set by hand. Tests: python3 tests/test_ci_cloud_overflow_switch.py fails on the previous commit (3 failures) and passes here (62 tests); actionlint 1.7.7 is clean.
… so it names no values
|
All contributors have signed the CLA ✍️ ✅ |
|
Review (subagent, correctness first): SHIP at eee17df Security lens: no permission is widened. The App key that writes variables now runs only on a literal ephemeral label. The jobs that CI_CONTROL_RUNNER moves hold only the job GITHUB_TOKEN. The new tests fail on the base (3 failures) and pass on the head (62). Non-blocking findings are in the Observed / Not run section of the description: the Blacksmith macOS queue cost, and the older LINUX_RUNNER gap for the same key. |
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Pin and verify the watch runner .github/workflows/ci-cloud-overflow-probe.yml, tests/test_ci_cloud_overflow_switch.py, docs/ci-runners.md |
The watch job uses the literal blacksmith-6vcpu-macos-26 label. Documentation specifies which nightly jobs can use CI_CONTROL_RUNNER. Tests check that jobs with secrets or GitHub App token creation use literal cloud-prefixed runner labels, and that jobs selected by CI_CONTROL_RUNNER meet the stated constraints. |
Priority:
Estimated code review effort: 2 (Simple) | ~10 minutes
Change: Bug fix
Possibly related PRs
- manaflow-ai/cmux#27533: Pins the overflow watch to the same Blacksmith runner and adds related documentation and regression checks.
- manaflow-ai/cmux#27022: Introduced
CI_CONTROL_RUNNERfor the overflow watch and nightly control jobs. - manaflow-ai/cmux#27121: Establishes the literal Blacksmith macOS runner choice for the overflow watch.
Suggested reviewers: teamleaderleo
Merge Risk: 
eee17
-
No merge-blocking issue is identified. The watch remains dependent on Blacksmith availability; a provider-wide outage requires manual failover configuration as documented.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Title check The title names the main change: the overflow watch job that holds the variable-write key (GLAEDA_ROUTE_APP_KEY) now keeps a literal ephemeral runner label instead of a variable-selected one. It match… Description check The description follows the template's main sections: Summary, Testing, Changelog, Proof, and Checklist. The Summary states the problem (a repository variable could place the variable-write key on a r… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Cmux Cloud Persistent Session And Early Input The pull request changes only a CI workflow runner label, runner documentation, and regression tests. It introduces no Cloud terminal creation, cmux-tui client or transport, PTY readiness, input attac… Cmux Swift Actor Isolation The pull request changes only a GitHub Actions workflow, documentation, and Python tests. It introduces no production Swift changes, so the Swift actor isolation check is not applicable. Cmux Swift Blocking Runtime The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It introduces no production Swift code, so the Swift blocking-runtime check is not applicable. Cmux Browser Automation Off-Main PASS: The PR changes only the overflow workflow, CI runner documentation, and its Python tests. It does not modify Sources/TerminalController.swift,ControlCommandExecutionPolicy.swift, browser so…Cmux Expensive Synchronous Load PASS: The pull request changes only a GitHub Actions workflow, documentation, and a Python test. It adds no production Swift changes and cannot introduce an expensive synchronous agent-history load on… Cmux Cache Substitution Correctness The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It introduces no production Swift, TypeScript, or JavaScript changes, so the cache-substitution corr… Cmux No Hacky Sleeps PASS: The pull request changes one GitHub Actions workflow YAML file, documentation, and a Python test file. The runtime-no-hacky-sleeps rule excludes workflow YAML and allows test-only scaffolding. T… Cmux Algorithmic Complexity The PR changes only workflow YAML, documentation, and Python test code. It adds no production Swift, TypeScript, JavaScript, shell, or runtime algorithm. The new collection scans are test-only workflo… Cmux Swift Concurrency The pull request changes only one YAML workflow, one Markdown document, and one Python test. The reviewed diff contains no Swift files or cmux-owned Swift code, so it introduces no legacy Swift concur… Cmux Swift @ConcurrentPASS: The pull request changes only two workflow/documentation files and one Python test. The reviewed diff contains no Swift files or Swift code, so it cannot introduce a @concurrentor actor-isola…Cmux Swift Package Boundaries PASS: The pull request changes only a GitHub Actions workflow, CI documentation, and a Python test. It introduces no production Swift changes, so the Swift package-boundary rule does not apply. Cmux Swiftpm Lockfiles The PR changes only one workflow, one documentation file, and one Python test. It does not change any Package.swift,Package.resolved,.gitignore, Xcode project, dependency declaration, or packa…Cmux Swift Logging The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It adds or materially changes no production Swift code or Swift logging statements, so the cmux Swif… Cmux User-Facing Error Privacy PASS: The pull request changes only an internal CI workflow, CI documentation, and CI tests. The changed text names Blacksmith, GitHub, runner labels, environment variables, and credentials only in de… Cmux Full Internationalization PASS: The pull request changes a GitHub Actions workflow, CI regression tests, and operational CI-runner documentation. The added text is developer/operations content, and the workflow identifiers and… Cmux Swiftui State Layout The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It contains no Swift or SwiftUI changes, so the SwiftUI state-layout rules do not apply. Cmux Architecture Rethink PASS. The pull request changes only YAML, Markdown, and Python files. It introduces no Swift architecture change, so the specified Swift failure conditions do not apply. Cmux Swift Auxiliary Window Close Shortcuts PASS: The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It introduces no Swift code or standalone cmux-owned windows, so the auxiliary-window close-sh… Cmux Source Artifacts The PR changes only a workflow configuration, documentation, and a Python test. These are intentional source, docs, and test changes. No logs, screenshots, recordings, caches, build output, temporary … Cmux No Test Or Debug Seam In Production Source PASS: The pull request changes only two YAML/docs files and one Python test file. It changes no Swift file under any production Sources/path, so the specified production test/debug seam condition i…Full details: Title check
-
Explanation
The title names the main change: the overflow watch job that holds the variable-write key (GLAEDA_ROUTE_APP_KEY) now keeps a literal ephemeral runner label instead of a variable-selected one. It matches the raw summary and PR objectives, which both describe pinning the watch to literal blacksmith-6vcpu-macos-26. The title is long (89 characters) and somewhat awkward, but it is specific and relevant.
Full details: Description check
-
Explanation
The description follows the template's main sections: Summary, Testing, Changelog, Proof, and Checklist. The Summary states the problem (a repository variable could place the variable-write key on a reused self-hosted Mac, and the job could rewrite its own runner selector) and the resulting behavior (the watch is pinned to the literal Blacksmith label). Testing separates what ran from what was not run, including an 'Observed / Not run' subsection. The Changelog is set to 'none', which the template allows for CI-only changes. The Proof section gives a one-line reason, as the template requires. The Checklist has one item unchecked (subagent review), which is a pending action rather than a missing section. The description is thorough, though somewhat long relative to the template's 'keep it proportional' guidance. It also includes auto-generated footer and cubic summary text, which is noise but does not change the assessment.
Full details: Cmux Cloud Persistent Session And Early Input
-
Explanation
The pull request changes only a CI workflow runner label, runner documentation, and regression tests. It introduces no Cloud terminal creation, cmux-tui client or transport, PTY readiness, input attachment, authentication, idempotency, revision, or lease behavior. The custom check is therefore not applicable.
Full details: Cmux Browser Automation Off-Main
-
Explanation
PASS: The PR changes only the overflow workflow, CI runner documentation, and its Python tests. It does not modify
Sources/TerminalController.swift,ControlCommandExecutionPolicy.swift, browser socket commands, WebKit/AppKit routing, or worker-lane policy tests. The browser automation failure conditions are not applicable.
Full details: Cmux Expensive Synchronous Load
-
Explanation
PASS: The pull request changes only a GitHub Actions workflow, documentation, and a Python test. It adds no production Swift changes and cannot introduce an expensive synchronous agent-history load onto the main actor or an interactive path.
Full details: Cmux Cache Substitution Correctness
-
Explanation
The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It introduces no production Swift, TypeScript, or JavaScript changes, so the cache-substitution correctness rule does not apply.
Full details: Cmux No Hacky Sleeps
-
Explanation
PASS: The pull request changes one GitHub Actions workflow YAML file, documentation, and a Python test file. The runtime-no-hacky-sleeps rule excludes workflow YAML and allows test-only scaffolding. The diff introduces no production TypeScript, JavaScript, shell, or build/runtime delay code.
Full details: Cmux Algorithmic Complexity
-
Explanation
The PR changes only workflow YAML, documentation, and Python test code. It adds no production Swift, TypeScript, JavaScript, shell, or runtime algorithm. The new collection scans are test-only workflow-validation scaffolding, which the rule explicitly excludes.
Full details: Cmux Swift Concurrency
-
Explanation
The pull request changes only one YAML workflow, one Markdown document, and one Python test. The reviewed diff contains no Swift files or cmux-owned Swift code, so it introduces no legacy Swift concurrency pattern.
Full details: Cmux Swift `@Concurrent`
-
Explanation
PASS: The pull request changes only two workflow/documentation files and one Python test. The reviewed diff contains no Swift files or Swift code, so it cannot introduce a
@concurrentor actor-isolation violation.
Full details: Cmux Swiftpm Lockfiles
-
Explanation
The PR changes only one workflow, one documentation file, and one Python test. It does not change any
Package.swift,Package.resolved,.gitignore, Xcode project, dependency declaration, or package-resolution command. The workflow diff changes runner selection and comments only, so the SwiftPM lockfile policy is not violated.
Full details: Cmux Swift Logging
-
Explanation
The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It adds or materially changes no production Swift code or Swift logging statements, so the cmux Swift logging rule does not apply.
Full details: Cmux User-Facing Error Privacy
-
Explanation
PASS: The pull request changes only an internal CI workflow, CI documentation, and CI tests. The changed text names Blacksmith, GitHub, runner labels, environment variables, and credentials only in developer/operator-facing CI material. No concrete path to a cmux end user, product UI, product CLI, or product API was introduced.
Full details: Cmux Full Internationalization
-
Explanation
PASS: The pull request changes a GitHub Actions workflow, CI regression tests, and operational CI-runner documentation. The added text is developer/operations content, and the workflow identifiers and runner labels are literal configuration tokens. It introduces no Swift UI text, app catalog or Info.plist entries, web UI/API copy, rendered product data, changelog, or locale-sensitive message keys.
Full details: Cmux Swift Auxiliary Window Close Shortcuts
-
Explanation
PASS: The pull request changes only a GitHub Actions workflow, Markdown documentation, and a Python test. It introduces no Swift code or standalone cmux-owned windows, so the auxiliary-window close-shortcut rule does not apply.
Full details: Cmux Source Artifacts
-
Explanation
The PR changes only a workflow configuration, documentation, and a Python test. These are intentional source, docs, and test changes. No logs, screenshots, recordings, caches, build output, temporary directories, dependency checkouts, or other source-control artifacts appear in the diff.
Full details: Cmux No Test Or Debug Seam In Production Source
-
Explanation
PASS: The pull request changes only two YAML/docs files and one Python test file. It changes no Swift file under any production
Sources/path, so the specified production test/debug seam condition is not applicable.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 3 functions across 1 files. (2 skipped: 2 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Merge receipt for |
… label (#27557) * test(ci): a job that holds a GitHub App key runs where no variable can move it Every job with a create-github-app-token step or a private-key must run on a literal Blacksmith label (behind the fork branch at most, or with only the overflow record choosing between two literal sizes) or the exact CI_TRUSTED_RUNNER selector. Generalizes #27540's test for the overflow watch to every workflow. On this commit it fails for the 15 jobs that still take vars.LINUX_RUNNER. * ci: every job that holds a GitHub App key runs on a literal ephemeral label Follow-up to #27540. Fifteen jobs mint a GitHub App token (GLAEDA_ROUTE_APP_KEY, CMUX_BOT_APP_KEY, the release App) but took their runner from vars.LINUX_RUNNER, which may name any label, a reused owned machine included. A variable could then move an App private key and its token to a machine where they outlive the job; the GLAEDA_ROUTE token can write that variable itself. Each now runs on the Blacksmith label it already fell back to (behind the fork branch). nightly-next-launch-smoke's pick keeps the overflow record's choice between two literal Blacksmith sizes. ci.yml's reverse-test-impact report follows `changes` (its test pins the two equal). Fork pull requests already got these labels; nothing widens. Not changed, listed in the test with the reason: - nightly.yml generate-nightly-deltas: pick-classic-sign-runner names only Blacksmith macOS labels. - dogfood-artifact-publish.yml capture: CMUX_BOT_APP_KEY on MACOS_RUNNER_DISPLAY, the GUI capture mini by design; its owner must split the token out. Tests: python3 tests/test_ci_fork_runner_routing.py fails on the previous commit (15 jobs) and passes here (20 tests). * ci: correct the attribution jobs' runner comment and the dogfood capture exception Review of #27557: the analyze jobs mint App tokens, so their comment no longer says they need no secret; the capture exception names how the key can reach a reused mini (CI_PAID_MACOS_OVERFLOW and MACOS_RUNNER_DISPLAY, both of which the overflow switch sets). * test: follow the literal runner of the SDK cut-tags and e2e runner jobs CI fast guards and protocol contract on #27557: test_e2e_sibling_build pinned the sibling's runner to the runner job's, and the SDK guard pinned cut-tags to vars.LINUX_RUNNER. cut-tags now asserts the literal label and no variable; the sibling, which holds no key, keeps LINUX_RUNNER and shares the runner job's Blacksmith fallback.
… it (#27622) * test(dogfood): the GUI capture job holds no secret and no App token The capture runs on MACOS_RUNNER_DISPLAY (the reused GUI mini, a variable the overflow switch also sets) and minted CMUX_BOT_APP_KEY there. The tests now require a capture job with no secret, no token and actions: read, and the cmux-bot token only in capture-inputs and capture-report on a literal label. The routing test's exception for it is removed. Both fail on this commit. * ci(dogfood): mint the cmux-bot token beside the GUI capture, never on it The capture job runs on MACOS_RUNNER_DISPLAY when CI_PAID_MACOS_OVERFLOW is 1, and the overflow switch sets both in a Blacksmith outage (the owned GUI pool). It minted CMUX_BOT_APP_KEY there, so a variable could put that App key on a reused mini. Follows #27540 and #27557 (which built on #27022). Split it: - capture-inputs (literal Blacksmith Linux label) mints the token, finds the exact base cmux-next artifact and puts it into this run. - capture (the GUI mini, unchanged runner) holds no secret and no token: actions: read, contents: read; it downloads this run's head and base artifacts and uploads the captures. - capture-report (literal label) mints the token and posts the PR link. pull-requests: write moves from the capture job to capture-report; the capture job loses actions: write and pull-requests: write. Nothing widens. Tests: test_ci_workflow_run_api_budget.py (1 failure, 2 errors) and test_ci_fork_runner_routing.py (1 failure) fail on the previous commit and pass here.
…inux sizes (#27631) * test(ci): key-holding Linux jobs follow LINUX_RUNNER only through a bounded selector #27557 pinned 16 key-holding Linux jobs to blacksmith-4vcpu-ubuntu-2404, so a 4 vCPU stall (four on 2026-10-28) left them queued. The routing test now evaluates each key-holding job's runs-on with GitHub's expression semantics for every repository variable value (adversarial ones included): every result must be a label the expression itself names, and a Blacksmith label inside manaflow-ai. It requires those 16 jobs to use the one bounded selector (LINUX_RUNNER only when it names the 4, 16 or 32 vCPU Blacksmith Linux label). The janitor and SDK cut-tags tests pin the new line. * ci: key-holding Linux jobs follow LINUX_RUNNER among the Blacksmith Linux sizes #27557 moved 16 jobs that mint a GitHub App token off vars.LINUX_RUNNER onto the literal blacksmith-4vcpu-ubuntu-2404, so no variable could put a key on a reused machine. The cost: a 4 vCPU stall, which the overflow switch and CI_CLOUD_FAILOVER answer by pointing LINUX_RUNNER at another size, left them queued, ci.yml changes among them. They now read LINUX_RUNNER through one bounded selector: contains(fromJSON('["blacksmith-4vcpu-ubuntu-2404","blacksmith-16vcpu-ubuntu-2404", "blacksmith-32vcpu-ubuntu-2404"]'), vars.LINUX_RUNNER) && vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404' Any other value, an owned label included, falls back to 4 vCPU; the bound is in the expression, so no variable write can widen it. The fork branches stay as before #27557 (ci.yml changes, ci-macos placements, test-ios runner, and the reverse-test-impact report that follows changes). Builds on #27022, #27540 and #27557. Tests: the routing test fails on the previous commit (16 jobs lack the selector), as do the janitor and SDK cut-tags pins, and all pass here. * test(ci): the runs-on evaluator refuses coercion, unlisted contexts and prefix picks Review of #27631 found expressions the evaluator accepted that GitHub can resolve to an unbounded vars.LINUX_RUNNER: number literals and mixed-type comparisons (!0, 'true' == true), branches for an event or repository the value sets did not list (merge_group, another repository), and startsWith or substring picks. Now a number literal or a mixed-type comparison is unsupported (refused); each github.* and inputs.* reference also gets every literal it is compared with and one value outside the list, and a reference used other than in a comparison gets every literal and its extensions, as a variable does; variables also get each literal extended (-x, glaeda-, upper case). A fromJSON of a non-JSON value errors the job and is skipped. Each review example is in the refused list. * test(ci): a key-holding runs-on reads a variable only inside the bounded selector Second review of #27631: an enumeration of values cannot cover every form (a ref compared with a variable, contains() over a literal or variable list, fromJSON of a variable as a condition). So beside the evaluator, a syntax allowlist: a key-holding runs-on may read a repository variable only inside the exact bounded LINUX_RUNNER selector, the exact CI_TRUSTED_RUNNER selector, or the literal overflow-record pair. The reviewer's eight bypasses and three more are in the refused list. * test(ci): a structural runner rule replaces the enumeration of values Third review of #27631 found more forms an enumeration misses (an owned literal behind a contains() over a literal list, the overflow pair inside a comparison). Enumerating values cannot be complete, so the rule is now structural and needs none: a secret-holding runs-on is a || chain whose every operand yields a Blacksmith label the expression writes out, whatever its conditions (a literal, conditions && 'literal', or a bounded pick conditions && contains(fromJSON('[labels]'), R) && R), ending with a literal; only the first operand may be the exact fork branch to a hosted label, and the exact CI_TRUSTED_RUNNER selector is accepted. Conditions only choose among safe results, so they need no evaluation. All reviewers' bypasses are in the refused list; the evaluator stays only to show the selector's results. * test(ci): a bounded pick's list is a bare fromJSON literal; the fork operand compares the owner ref Fourth review of #27631: the parser dropped an index after a call, so contains(fromJSON('["blacksmith-self-hosted"]')[0], vars.X) && vars.X passed as a bounded pick, though GitHub then does a substring test and vars.X = 'self-hosted' would win. The parser now keeps the index as a fourth element of the call node (refs unchanged), and a bounded pick needs the bare call. The fork operand must compare the ref github.repository_owner with the string 'manaflow-ai', not two strings. Both forms are in the refused list.
…ble can move (#27648) * test(ci): every job that holds a secret runs where no variable can move it Extends the App-key rule (#27540, #27557, #27631) to every secret: R2, Vercel, Cloudflare, Stack server keys, Sentry, signing material, GHOSTTY_RELEASE_TOKEN. A job that reads `${{ secrets.X }}` (not its own GITHUB_TOKEN) or passes `secrets: inherit` must run on a fixed ephemeral label, a Blacksmith or GitHub-hosted one, or a runs-on whose every result, for every variable value, is a Blacksmith label the expression names (the bounded LINUX_RUNNER selector, the overflow record's literal pair). A matrix runner is checked per value the job's matrix lists; a local reusable call is checked through its callee's jobs. Jobs that take a picker's output are listed with a verified reason; four that hold a secret on an owned Mac by design are listed as open. It fails on this commit for 31 jobs that still read LINUX_RUNNER, MACOS_RUNNER_BACKGROUND, MACOS_RUNNER_IOS/PR or MACOS_RUNNER_15 unbounded. * ci: every job that holds a secret runs on an ephemeral label no variable can move Extends #27540, #27557, #27622 and #27631 (which built on #27022) from App keys to every secret. 31 jobs read a secret while a variable could point their runner at a reused machine. Linux, now through the bounded LINUX_RUNNER selector (#27631), so a 4 vCPU stall still moves them: ci-artifact-canary canary, cloud-capture-provision provision, cloud-vm-canary canary, cloud-vm-dogfood journey, cloud-vm-env-audit audit, cloud-vm-image-contract promote, cloud-vm-image-reachability reachable, cloud-vm-migrate migrate-staging and migrate-production, cloud-vm-smoke smoke, cmux-tui-artifacts publish, docs-deploy-reusable deploy, dogfood-artifact-publish publish, fleet-artifact-cdn deploy, ios-e2e backend, iroh-v2 deploy-development, presence deploy, r2-binaries-prune prune, r2-cache-prune prune, r2-storage-census binaries, cef and ci-cache, sdk-bootstrap-crates publish-sdk and publish-sidebar, update-homebrew update-cask, vercel-auth-health check. Fork branches are unchanged. macOS, now on the literal label they fell back to: - build-ghosttykit (GHOSTTY_RELEASE_TOKEN): was MACOS_RUNNER_BACKGROUND. - ios-e2e mac-ios-e2e (dogfood Stack login): was MACOS_RUNNER_IOS/PR. - ios-streamed-validate validate (CLAUDE_CODE_OAUTH_TOKEN): was MACOS_RUNNER_IOS. - iroh-release-gate simulator-e2e (production Stack keys): was MACOS_RUNNER_15 under CI_PAID_MACOS_OVERFLOW. - nightly refresh-test-compilation-cache (CI_CACHE_R2): its first matrix pool no longer falls through to MACOS_RUNNER_PR. Today every changed variable holds the label these jobs now name, so nothing moves now. Four jobs keep a secret on an owned Mac by design and are listed open in the test (build-nightly-app, seed-derived-data seed, test-ios ios-simulator-build, ios-screenshots screenshots). One consequence: no signing job on main reads MACOS_RUNNER_15 any more (iroh-release-gate simulator-e2e did), so signing_runner_variables.py no longer derives it, and the overflow switch now fails it over to the owned std pool like MACOS_RUNNER_PR; only jobs without a secret follow it. Its tests and docs/ci-runners.md say so. test_ci_pr_runner_pool.py pins the two new literal labels. The trusted fork exclusion also refuses an index after contains() or fromJSON() (review of #27631; no workflow used it). Tests: the routing test fails on the previous commit (31 jobs) and passes here. * ci: the App Store upload takes the capability map's label only when it is Blacksmith Review of #27648 found the ios-app-store exception false: its runner came from the capability map, which reads CMUX_CI_RUNNER_OVERRIDES, a variable that may name any machine, and the job holds the App Store Connect key. Its runs-on now keeps the map's answer only when it is blacksmith-6vcpu-macos-26 (else that label), and the exception is gone; the structural rule's bounded pick accepts the same expression in contains() and as the result (a capability lookup), never two different ones. The detector also counts a workflow-level env, toJSON(secrets) and a bare secrets context; a computed matrix list or include fails closed. Docs no longer list build-ghosttykit in the background lane, ios-streamed-validate under MACOS_RUNNER_IOS or the refresh seeder as running on MACOS_RUNNER_PR. * test(seed): the nightly seeder's pools match the per-push seeder's by value CI on #27648: test_seed_derived_data compared the nightly refresh-test-compilation-cache pool expressions with the seeder's as text, so the nightly row that no longer falls through to MACOS_RUNNER_PR failed it. It now evaluates both for each lane MACOS_RUNNER_PR may name (Blacksmith sizes, an owned pool, unset): each nightly pool is a pool the per-push seeder uses, and a Blacksmith label. On the base it fails for an owned lane: the nightly seeder, which holds CI_CACHE_R2, ran on the overflow switch's owned pool.
…never on the Mac (#27718) * test(ios): the simulator build holds no secret; the R2 save runs beside it ios-simulator-build runs on ios_runner_pool.py's pick, an owned simulator Mac when CI_IOS_OWNED is 1, or whatever MACOS_RUNNER_TESTS names, and it named CI_CACHE_R2_* there (in the seed_cache preflight) and entered ci-cache-writer on main. The tests now require: - the build job names no secret and enters no environment; - the R2 restore stays in the build job (it reads the public URL, no credentials), with the same key, if and backend; - the GitHub/Warp save stays in the build job (no secret, and a cache written on Linux does not restore on macOS); for R2 the build packs the packages and uploads them as an artifact of this run; - ios-simulator-cache-save, on a literal ephemeral Linux label in ci-cache-writer, runs the preflight, downloads the artifact and saves it to R2 under the build's key and platform; - r2-cache.sh takes CI_CACHE_R2_PLATFORM, so a Linux job can save a macOS archive where macOS restores look. The routing test's exception for the job is removed. All three fail on this commit. * ci(ios): save the simulator build's R2 cache beside it, never on the Mac ios-simulator-build runs on ios_runner_pool.py's pick: an owned simulator Mac when CI_IOS_OWNED is 1, or whatever MACOS_RUNNER_TESTS names. On a main seed_cache run it named CI_CACHE_R2_ACCESS_KEY_ID, CI_CACHE_R2_SECRET_ACCESS_KEY and CI_CACHE_R2_ACCOUNT_ID (the "Seed iOS caches from main" preflight), and on every main run it entered ci-cache-writer. Follows #27022, #27540, #27557 and #27622 (dogfood capture-inputs / capture / capture-report). What moved where: - The R2 restore stays in the build job. It reads the public URL and needs no credentials (cache-restore, r2-cache.sh restore), and an owned Mac skips it when it keeps its packages. Moving it would add a Linux job, an artifact upload and a download (the R2 entry is 2.17 GB) before every build on the pull-request path, for no secret. - The GitHub and Warp saves stay in the build job: they need no secret, and a GitHub-store entry written on Linux would not restore on macOS. Their `if:` adds `backend != 'r2'`. - For R2 the build job packs .ci-ios-source-packages (tar | gzip -1) and uploads it as an artifact of this run (1 day), with its key and platform as new outputs. Same `if:` as before plus `backend == 'r2'`. - New ios-simulator-cache-save, on the literal ephemeral Linux label, in ci-cache-writer on main, with contents: read and actions: read: the preflight (unchanged), the download, and the cache-save action with the R2 keys, under the build's key and its platform (CI_CACHE_R2_PLATFORM, new in r2-cache.sh, so a Linux job writes where macOS restores look). - ios-tests needs the new job, so a failed save still fails the gate. The build job no longer names a secret, no longer enters ci-cache-writer, and can no longer write to R2. Its other outputs, the restore key, hit and miss behavior and the job's `if:` are unchanged. One behavior change: the old R2 "Save iOS SwiftPM cache" step had no credentials in its env (since #13314 only the preflight had them), so on R2 it warned and saved nothing. The new save has them and writes the key the upload lanes already write. * test(ios): only the seed preflight moves; the R2 save stays a no-op as on base Review decision on #27718: a security split must not change behavior. The R2 "Save iOS SwiftPM cache" step never had the bucket keys in its env (only the preflight step did), so on R2 it warned and saved nothing. Making it write is a functional change for its own PR. The test now requires the build job's restore and save steps exactly as on base (no env, same key, if and backend), no new outputs, and the preflight alone in ios-simulator-cache-seed on a literal ephemeral label, in ci-cache-writer on main, after a successful build, gated by ios-tests. The CI_CACHE_R2_PLATFORM case in test_ci_r2_cache.sh goes, with its script change. Fails on this commit (the save path of the previous commit is still there). * ci(ios): move only the seed preflight off the simulator build Review decision on #27718: a security split keeps behavior. The previous commit made the R2 save actually write; that is a functional change for its own PR. This takes it back: - r2-cache.sh is as on base (no CI_CACHE_R2_PLATFORM). - ios-simulator-build has no pack or upload step and no new output. Its "Save iOS SwiftPM cache" step is as on base: the cache-save action takes the bucket keys from the calling step's env, which has none, so on R2 it warns and saves nothing, exactly as before. - ios-simulator-cache-save becomes ios-simulator-cache-seed: the preflight alone ("Seed iOS caches from main", unchanged), on the literal ephemeral Linux label, in ci-cache-writer on main, contents: read, after a successful build on a main seed_cache run. ios-tests needs it, so a missing key still fails the run. The build job still names no secret and enters no environment.
Summary
Replaces #27533, whose commits had an empty author email, which made CLA Assistant fail closed. The tree is the same, plus one doc sentence.
Since #27022 (thanks @teamleaderleo for the Blacksmith Linux outage fix), the overflow
watchjob inci-cloud-overflow-probe.ymltakes its runner fromvars.CI_CONTROL_RUNNER, and its comment suggests an owned Mac label. The same job mints theGLAEDA_ROUTE_APP_KEYtoken, which can write repository variables. As a result:This PR puts the watch back on the literal
blacksmith-6vcpu-macos-26.promote-nightly-nextkeeps its literal label for the same reason (#27022: "only a literal ephemeral label may carry" App credentials).nightly-next-tick.ymlandnightly-next-feed-watch.ymlkeepCI_CONTROL_RUNNER. I checked them: they hold no secret and no App token, only their job-scopedGITHUB_TOKEN.Why not split the watch. One option is a read-only watch on the movable runner plus a separate writer that holds the key on a literal label. That split gives no benefit here. The repo allows only Blacksmith as a literal ephemeral label, because
check_no_github_hosted_runners(Leo, 2026-10-27) refuses GitHub-hosted labels. So the writer would wait on Blacksmith macOS, the same as this watch does. A first version of this branch put the writer onubuntu-24.04, and that guard refused it.What stays of Leo's outage path:
CI_CONTROL_RUNNERstill moves the nightly control jobs.New guards in
tests/test_ci_cloud_overflow_switch.py:blacksmith-*label;CI_CONTROL_RUNNERmoves holds a secret or mints an App token.Testing
python3 tests/test_ci_cloud_overflow_switch.py: 3 failures on the test commit 57e7840 (the base workflow), and 62 passes on the head.tests/test_ci_runner_pools.py,tests/test_ci_nightly_off_blacksmith.py,tests/test_ci_fork_runner_routing.py,tests/test_ci_repo_variable_defaults.py,tests/test_ci_macos_xcode_selection.py,tests/test_ci_fast_checks_failover.py,tests/test_ci_change_areas.py,tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py,bash tests/test_ci_self_hosted_guard.sh, andpython3 scripts/verify-local.py --affected origin/main(16/16).ci-cloud-overflow-probe.yml.Observed / Not run
CI_CONTROL_RUNNERunset (so the label was the same as this PR's): watch jobs queued 6 to 39 minutes onblacksmith-6vcpu-macos-26while both Linux probes started in seconds (run 40938852843: created 19:18Z, watch started 19:57Z). A waiting watch holds the fixed concurrency group, so later scheduled runs were cancelled (40940070200, 40941668804, 40942901017). This PR does not change that, because the variable is unset. It does remove the only way to move the watch without a PR. A follow-up could add a selector bounded to Blacksmith labels, like theCI_TRUSTED_RUNNERone; such a selector cannot name a reused machine.GLAEDA_ROUTE_APP_KEYis minted in jobs whose runner comes fromvars.LINUX_RUNNERwith no allowlist (for exampleci-queue-janitor.yml,ci-hosted-queue-rescue.yml,test-e2e.yml). What limits that today:LINUX_RUNNERis a signing variable, so the switch never writes a reused machine to it, andcheck_repo_variables.pyreports drift.blacksmith-6vcpu-macos-26. That label is the one it used before ci: the overflow watch and nightly control jobs never wait on Blacksmith Linux #27022, and it is also the default whenCI_CONTROL_RUNNERis unset.Changelog
none
Proof
CI only: no user-visible change.
Checklist
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Restores the overflow
watchjob to the literalblacksmith-6vcpu-macos-26runner label instead of usingCI_CONTROL_RUNNERwhen set. The watch mints theGLAEDA_ROUTE_APP_KEYtoken that writes repository variables, so a variable-driven runner could place the key on a reused machine where it outlives the job and could rewrite its own runner.Adds tests that require secret-holding jobs in the overflow workflow to use literal Blacksmith labels and that reject secrets in any job
CI_CONTROL_RUNNERmoves.Updates
docs/ci-runners.mdto document the literal-label rule and the failover behavior when all Blacksmith instances are down.Written for commit eee17df. Summary will update on new commits.
Summary by CodeRabbit
CI_CONTROL_RUNNERoverride is limited to eligible nightly jobs; credential-sensitive jobs remain on fixed cloud runners.