[teamleaderleo]

Why

The overflow switch (ci-cloud-overflow-probe.yml) runs on a schedule, so it runs from main. On 2026-10-28 it failed over three times (21:59Z, 22:13Z, 22:53Z) and each time pointed MACOS_RUNNER_26 at the std owned pool. Signing jobs on feat-cmux-next read that variable in runs-on (rule 7 of tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py), so cmux-next-guards.yml's signing runner variables are ephemeral job went red on feat-cmux-next and on every pull request into it. The variables are repository-wide.

#27094 fixes this in the switch, but on feat-cmux-next. The scheduled run never executes that copy, so on its own it cannot stop the re-fire. This PR ports #27094 onto main's newer switch (Linux failover to another Blacksmith size, the hosted-Ubuntu record migration). Keep both: #27094 also adds the rule 7 test's list check, which exists only on feat-cmux-next.

What

• scripts/ci/signing_runner_variables.py: the one list of variables signing jobs read (LINUX_RUNNER, MACOS_RUNNER_26) and the ephemeral rule (blacksmith-* or ubuntu-*). It matches ci(overflow): never point a signing runner variable at an owned mini #27094's file plus one note that main carries a copy.
• cloud_overflow_switch.py:
  • MACOS_RUNNER_26 is removed from STD_VARIABLES. While Blacksmith is down, its jobs wait for Blacksmith macOS rather than moving to a reused mini.
  • MACOS_RUNNER_26 is also removed from what CI_CLOUD_FAILOVER may name. On main, every value an override could give it is already refused: GitHub-hosted, a Blacksmith label for a macOS variable, or a reused machine.
  • An override that points LINUX_RUNNER at a non-ephemeral label is refused.
  • plan_off and plan_resume never write a non-ephemeral label to a listed variable. That covers records written before this change, like today's, which still name MACOS_RUNNER_26 with an owned after. plan_on still restores a recorded value.
• check_repo_variables.py never excuses a listed variable that names a reused machine, even when a switch record names it.
• The guard is unchanged and not loosened. Owned-Mac signing keeps its own fail-closed route (ci(signing): the nightly identity signing job may take a drained owned Mac, held whole #26944, ci(signing): owned Macs first for the release, repair and DEV signing jobs, with a stall-aware fallback #27066, ci(signing): the owned signing pick accepts the nightly slot label #27113): a picker output in runs-on, never a shared overflow variable. If signing jobs later choose their runner through that picker alone, the guard can check the pick instead. That change belongs with those PRs.

Not in this PR (coordination)

• The switch already restores itself. When its probe starts again it puts back the record's before values and deletes the record; it did so at 23:21:20Z today, and the guard went green after that.
• Probing more than one label and stall-aware failover are separate work in flight. ci: the overflow watch and nightly control jobs never wait on Blacksmith Linux #27022 moves the watch off Blacksmith Linux. This PR leaves the probe and the workflow alone, so it does not conflict with either.

Tests

• tests/test_ci_cloud_overflow_switch.py: 46 pass. The signing-variable tests are ported from ci(overflow): never point a signing runner variable at an owned mini #27094 and adapted: on main, no override can move MACOS_RUNNER_26, and a Blacksmith Linux override for LINUX_RUNNER is kept.
• tests/test_ci_check_repo_variables.py and tests/test_ci_no_github_hosted_macos.py pass.

Risk: low. CI routing only. While Blacksmith macOS is down, jobs that read MACOS_RUNNER_26 queue for it instead of moving to the owned std pool; rule 7 requires that. The other macOS variables still fail over to the owned pools as before.

🤖 Generated with Claude Code

Agent session: agent session

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Stops main's scheduled overflow switch from ever pointing signing runner variables at reused machines, fixing the intermittent signing runner variables are ephemeral guard failures on feat-cmux-next and its pull requests.

The switch runs from main, so the existing feat-cmux-next fix cannot stop it: on 2026-10-28 its failover pointed MACOS_RUNNER_26 at the owned std pool three times, turning the guard red on every PR.

Changes

• Adds scripts/ci/signing_runner_variables.py as the single list of variables signing jobs read (LINUX_RUNNER, MACOS_RUNNER_26) and the ephemeral rule (blacksmith-* or ubuntu-*).
• Removes MACOS_RUNNER_26 from the owned failover and from what overrides may name; its jobs wait while Blacksmith is down.
• Refuses overrides naming a reused machine for a signing variable, and a stalled probe never re-applies an older record's owned value; restoring overflow still puts back the recorded value.
• check_repo_variables.py never excuses a reused machine in these variables, even when a switch record names it.

Risk is low: CI routing only. The other macOS variables still fail over to the owned pools, and owned-Mac signing keeps its fail-closed picker route. Keep the signing_runner_variables.py list in sync with the feat-cmux-next copy.

^{Written for commit 7005919. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes
  • Signing jobs are protected from failover settings that would route restricted runners to reused machines.
  • Unsafe runner overrides and retry destinations are ignored, while valid failover restoration continues to work.
  • Runner-variable checks no longer treat a saved overflow record as sufficient to allow an invalid restricted runner value.
• Chores
  • CI checks now run when signing-runner configuration changes.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Walkthrough

The changes define allowed values for signing-runner variables and apply those checks to overflow failover, resume handling, and repository-variable drift checks. Tests cover the updated behavior, and the CI workflow now runs when the shared policy module changes.

Changes: Signing runner variable protections

Layer / File(s)	Summary
Define signing-runner value policy scripts/ci/signing_runner_variables.py	The new shared module identifies LINUX_RUNNER and MACOS_RUNNER_26 as restricted variables. It accepts unset values or labels with the specified blacksmith- or ubuntu- prefixes and suffix characters.
Enforce policy in overflow handling scripts/ci/cloud_overflow_switch.py, docs/ci-runners.md, tests/test_ci_cloud_overflow_switch.py	The switch filters unsafe overrides and failover destinations, and skips unsafe recorded destinations during resume. Tests cover these paths and confirm that restoring overflow still uses the recorded value.
Apply policy to repository-variable checks scripts/ci/check_repo_variables.py, .github/workflows/ci-repo-variables.yml, tests/test_ci_cloud_overflow_switch.py	Recorded overflow values excuse a drift finding only when they pass the shared policy and the existing hosted-Mac check. The workflow path filter includes the policy module, and a test covers an invalid recorded value.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Possibly related issues

• manaflow-ai/cmux#26450: This change checks reused-machine values for MACOS_RUNNER_26 in the overflow and repository-variable paths described by the issue.
• manaflow-ai/cmux#26576: This change protects signing variables in overflow handling, but does not implement the separate watch and prepare-job fixes described by the issue.

Suggested labels: risk:low

Suggested reviewers: lawrencecchen

Merge Risk: Minimal · up to 70059

The change keeps signing runner variables on ephemeral labels in the overflow switch and stops the repo-variable checker from excusing reused-machine values. No merge-blocking risk was established. A reviewer comment about the checker not reporting every non-ephemeral signing value was not substantiated and concerns behavior this change leaves as it was.

Pre-merge checks: 23 · 2

Failed checks: 2 warnings

Check name	Status	Explanation	Resolution
Description check		The description clearly explains the problem, implementation, scope, testing, and risk. It does not use the required Summary, Testing, Changelog, Proof, and Checklist sections, and it omits the requir…	Restructure the description using the repository template. Add Summary, Testing, Changelog, Proof, and Checklist sections. Include a changelog entry or state "none", and add a one-line Proof statement explaining why no screenshots or video …
Docstring Coverage		Docstring coverage is 38.46% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 23 passed

Check name	Status	Explanation
Title check		The title clearly describes the primary change: preventing the overflow switch from assigning signing runner variables to owned machines.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input		PASS: The pull request changes CI runner-variable failover logic, documentation, and related tests only. The changed-file inventory contains no Cloud terminal creation, cmux-tui, Ghostty, transport, P…
Cmux Swift Actor Isolation		The pull request changes only YAML, Markdown, Python, and Python test files. The diff contains no Swift or Swift project files, so it introduces no Swift 6 actor-isolation issue.
Cmux Swift Blocking Runtime		The pull request changes Python, YAML, Markdown, and test files only. The reviewed diff contains no Swift, Objective-C, or Objective-C++ files, so it does not introduce or expand Swift blocking or tim…
Cmux Browser Automation Off-Main		PASS. The reviewed diff changes CI overflow routing, runner-variable validation, documentation, and Python tests only. It introduces no browser.* commands, socket-worker routing, WebKit/AppKit acces…
Cmux Expensive Synchronous Load		PASS: The pull request changes only YAML, Markdown, Python, and tests. It adds no production Swift code or Swift call sites, so the expensive synchronous Swift load condition does not apply.
Cmux Cache Substitution Correctness		PASS: The pull request changes only YAML, Markdown, Python, and Python tests. It contains no production Swift, TypeScript, or JavaScript change, so the cache-substitution correctness condition is not …
Cmux No Hacky Sleeps		The PR adds runner-variable validation and failover filtering only. It does not add or expand sleep, polling, timers, backoff, or wall-clock waits. The existing time.sleep polling and cancellation…
Cmux Algorithmic Complexity		PASS: The pull request changes only Python CI routing, documentation, workflow metadata, and tests. The new scans in cloud_overflow_switch.py cover the fixed runner-variable maps and the two signing…
Cmux Swift Concurrency		PASS: The reviewed diff changes only workflow YAML, documentation, Python scripts, and Python tests. It contains no Swift code, so it does not introduce or expand any Swift legacy async pattern covere…
Cmux Swift @Concurrent		PASS: The pull request changes only YAML, Markdown, Python, and Python test files. The authoritative diff contains no Swift files or Swift isolation changes, so the Swift @concurrent check is not appl…
Cmux Swift Package Boundaries		The pull request changes only YAML, Markdown, Python, and Python test files. The authoritative diff contains no Swift changes, so the Swift package-boundary check is not applicable.
Cmux Swiftpm Lockfiles		PASS. The PR changes one workflow path filter and CI/docs/Python/tests only. It does not change a cmux-owned .gitignore, Package.swift, Package.resolved, or Xcode project package references. The…
Cmux Swift Logging		PASS: The pull request changes no Swift files. It adds or modifies Python, YAML, Markdown, and test files only, so it introduces no production Swift logging covered by the rule.
Cmux User-Facing Error Privacy		The diff changes GitHub Actions CI routing, operator diagnostics, CI documentation, and tests only. New warnings and errors from cloud_overflow_switch.py and check_repo_variables.py appear in work…
Cmux Full Internationalization		PASS. The diff changes CI workflows/scripts, tests, and operational runner documentation only. It adds no Swift or web UI text, string catalog, Info.plist, API response, rendered markdown, changelog, …
Cmux Swiftui State Layout		PASS: The pull request changes CI workflows, documentation, Python scripts, and Python tests only. The authoritative diff contains no Swift or SwiftUI files, so the SwiftUI state-layout rules do not a…
Cmux Architecture Rethink		PASS: The pull request changes only YAML, Markdown, Python, and Python test files. It contains no Swift architecture changes, so the specified Swift architectural-rethink failure conditions do not app…
Cmux Swift Auxiliary Window Close Shortcuts		PASS. The reviewed diff changes only CI workflow, documentation, Python scripts, and Python tests. It contains no Swift or window declarations, so the auxiliary-window close-shortcut rule does not app…
Cmux Source Artifacts		The diff changes only intentional workflow configuration, documentation, Python source, and tests: .github/workflows/ci-repo-variables.yml, docs/ci-runners.md, three scripts/ci files, and `tests…
Cmux No Test Or Debug Seam In Production Source		The pull request changes only YAML, Markdown, Python, and test files. The authoritative diff contains no Swift file under a production Sources/ path, so the custom check does not apply.

Full details: Cmux Cloud Persistent Session And Early Input

Explanation

PASS: The pull request changes CI runner-variable failover logic, documentation, and related tests only. The changed-file inventory contains no Cloud terminal creation, cmux-tui, Ghostty, transport, PTY, attachment, or input-handling code, so the stated cloud persistent-session rule is not applicable.

Full details: Cmux Swift Blocking Runtime

Explanation

The pull request changes Python, YAML, Markdown, and test files only. The reviewed diff contains no Swift, Objective-C, or Objective-C++ files, so it does not introduce or expand Swift blocking or timing-based synchronization.

Full details: Cmux Browser Automation Off-Main

Explanation

PASS. The reviewed diff changes CI overflow routing, runner-variable validation, documentation, and Python tests only. It introduces no browser.* commands, socket-worker routing, WebKit/AppKit access, main-actor routing, or browser automation policy tests. Therefore the browser automation off-main failure condition is not applicable.

Full details: Cmux Cache Substitution Correctness

Explanation

PASS: The pull request changes only YAML, Markdown, Python, and Python tests. It contains no production Swift, TypeScript, or JavaScript change, so the cache-substitution correctness condition is not applicable.

Full details: Cmux No Hacky Sleeps

Explanation

The PR adds runner-variable validation and failover filtering only. It does not add or expand sleep, polling, timers, backoff, or wall-clock waits. The existing time.sleep polling and cancellation waits in cloud_overflow_switch.py are unchanged from the base revision.

Full details: Cmux Algorithmic Complexity

Explanation

PASS: The pull request changes only Python CI routing, documentation, workflow metadata, and tests. The new scans in cloud_overflow_switch.py cover the fixed runner-variable maps and the two signing variables; allowed_value performs constant-size tuple membership and one label regex check. The added checks do not nest a full scan per target, rescan workspace/user-owned records, or add a hot-path sort/filter. No changed Swift, TypeScript, JavaScript, or shell algorithm introduces the stated complexity failures.

Full details: Cmux Swift Concurrency

Explanation

PASS: The reviewed diff changes only workflow YAML, documentation, Python scripts, and Python tests. It contains no Swift code, so it does not introduce or expand any Swift legacy async pattern covered by this check.

Full details: Cmux Swift `@Concurrent`

Explanation

PASS: The pull request changes only YAML, Markdown, Python, and Python test files. The authoritative diff contains no Swift files or Swift isolation changes, so the Swift @concurrent check is not applicable.

Full details: Cmux Swiftpm Lockfiles

Explanation

PASS. The PR changes one workflow path filter and CI/docs/Python/tests only. It does not change a cmux-owned .gitignore, Package.swift, Package.resolved, or Xcode project package references. Therefore, no SwiftPM lockfile requirement is triggered and no rule violation is present.

Full details: Cmux User-Facing Error Privacy

Explanation

The diff changes GitHub Actions CI routing, operator diagnostics, CI documentation, and tests only. New warnings and errors from cloud_overflow_switch.py and check_repo_variables.py appear in workflow logs and step summaries, with no concrete path to a cmux app UI, product CLI, or product API. The provider names and runner-variable details are allowed internal CI diagnostics, and the changed docs and tests are explicitly allowed surfaces. No changed text exposes credentials, tokens, personal data, or payload dumps.

Full details: Cmux Full Internationalization

Explanation

PASS. The diff changes CI workflows/scripts, tests, and operational runner documentation only. It adds no Swift or web UI text, string catalog, Info.plist, API response, rendered markdown, changelog, or locale-specific data. The documentation change is covered by the operational-docs exemption.

Full details: Cmux Swiftui State Layout

Explanation

PASS: The pull request changes CI workflows, documentation, Python scripts, and Python tests only. The authoritative diff contains no Swift or SwiftUI files, so the SwiftUI state-layout rules do not apply.

Full details: Cmux Architecture Rethink

Explanation

PASS: The pull request changes only YAML, Markdown, Python, and Python test files. It contains no Swift architecture changes, so the specified Swift architectural-rethink failure conditions do not apply.

Full details: Cmux Swift Auxiliary Window Close Shortcuts

Explanation

PASS. The reviewed diff changes only CI workflow, documentation, Python scripts, and Python tests. It contains no Swift or window declarations, so the auxiliary-window close-shortcut rule does not apply.

Full details: Cmux Source Artifacts

Explanation

The diff changes only intentional workflow configuration, documentation, Python source, and tests: .github/workflows/ci-repo-variables.yml, docs/ci-runners.md, three scripts/ci files, and tests/test_ci_cloud_overflow_switch.py. No logs, screenshots, recordings, temporary or cache directories, dependency checkouts, build output, or broad artifact directories were added.

Full details: Description check

Explanation

The description clearly explains the problem, implementation, scope, testing, and risk. It does not use the required Summary, Testing, Changelog, Proof, and Checklist sections, and it omits the required changelog and proof statements.

Resolution

Restructure the description using the repository template. Add Summary, Testing, Changelog, Proof, and Checklist sections. Include a changelog entry or state "none", and add a one-line Proof statement explaining why no screenshots or video apply.

Full details: Docstring Coverage

Explanation

Docstring coverage is 38.46% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 13 functions across 4 files. (2 skipped: 2 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🔇 Additional comments: 1

scripts/ci/check_repo_variables.py-131-132: 1

131-132: Major · Security & Privacy · Quick win

⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Check every signing variable, not only drift findings.

If MACOS_RUNNER_26 holds warp-macos-15-arm64-6x, drifted_runner_variables does not return it: the older matcher explicitly permits that label. The new allowed_value check therefore never runs, and this checker reports no problem even though the value violates the signing policy. Check the restricted variables independently before applying recorded-failover exemptions. (raw.githubusercontent.com)

---

Review info

Run configuration

• Source mode: Auto
• Repository: manaflow-ai/cmux
• Branch: dep-fleet/overflow-signing-vars-main
• File: .coderabbit.yaml
• Commit: 7005919
• Layers: Repository YAML
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 6b4a571a-5c9b-4e9d-b1a9-03a33f41b277

Commits

Reviewing files that changed from the base of the PR and between ef3d130 and 7005919.

Files selected for processing: 6

• .github/workflows/ci-repo-variables.yml
• docs/ci-runners.md
• scripts/ci/check_repo_variables.py
• scripts/ci/cloud_overflow_switch.py
• scripts/ci/signing_runner_variables.py
• tests/test_ci_cloud_overflow_switch.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Merge receipt for 70059192bf, merged 2026-10-28 23:41:54 UTC

• Not verified at merge: ci-status (not reported), GhosttyKit release check (in progress), guards (18) (in progress)
• Verified: backend migrations applied, CI fast guards, Fast static checks, merge freeze, plan, web-validation
• Skipped by policy: apply-production, apply-staging, browser, Claude request, Claude wrapper regressions, close lapsed freezes, Dogfood build #​${{ github.event.pull_request.number }}, macos, remote-daemon, suite-coverage, web, web-build, and 2 more
• Full suite: runs on main after merge.

Labeled merged-unverified: if main breaks near this merge, look here first.

[teamleaderleo]

Valid. Fixed in the follow-up (the probe-set PR from dep-fleet/overflow-probe-set): check_repo_variables.py now checks LINUX_RUNNER and MACOS_RUNNER_26 with allowed_value before the drift findings, with a test that fails on the old checker.