ci: try the nightly app compile on an owned Mac mini first, Blacksmith fallback

[teamleaderleo]

Slice 1 of #14207.

Lawrence asked on 2026-09-24 for dev builds, nightlies and CI/CD to run on the Mac minis and spill over to Blacksmith. This PR does that for the nightly's biggest job, the universal app compile. The design follows the persistent compile lane that is already merged: the mini builds, and a hosted job decides whether to use what it built.

It changes nothing until a maintainer sets NIGHTLY_MAC_MINI. While the variable is unset, the route job does not run and build-nightly-app behaves as it does today.

What moves

• nightly-mini-build.yml (new)
  • Dispatch-only, permissions: {}, no secrets.
  • Fetches public source with git fetch (no actions/checkout), and refuses a commit that is not on the ref it was dispatched from.
  • Builds the same unsigned universal Release app as build-nightly-app. Its persistent build-universal/ directory serves as the cache.
  • Runner: group cmux-nightly-mini, labels [self-hosted, macOS, ARM64, cmux-nightly-mini-build]. The label avoids the bare word nightly, which the HQ build-fleet controller reserves as a tag.
  • In a fork it runs on hosted macos-26.
• route-nightly-mini in nightly.yml
  • A Linux job that runs scripts/ci/nightly_mini_route.py, which reuses the wait and cancel code in persistent_mac_route.py.
  • It dispatches the producer, waits up to NIGHTLY_MAC_MINI_QUEUE_SECONDS (300) for a mini to pick it up and up to NIGHTLY_MAC_MINI_EXECUTION_SECONDS (2700) for the build, then copies the products into this run.
  • On a queue timeout, an overrun, a producer failure or a missing artifact, it cancels the request and reports a fallback reason.
• build-nightly-app
  • A new Adopt owned-Mac products step uses the products only if the source SHA, the tree, the full xcodebuild -version and the archs all match. Otherwise it compiles hosted.
  • When it adopts, it skips the compile and cache steps and runs on MACOS_RUNNER_26 instead of the 12 vCPU machine.
  • The job runs on !cancelled(), so a failed route job changes nothing.
• Guard
  • New check_nightly_mini_lane: the producer must be dispatch-only, credential-free and on the exact runner line, the sign and publish jobs must not reference it, and the adoption checks must be present.
  • check_no_self_hosted_fleet_runners gains one exact-line exemption, and cmux-nightly-mini joins its fleet regex, so no other file can name the group.
• Docs: a "Nightly lane" section in docs/ci/mac-fleet.md, plus updates to ci-runners.md and CLAUDE.md.

NIGHTLY_MAC_MINI values

Value	Effect
unset	no change
build-only	unsigned build_only runs try a mini first
all	published nightlies also compile on a mini first; signing, notarization and publication stay hosted

A manual build_only dispatch can also pass mac_mini: true.

What does not move, and why

• Signing, notarization, Sparkle and publication. The signing keys never reach a mini.
• all is a trust decision for Leo and Manaflow, not a routing one. Setting it means Developer ID signs bits compiled on a persistent machine that keeps warm state between runs. The adoption check proves which commit and toolchain the producer reports; it does not prove the warm state was clean. build-only ships nothing and is safe as soon as the runner exists.
• The iOS hourly TestFlight uploads, App Store upload, release.yml and the Iroh release gate are untouched.

Before anything runs on a mini (org admin)

1. Create the runner group cmux-nightly-mini, restricted to manaflow-ai/cmux/.github/workflows/nightly-mini-build.yml@refs/heads/main.
2. Register one M4 Pro mini in that group with the labels above and the Xcode named by CMUX_CI_XCODE_APP_MACOS_26.
   • glaeda Up/down arrow keys not working in browser surface #1146's runner registers glaeda-mini, which is the wrong label for this group.
   • scripts/persistent-compile up registers the compile lane's labels, not these.
3. gh variable set NIGHTLY_MAC_MINI -b build-only, then dispatch nightly.yml with build_only. Read the "Nightly owned-Mac route" summary.
4. Rollback: gh variable delete NIGHTLY_MAC_MINI.

Validation

• Local:
  • tests/test_ci_self_hosted_guard.sh passes. It fails when the producer's runner line drifts; I checked that by editing the label.
  • tests/test_nightly_mini_route.py covers eligibility, queue timeout with cancel, producer failure, overrun, missing artifact, success and an unobservable dispatch. All 11 cases pass.
  • test_ci_change_areas, test_ci_persistent_mac_compile, test_runner_label_policy, test_ci_workflow_guards_are_wired, actionlint and scripts/verify-local.py all pass.
• Fork: runs on teamleaderleo/cmux are linked in a comment below.
  • The fork can only exercise the fallback path. GitHub will not dispatch a workflow that is not on the default branch, so there, and on this repository until merge, the route falls back with routing_error.
  • A real mini success needs the runner group and one registered mini (step 3 above).

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Runs the nightly app compile on an owned Mac mini first, with Blacksmith as automatic fallback. Dark by default: nothing changes until a maintainer sets NIGHTLY_MAC_MINI, whose values admit only unsigned build_only runs (build-only) or also published nightlies (all).

• New dispatch-only producer in nightly-mini-build.yml fetches public source without checkout credentials and builds the unsigned universal app, keeping warm state across runs. It rebuilds Products each run so stale resources or dSYMs cannot ship.
• route-nightly-mini dispatches it, waits bounded queue and execution times, and cancels and reports a fallback reason on timeout, overrun, failure or missing artifact. A dispatch the listing shows late is found and cancelled rather than left holding a mini.
• build-nightly-app adopts the products only when source SHA, tree, xcodebuild -version, macOS SDK and archs all match; otherwise it compiles hosted exactly as before, as does any other route outcome. Re-home steps continue on error and a non-numeric budget falls back.
• Signing, notarization, Sparkle and publication stay on hosted runners; signing keys never reach a mini. Those jobs gate on !cancelled() and explicit results so a skipped route cannot stop them, and check_nightly_mini_lane plus the updated universal build contract test enforce that shape.

Migration

• Org admin creates runner group cmux-nightly-mini, restricted to nightly-mini-build.yml on main, registers one M4 Pro mini with cmux-nightly-mini-build, then sets NIGHTLY_MAC_MINI to build-only.
• all is a trust decision for maintainers, not addressed by this change: it means Developer ID signs bits compiled on a machine with warm state across runs. Rollback is gh variable delete NIGHTLY_MAC_MINI.

^{Written for commit 648e754. Summary will update on new commits.}

Summary by CodeRabbit

• New Features
  • Nightly app compilation can optionally run on an owned Mac mini. Hosted runners verify the build before signing and publishing, and take over if the mini build is unavailable or fails.
  • A manual build-only workflow can produce an unsigned nightly app for download as a short-lived artifact.
• Documentation
  • Updated CI guidance with nightly build routing, fallback behavior, and configuration details.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: c386f821-019b-4dc0-8d80-8a6682c81041

📥 Commits

Reviewing files that changed from the base of the PR and between f9b1a13 and 648e754.

📒 Files selected for processing (13)

• .github/workflows/ci-guards.yml
• .github/workflows/nightly-mini-build.yml
• .github/workflows/nightly.yml
• CLAUDE.md
• docs/ci-runners.md
• docs/ci/mac-fleet.md
• scripts/ci/detect_ci_change_areas.py
• scripts/ci/nightly_mini_route.py
• scripts/ci/workflow_guard_groups.py
• tests/test-execution.toml
• tests/test_ci_self_hosted_guard.sh
• tests/test_nightly_mini_route.py
• tests/test_nightly_universal_build.sh

 _______________________________
< Grow a pair... of test cases. >
 -------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.