ci: build the nightly app on cmux15's trusted runner first, Blacksmith as fallback

.github/workflows/ci-health-report.yml

@@ -87,7 +87,9 @@ jobs:
           # every read; `macos-15` is an approved label either way.
           # CI_PR_POOL_ORDER is the one list that may also name owned pools.
           CMUX_CI_RUNNER_VARIABLES: |
+            CI_NIGHTLY_TRUSTED_RUNNER=${{ vars.CI_NIGHTLY_TRUSTED_RUNNER }}
             CI_PR_POOL_ORDER=${{ vars.CI_PR_POOL_ORDER }}
+            CI_SEED_TRUSTED_POOL=${{ vars.CI_SEED_TRUSTED_POOL }}
             CI_SIDE_LANE_RUNNER=${{ vars.CI_SIDE_LANE_RUNNER }}
             CMUX_CI_RUNNER_FLEET=${{ vars.CMUX_CI_RUNNER_FLEET }}
             CMUX_CI_RUNNER_OVERRIDES=${{ vars.CMUX_CI_RUNNER_OVERRIDES }}

.github/workflows/ci-owned-pool-rescue.yml

@@ -37,6 +37,11 @@ run-name: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow
 #   terminal-hang-diagnostics) have no picker: while vars.CI_SIDE_LANE_RUNNER
 #   names a glaeda-side-* label, every same-repository attempt-1 pull request
 #   run of theirs is on the fleet, so a dispatch would save no run.
+# - nightly.yml: build-nightly-app has no picker either. While
+#   vars.CI_SEED_TRUSTED_POOL names a glaeda-trusted-* pool and
+#   vars.CI_NIGHTLY_TRUSTED_RUNNER a glaeda-runner-* label, attempt 1 of every
+#   push or schedule run on main asks for that one trusted mini, so every such
+#   run is watched (owned_pool_rescue.NIGHTLY_WORKFLOW_PATH).
 # It needs actions: write, and its code comes from main. It runs whenever
 # owned pools are on (CI_PR_POOL_OWNED is 1), because a run on an owned pool
 # has no other way off it; CI_OWNED_POOL_RESCUE=0 turns it off. The switch
@@ -58,6 +63,9 @@ on:
       - IROH v2
       - Relay TLS system trust
       - Terminal hang diagnostics
+      # nightly.yml: its app build takes the trusted owned pool on attempt 1
+      # of main's push and schedule runs (owned_pool_rescue.NIGHTLY_WORKFLOW_PATH).
+      - Nightly macOS build
     types: [requested]
   workflow_dispatch:
     inputs:
@@ -89,6 +97,7 @@ env:
     .github/workflows/cloud-machine-tests.yml .github/workflows/cloud-task-local-tests.yml
     .github/workflows/iroh-v2.yml .github/workflows/relay-tls.yml
     .github/workflows/terminal-hang-diagnostics.yml
+    .github/workflows/nightly.yml
 
 concurrency:
   group: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' && !inputs.run_id) && 'owned-pool-sweeper' || format('owned-pool-rescue-{0}-{1}', inputs.run_id || github.event.workflow_run.id, inputs.run_attempt || github.event.workflow_run.run_attempt) }}
@@ -98,7 +107,7 @@ concurrency:
 jobs:
   rescue:
     name: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' && !inputs.run_id) && 'Sweep runs on persistent pools' || 'Rescue a run stuck on a persistent pool' }}
-    if: ${{ vars.CI_PR_POOL_OWNED == '1' && (vars.CI_OWNED_POOL_RESCUE || '1') != '0' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event.workflow_run.event == 'pull_request' && startsWith(vars.CI_SIDE_LANE_RUNNER, 'glaeda-side-') || github.event.workflow_run.path == '.github/workflows/ios-screenshots.yml' && github.event.workflow_run.event == 'workflow_dispatch') && github.event.workflow_run.head_repository.full_name == github.repository && github.event.workflow_run.run_attempt == 1) }}
+    if: ${{ vars.CI_PR_POOL_OWNED == '1' && (vars.CI_OWNED_POOL_RESCUE || '1') != '0' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event.workflow_run.event == 'pull_request' && startsWith(vars.CI_SIDE_LANE_RUNNER, 'glaeda-side-') || github.event.workflow_run.path == '.github/workflows/ios-screenshots.yml' && github.event.workflow_run.event == 'workflow_dispatch' || github.event.workflow_run.path == '.github/workflows/nightly.yml' && (github.event.workflow_run.event == 'push' || github.event.workflow_run.event == 'schedule') && github.event.workflow_run.head_branch == 'main' && startsWith(vars.CI_SEED_TRUSTED_POOL, 'glaeda-trusted-') && startsWith(vars.CI_NIGHTLY_TRUSTED_RUNNER, 'glaeda-runner-')) && github.event.workflow_run.head_repository.full_name == github.repository && github.event.workflow_run.run_attempt == 1) }}
     runs-on: ubuntu-24.04 # github-hosted-required: polls the Actions API; keeps CI's Linux pool free
     # A sweeper adopts runs for 300 minutes and gives a rescue 25 more to
     # settle (SWEEP_SECONDS, RESCUE_GRACE_SECONDS). A single run's watch is

.github/workflows/ci-repo-variables.yml

@@ -53,7 +53,9 @@ jobs:
           CI_OWNED_POOL_SLOTS: ${{ vars.CI_OWNED_POOL_SLOTS }}
           CMUX_CI_XCODE_APP_PR: ${{ vars.CMUX_CI_XCODE_APP_PR }}
           CMUX_CI_RUNNER_VARIABLES: |
+            CI_NIGHTLY_TRUSTED_RUNNER=${{ vars.CI_NIGHTLY_TRUSTED_RUNNER }}
             CI_PR_POOL_ORDER=${{ vars.CI_PR_POOL_ORDER }}
+            CI_SEED_TRUSTED_POOL=${{ vars.CI_SEED_TRUSTED_POOL }}
             CI_SIDE_LANE_RUNNER=${{ vars.CI_SIDE_LANE_RUNNER }}
             CMUX_CI_RUNNER_FLEET=${{ vars.CMUX_CI_RUNNER_FLEET }}
             CMUX_CI_RUNNER_OVERRIDES=${{ vars.CMUX_CI_RUNNER_OVERRIDES }}

.github/workflows/nightly.yml

@@ -903,13 +903,32 @@ jobs:
       daemon_build: ${{ steps.remote_daemon.outputs.build }}
       daemon_version: ${{ steps.remote_daemon.outputs.version }}
     if: needs.decide.outputs.should_build == 'true' && (github.event_name != 'schedule' || github.event.schedule == '47 8 * * *')
-    # Full runs share the cache warmer's and stable release lane's image and
-    # toolchain, which is what the compilation cache is keyed on — OS, arch and
-    # toolchain, never instance size, which is deliberately larger here. Fast
+    # Owned minis first. Attempt 1 of a push or schedule run on main takes one
+    # trusted owned mini while CI_PR_POOL_OWNED is 1 and both variables below
+    # are set: runs-on asks for the trusted pool label (vars.CI_SEED_TRUSTED_POOL,
+    # glaeda-trusted-<class>-xcode-<version>) and that runner's own label
+    # (vars.CI_NIGHTLY_TRUSTED_RUNNER, glaeda-runner-<runner name>), so only a
+    # runner carrying both can take it. Today that is cmux15-glaeda: the
+    # trusted mini with no pull request runners and no dev-build worker. The
+    # other trusted mini (cmuxs-mac-mini-6) builds team dev builds as the same
+    # user, so it keeps seeding but never builds the shipped app. Its
+    # job-started hook admits only main's own push and schedule jobs. Not the
+    # pull request pool: this job holds the ci-cache-writer R2 keys and the
+    # Sentry token, and its app is what build-sign-notarize-nightly signs and
+    # ships. runner_label_policy.py holds both variables to those shapes.
+    # ci-owned-pool-rescue.yml watches the run (owned_pool_rescue.py,
+    # NIGHTLY_WORKFLOW_PATH): when the mini does not take the job within its
+    # budget, or refuses it at job start, the failed jobs are re-run, and
+    # every later attempt takes Blacksmith below. Either variable empty is
+    # Blacksmith. Signing stays on Blacksmith (build-sign-notarize-nightly).
+    # Blacksmith runs share the cache warmer's and stable release lane's image
+    # and toolchain, which is what the compilation cache is keyed on — OS, arch
+    # and toolchain, never instance size, which is deliberately larger here.
+    # An owned mini keys its own lineage (see the cache key step). Fast
     # branch dogfood always uses the dedicated Blacksmith image. A
     # repository-wide runner override may point at a slower shared builder,
     # which defeats the purpose of the one-architecture path.
-    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'macos-26' || (needs.decide.outputs.fast_build == 'true' && 'blacksmith-12vcpu-macos-26' || vars.CI_PAID_MACOS_OVERFLOW == '1' && vars.MACOS_RUNNER_26_LARGE || 'blacksmith-12vcpu-macos-26') }}
+    runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'macos-26' || needs.decide.outputs.fast_build != 'true' && github.run_attempt == 1 && (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main' && vars.CI_PR_POOL_OWNED == '1' && vars.CI_SEED_TRUSTED_POOL != '' && vars.CI_NIGHTLY_TRUSTED_RUNNER != '' && fromJSON(format('["{0}", "{1}"]', vars.CI_SEED_TRUSTED_POOL, vars.CI_NIGHTLY_TRUSTED_RUNNER)) || (needs.decide.outputs.fast_build == 'true' && 'blacksmith-12vcpu-macos-26' || vars.CI_PAID_MACOS_OVERFLOW == '1' && vars.MACOS_RUNNER_26_LARGE || 'blacksmith-12vcpu-macos-26') }}
     environment: ${{ github.ref == 'refs/heads/main' && 'ci-cache-writer' || '' }}
     # The Blacksmith cache is scoped per branch and also drops main's own
     # entry (runs 35179030871 and 35182663752 restored nothing and were
@@ -941,11 +960,28 @@ jobs:
       - name: Select Xcode
         run: ./scripts/select-ci-xcode.sh
 
+      # A persistent owned mini keeps the workspace between jobs. Checkout's
+      # clean already drops untracked output; clear the build tree by name as
+      # well, so a stale product or dSYM from an earlier job can never ship.
+      - name: Clear build outputs a persistent runner kept
+        run: scripts/ci/clear-dirs.sh build-universal remote-daemon-assets
+
       - name: Compute Xcode compilation cache key
         id: compilation-cache-key
         run: |
           set -euo pipefail
-          echo "toolchain=$(xcodebuild -version | shasum -a 256 | awk '{print $1}')" >> "$GITHUB_OUTPUT"
+          # The key is OS, arch and toolchain. An owned mini (runner
+          # <member>-glaeda[-K]) runs the same Xcode build as Blacksmith but
+          # compiles in another workspace path, so its cache entries would
+          # mostly miss there and push Blacksmith's own entry out of the
+          # prefix restore. Fold its workspace path into the toolchain hash:
+          # each lineage restores only its own, and Blacksmith's key is
+          # unchanged.
+          identity="$(xcodebuild -version)"
+          case "${RUNNER_NAME:-}" in
+            *-glaeda | *-glaeda-[0-9]*) identity="$identity"$'\n'"owned-workspace:$GITHUB_WORKSPACE" ;;
+          esac
+          echo "toolchain=$(printf '%s\n' "$identity" | shasum -a 256 | awk '{print $1}')" >> "$GITHUB_OUTPUT"
 
       - name: Restore Xcode compilation cache
         id: compilation-cache-restore

docs/ci-runners.md

@@ -644,11 +644,28 @@ root) stay on Blacksmith. Clear the variable to send every side lane back.
 
 Owned minis run macOS 26 with Xcode 26.6 only, run same-repository pull
 request code, and keep their home directory and caches between jobs. So a job
-stays on Blacksmith when it signs, notarizes, uploads or publishes (anything
-with signing, store or release secrets, or whose output ships or seeds a
-shared cache), when it runs fork code, or when it needs an OS or Xcode the
-minis lack. Everything else routes through a picker, with Blacksmith as the
-overflow and ci-owned-pool-rescue.yml as the way off a busy or refusing mini.
+stays off the pull request pools when it signs, notarizes, uploads or
+publishes (anything with signing, store or release secrets, or whose output
+ships or seeds a shared cache), when it runs fork code, or when it needs an OS
+or Xcode the minis lack. Everything else routes through a picker, with
+Blacksmith as the overflow and ci-owned-pool-rescue.yml as the way off a busy
+or refusing mini.
+
+The trusted pool (`vars.CI_SEED_TRUSTED_POOL`,
+`glaeda-trusted-<class>-xcode-<version>`) is the owned home for main's own
+cache writers and builds: minis with no pull request runners, whose
+job-started hook admits only a push or schedule run on main. The DerivedData
+seed takes it on every main push. The nightly app compile takes one runner of
+it first, `vars.CI_NIGHTLY_TRUSTED_RUNNER` (`glaeda-runner-cmux15-glaeda`):
+runs-on asks for the pool label and that runner's own label together, so it
+never lands on cmuxs-mac-mini-6, whose dev-build worker builds team code as the
+same user. Either variable empty sends it to Blacksmith, which is also its
+fallback. `runner_label_policy.py` refuses any other shape for either
+variable. glaeda classes the job `isolated` (teamleaderleo/glaeda#1287), so it
+never holds the canonical root a seed on the same mini waits for. Signing and
+notarization are not on it: no signing run on an owned Mac has been proven,
+and the retired self-hosted fleet failed `codesign` with
+`errSecInternalComponent` (#6264).
 
 | Jobs | Route | Why |
 | --- | --- | --- |
@@ -664,8 +681,11 @@ overflow and ci-owned-pool-rescue.yml as the way off a busy or refusing mini.
 | low-volume dispatches: `test-macos-suite`, `tmux-corpus`, `perf-activation`, command palette benchmarks, `iroh-release-gate` version skew | Blacksmith or the caller's runner input | a few runs a week; benchmarks want a quiet machine |
 | `relay-tls` `system-keychain` | Blacksmith | edits the System keychain trust store |
 | `plain-paste-worker`, `ci-macos-compat`, `seed-swiftpm-manifests`, release and nightly Ghostty helpers | Blacksmith macOS 15 / 14 | an OS or SDK the minis lack |
-| `release.yml`, nightly sign/notarize, `ios-testflight`, `ios-app-store`, `ios-appstore-upload` | Blacksmith | signing and store secrets |
-| nightly app and compilation caches, `seed-derived-data` Blacksmith pools, `build-ghosttykit`, `cmux-tui-build-package` (artifacts, nightly, release), `relay-publish-npm` | Blacksmith | publish, or write a cache other runs trust, with R2 or release secrets |
+| `release.yml`, nightly sign/notarize, `ios-testflight`, `ios-app-store`, `ios-appstore-upload` | Blacksmith | signing and store secrets; signing on an owned Mac is unproven |
+| `nightly.yml` `build-nightly-app` | one trusted runner (`CI_SEED_TRUSTED_POOL` plus `CI_NIGHTLY_TRUSTED_RUNNER`, cmux15) on attempt 1 of main's push and schedule runs; Blacksmith 12 vCPU otherwise, for `rc/**`, dispatches and fast dogfood, and on every re-run | ci-owned-pool-rescue.yml watches it (`NIGHTLY_WORKFLOW_PATH`): stuck one queue round past `CI_OWNED_POOL_RESCUE_SECONDS`, or refused, its failed jobs re-run on Blacksmith. Its compilation cache keys its own lineage (the mini's workspace path) |
+| `seed-derived-data` trusted pool | trusted owned pool, push to main | the minis' own j14 seed |
+| `nightly.yml` `refresh-compilation-cache`, `refresh-test-compilation-cache`, `seed-derived-data` Blacksmith pools | Blacksmith | they seed Blacksmith's own lanes: the release cache the nightly fallback restores, and the pull request admission seeds for each Blacksmith pool |
+| `build-ghosttykit`, `cmux-tui-build-package` (artifacts, nightly, release), `relay-publish-npm` | Blacksmith | publish with R2 or release secrets |
 | `ios-streamed-validate`, `iroh-release-gate` simulator E2E | Blacksmith | secrets in the job, fixed ports, GUI session changes |
 
 ## Retired: Tart VM fleet

docs/ci/mac-fleet.md

@@ -216,8 +216,10 @@ them (section 5).
    job gets a fresh VM and an Aqua login session. A shared mini cannot give
    it either. (The isolated Tart pool that once offered this was retired on
    2026-09-25; see `ci-runners.md`.)
-4. **Nightly app compile** - nightlies run on Blacksmith until Glaeda routing
-   (glaeda#1174) sends every job std > light > Blacksmith > GitHub-hosted.
+4. **Nightly app compile** - `build-nightly-app` takes the trusted runner on cmux15
+   (`CI_SEED_TRUSTED_POOL` plus `CI_NIGHTLY_TRUSTED_RUNNER`) first on main's push and schedule runs, with
+   Blacksmith as the fallback through ci-owned-pool-rescue.yml
+   (`docs/ci-runners.md`). Signing and notarization stay on Blacksmith.
 5. **`release-build`, signing, notarization, TestFlight** - never.
    Unchanged from `ci-runners.md`.