Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/actionlint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ self-hosted-runner:
# macOS 26 needs route to Blacksmith cloud, not warp-macos-26-arm64-6x: our
# self-hosted minis carry that label, and GitHub prefers a matching
# self-hosted runner. See check_no_self_hosted_fleet_runners.
# Manual E2E canary only. Required CI remains routed through repo variables.
- tart-canary
# Dispatch-only compile-admission producer; guarded separately from required jobs.
- cmux-persistent-macos-compile
# Linux: Blacksmith primary (LINUX_RUNNER), WarpBuild overflow fallback.
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/cmux-tui-artifacts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ on:
inputs:
macos_runner:
# A branch dogfood publish only needs its commit-addressed objects; let
# it build the macOS targets on an idle self-hosted label (for example
# tart-macos-15) instead of waiting behind the shared hosted queue.
# it build the macOS targets on an idle cloud label (for example
# blacksmith-6vcpu-macos-15) instead of waiting behind the hosted queue.
description: "Optional macOS runner label override for this publish"
required: false
default: ""
Expand Down
43 changes: 1 addition & 42 deletions .github/workflows/test-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ on:
default: true
type: boolean
runner:
description: "Runner OS (auto follows MACOS_RUNNER_TESTS; tart-* choices use isolated VMs)"
description: "Runner OS (auto follows MACOS_RUNNER_TESTS)"
required: false
default: "auto"
type: choice
Expand All @@ -39,9 +39,6 @@ on:
- blacksmith-6vcpu-macos-26
- blacksmith-12vcpu-macos-26
- blacksmith-6vcpu-macos-latest
- tart-canary
- tart-dual
- tart-small

concurrency:
group: e2e-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_TESTS || 'blacksmith-6vcpu-macos-26') || inputs.runner }}-${{ inputs.ref || github.ref_name }}-${{ inputs.test_filter }}
Expand Down Expand Up @@ -217,25 +214,6 @@ jobs:
CMUX_PRODUCT_RUNNER: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_TESTS || 'blacksmith-6vcpu-macos-26') || inputs.runner }}
steps:

- name: Validate Tart canary identity
if: ${{ startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_TESTS || 'blacksmith-6vcpu-macos-26') || inputs.runner, 'tart-') }}
env:
REQUESTED_RUNNER: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_TESTS || 'blacksmith-6vcpu-macos-26') || inputs.runner }}
RUNNER_CONTEXT_NAME: ${{ runner.name }}
run: |
set -euo pipefail
case "$RUNNER_CONTEXT_NAME" in
tart-cmux-*) ;;
*)
echo "::error::$REQUESTED_RUNNER resolved to unexpected runner $RUNNER_CONTEXT_NAME"
exit 1
;;
esac
test -f /etc/cmux-tart-ci || {
echo "::error::$REQUESTED_RUNNER runner is missing the immutable VM identity marker"
exit 1
}

- name: Clear stale git locks (self-hosted reused workspace)
shell: bash
run: |
Expand Down Expand Up @@ -641,25 +619,6 @@ jobs:
CMUX_APP_HOST_CAPTURE_XCRESULTS: "1"
steps:

- name: Validate Tart canary identity
if: ${{ startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_TESTS || 'blacksmith-6vcpu-macos-26') || inputs.runner, 'tart-') }}
env:
REQUESTED_RUNNER: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_TESTS || 'blacksmith-6vcpu-macos-26') || inputs.runner }}
RUNNER_CONTEXT_NAME: ${{ runner.name }}
run: |
set -euo pipefail
case "$RUNNER_CONTEXT_NAME" in
tart-cmux-*) ;;
*)
echo "::error::$REQUESTED_RUNNER resolved to unexpected runner $RUNNER_CONTEXT_NAME"
exit 1
;;
esac
test -f /etc/cmux-tart-ci || {
echo "::error::$REQUESTED_RUNNER runner is missing the immutable VM identity marker"
exit 1
}

- name: Clear stale git locks (self-hosted reused workspace)
shell: bash
run: |
Expand Down
48 changes: 1 addition & 47 deletions .github/workflows/test-ios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,13 @@ on:
- iphone
- ipad
runner:
description: "macOS runner (auto follows MACOS_RUNNER_IOS; tart-ios uses the isolated VM fleet)"
description: "macOS runner (auto follows MACOS_RUNNER_IOS)"
required: false
default: "auto"
type: choice
options:
- auto
- blacksmith-6vcpu-macos-26
- tart-ios
cache_backend:
description: "Cache store for this run. default follows CI_CACHE_BACKEND."
required: false
Expand Down Expand Up @@ -225,21 +224,6 @@ jobs:
*) echo "::error::Unsupported Swift package: $SELECTED_PACKAGE"; exit 1 ;;
esac

- name: Validate Tart runner identity
if: ${{ startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_IOS || 'blacksmith-6vcpu-macos-26') || inputs.runner, 'tart-') }}
env:
RUNNER_CONTEXT_NAME: ${{ runner.name }}
run: |
set -euo pipefail
[[ "$RUNNER_CONTEXT_NAME" == tart-cmux-* ]] || {
echo "::error::tart-ios resolved to unexpected runner $RUNNER_CONTEXT_NAME"
exit 1
}
test -f /etc/cmux-tart-ci || {
echo "::error::tart-ios runner is missing the immutable VM identity marker"
exit 1
}

- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
Expand Down Expand Up @@ -387,21 +371,6 @@ jobs:
package_seconds: ${{ steps.package-product.outputs.seconds }}
upload_seconds: ${{ steps.upload-metrics.outputs.seconds }}
steps:
- name: Validate Tart runner identity
if: ${{ startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_IOS || 'blacksmith-6vcpu-macos-26') || inputs.runner, 'tart-') }}
env:
RUNNER_CONTEXT_NAME: ${{ runner.name }}
run: |
set -euo pipefail
[[ "$RUNNER_CONTEXT_NAME" == tart-cmux-* ]] || {
echo "::error::tart-ios resolved to unexpected runner $RUNNER_CONTEXT_NAME"
exit 1
}
test -f /etc/cmux-tart-ci || {
echo "::error::tart-ios runner is missing the immutable VM identity marker"
exit 1
}

- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
Expand Down Expand Up @@ -693,21 +662,6 @@ jobs:
matrix:
family: ${{ fromJSON(needs.detect-ios-changes.outputs.device_families) }}
steps:
- name: Validate Tart runner identity
if: ${{ startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_IOS || 'blacksmith-6vcpu-macos-26') || inputs.runner, 'tart-') }}
env:
RUNNER_CONTEXT_NAME: ${{ runner.name }}
run: |
set -euo pipefail
[[ "$RUNNER_CONTEXT_NAME" == tart-cmux-* ]] || {
echo "::error::tart-ios resolved to unexpected runner $RUNNER_CONTEXT_NAME"
exit 1
}
test -f /etc/cmux-tart-ci || {
echo "::error::tart-ios runner is missing the immutable VM identity marker"
exit 1
}

- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
Expand Down
72 changes: 13 additions & 59 deletions docs/ci-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,9 @@ Every CI/CD job picks its runner from a repository variable instead of a
hardcoded label. Changing a runner type is a single repository-variable update
that takes effect on the next workflow run.

Linux uses Blacksmith. macOS uses Blacksmith cloud runners, with the
self-hosted Tart fleet described below carrying specific lanes as they are
qualified. WarpBuild is paid overflow and is not a steady state for any lane.
Non-urgent macOS work runs on free GitHub-hosted runners through the
background lane described below.
Linux uses Blacksmith. macOS uses Blacksmith cloud runners. WarpBuild is paid
overflow and is not a steady state for any lane. Non-urgent macOS work runs on
free GitHub-hosted runners through the background lane described below.

**The table below is the intended steady state, not a live readout.** Repository
variables drift, and a stale table is worse than no table. For what is actually
Expand Down Expand Up @@ -268,31 +266,11 @@ compile, warning validation, product publication, total wall time, runner time,
and the `hot` / `partially-warm` / `cold-reset` / `hosted fallback`
classification.

## Tart isolation and capacity

Each GitHub runner identity is sealed into a Tart template. A job runs in a
fresh clone with an Aqua login session, then the host deletes the clone. This
provides the GUI session required by macOS XCTest and prevents DerivedData,
simulators, credentials, and workspaces from leaking into later jobs.

The fleet has 18 Sequoia slots: two each on the seven 48 GB or larger hosts and
one each on the two 16 GB hosts. The 16 large-host slots accept GUI and iOS
jobs; all 18 accept ordinary macOS 15 jobs. macOS 26 and release builds stay on
Blacksmith until a Tahoe VM image passes the same runner and GUI canaries. Hosts
reject new jobs below their free-space threshold, delete every job VM after
use, and reap stale clones.

Do not route jobs to the physical mini runner records. The supported
self-hosted labels are the `tart-*` labels, and each Tart-aware canary checks
that the resolved runner name starts with `tart-cmux-` and that the guest has
the immutable `/etc/cmux-tart-ci` marker.

## Shared physical-host interoperability

The current required-CI policy continues to use isolated Tart guests or hosted
providers. Any future path that executes directly on shared CMUX-owned hardware
must preserve a separate caller identity, semantic workload request, and
machine-local physical lease.
The current required-CI policy uses hosted providers. Any future path that
executes directly on shared CMUX-owned hardware must preserve a separate caller
identity, semantic workload request, and machine-local physical lease.

Examples of callers that may share a host include GitHub Actions, `cmux-ci`,
developer/build tooling, direct agents, operator commands, and reviewed fleet
Expand Down Expand Up @@ -321,9 +299,8 @@ admission or is draining, pressured, or unavailable.

## Break-glass: switch a runner type to a paid provider

There is no automatic overflow. If the Tart pool is unavailable or its queue is
too long, set the affected variable to a paid provider. Restore Tart after the
fleet recovers.
There is no automatic overflow. If the Blacksmith queue is too long, set the
affected variable to a paid provider, and restore it once the queue recovers.

Four runner variables exist to name **metered WarpBuild capacity**, so they are
read through a second switch that lives in this repository rather than in
Expand Down Expand Up @@ -371,26 +348,6 @@ Leave `MACOS_RUNNER_PR` and `MACOS_RUNNER_TESTS` unset in either recipe.
They exist to hold the pull-request and manual test lanes on Blacksmith
independently of whatever the pool above is set to.

Restore the self-hosted pool with explicit labels. The gate above applies
here too: `MACOS_RUNNER_15`, `MACOS_RUNNER_DISPLAY` and the other gated
variables are read only when `CI_PAID_MACOS_OVERFLOW=1`, so Tart needs that
flag set even though Tart is free. Without it, these values are ignored and
every lane stays on its Blacksmith fallback, with no error. `MACOS_RUNNER_26`
is ungated, so repointing the ordinary macOS 26 pool does not require the paid
overflow switch.

```bash
gh variable set MACOS_RUNNER_15 --repo manaflow-ai/cmux -b tart-macos-15
gh variable set MACOS_RUNNER_DUAL_XCODE --repo manaflow-ai/cmux -b blacksmith-6vcpu-macos-15
gh variable set MACOS_RUNNER_26 --repo manaflow-ai/cmux -b blacksmith-6vcpu-macos-26
gh variable set MACOS_RUNNER_26_LARGE --repo manaflow-ai/cmux -b blacksmith-12vcpu-macos-26
gh variable set MACOS_RUNNER_DISPLAY --repo manaflow-ai/cmux -b tart-gui
gh variable set MACOS_RUNNER_IOS --repo manaflow-ai/cmux -b tart-ios
```

`MACOS_RUNNER_DUAL_XCODE` remains on Blacksmith because the Tart macOS 15
image currently carries Xcode 26 only and cannot build the SDK 15 helper.

Check current values:

```bash
Expand All @@ -404,9 +361,7 @@ defaults to `auto`. Manual `auto` runs follow `MACOS_RUNNER_15` then the Blacksm
fallback, so flipping the repo variable redirects those workflows. An explicit
manual choice wins over the variable; both dropdowns expose Blacksmith, Warp,
and `depot-macos-*` choices, with a Depot identity guard for GUI-activation
runs. `test-e2e.yml` also exposes `tart-canary`, `tart-dual`, and `tart-small`
for targeted fleet validation. These choices are available only through
`workflow_dispatch`.
runs. These choices are available only through `workflow_dispatch`.

## Guard

Expand All @@ -426,9 +381,9 @@ repository per minute, since Blacksmith is sponsored for this organization.
The CI health report counts those two. Keep new labels in
`.github/actionlint.yaml`.

The fleet-label guard allows Tart labels only as exact manual canary choices.
Required jobs continue to reference repository variables, so cutover and
break-glass remain configuration changes instead of workflow edits.
The fleet-label guard rejects `tart-*` labels everywhere; the Tart VM pool no
longer exists. Required jobs continue to reference repository variables, so
cutover and break-glass remain configuration changes instead of workflow edits.

## CMUX-owned machine enrollment

Expand Down Expand Up @@ -459,5 +414,4 @@ carries no repository secrets, and grants its hot state zero result authority.
Every required macOS fallback still routes to the paid hosted path.
`check_no_self_hosted_fleet_runners` in
`tests/test_ci_self_hosted_guard.sh` enforces that exact exception and rejects
any second required-job or generic fleet route. Repository variables may keep
pointing at the isolated `tart-*` pool for their existing jobs.
any second required-job or generic fleet route.
11 changes: 5 additions & 6 deletions docs/ci/mac-fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This document is the capacity and operations layer. It does not restate the
routing contract, which already exists:

- [`ci-runners.md`](../ci-runners.md) owns the runner-variable table, the
persistent compile-admission pilot contract, the Tart pool, and the
persistent compile-admission pilot contract and the
direct-physical-host boundary.
- [`fleet-enrollment.md`](../fleet-enrollment.md) owns machine onboarding.
- [`workload-profiles.md`](../workload-profiles.md) owns workload identity.
Expand Down Expand Up @@ -197,9 +197,9 @@ them (section 5).
owned-Mac lane is a deliberate guard edit, not an accident.
3. **`app-host unit tests`** - do **not** move to minis, despite being 55% of
the minutes. It needs a foreground GUI session, it is six shards of
XCTest, and it is a required check. Its home is the isolated Tart pool
(18 slots, `ci-runners.md`), where each job gets a fresh VM clone and an
Aqua login session. A shared mini cannot give it either.
XCTest, and it is a required check. Its home is the cloud macOS pool, where
each job gets a fresh machine and an Aqua login session. A shared mini
cannot give it either.
4. **`release-build`, signing, notarization, nightly, TestFlight** - never.
Unchanged from `ci-runners.md`.

Expand Down Expand Up @@ -514,8 +514,7 @@ There is no macOS ephemeral-runner primitive anywhere in either repository.
The honest statement of this design is: **the macOS fleet is a persistent,
credential-minimized, artifact-producing machine whose output carries no
authority, not an ephemeral runner.** If per-job macOS isolation is ever
required, the existing Tart pool provides it (fresh VM clone per job, deleted
after) and is where that requirement belongs.
required, the cloud macOS pools provide it (a fresh machine per job).

## 5. Rollout

Expand Down
3 changes: 0 additions & 3 deletions scripts/ci/dispatch-focused-test.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,6 @@
"blacksmith-6vcpu-macos-26",
"blacksmith-12vcpu-macos-26",
"blacksmith-6vcpu-macos-latest",
"tart-canary",
"tart-dual",
"tart-small",
)
# Half of all commits compile on the large macOS 26 SKU, so the two sizes are
# compared on real focused-run traffic rather than one benchmark. The split is
Expand Down
2 changes: 1 addition & 1 deletion tests/test_ci_e2e_compilation_cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ def test_failed_restore_discards_partial_cache_without_removing_products(self):
def test_failure_guard_only_allows_optional_compilation_cache_steps(self):
guard = (ROOT / 'tests/test_ci_self_hosted_guard.sh').read_text()
start = guard.index('check_e2e_runner_fallbacks() {')
end = guard.index('\ncheck_ios_tart_canary()', start)
end = guard.index('\ncheck_xcode_selection()', start)
invoke = guard[start:end] + '\ncheck_e2e_runner_fallbacks\n'
workflow = (ROOT / '.github/workflows/test-e2e.yml').read_text()
candidate = self.root / 'workflow.yml'
Expand Down
Loading
Loading