Skip to content

docs: route CI runner policy from fleet entry point #15638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
teamleaderleo merged 1 commit into from
Sep 29, 2026
Merged

docs: route CI runner policy from fleet entry point #15638

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ Read the matching skill before changing an area, then only the references needed
including `verify-local.py --help`, `--list` and `--repo`. Push does not run checks for you.
- Use [CONTRIBUTING.md](CONTRIBUTING.md#getting-started) for setup. Outside cmuxterm-hq-created
checkouts, set `CMUX_DEV_BACKEND_MODE=local` for dev builds. Follow [tagged builds](skills/cmux-dev-workflow/references/tagged-builds.md)
for commands and cache reuse; team fleet rules live in cmuxterm-hq.
for commands and cache reuse; team fleet tasks start at
[Fleet and CI: start here](https://github.com/manaflow-ai/cmuxterm-hq/blob/main/build-fleet/FLEET-AND-CI.md).
Never use bare `xcodebuild` or an untagged `cmux DEV.app`. Clean up only your own tags.
- A same-repo app PR gets a fleet dogfood build and link comment only while it has the
`dev-build` label. Add it when someone will dogfood the PR, not by default; under load
Expand Down
21 changes: 21 additions & 0 deletions docs/ci-runners.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
# CI runners

> **Fleet route:** start at the hq [Fleet and CI: start here](https://github.com/manaflow-ai/cmuxterm-hq/blob/main/build-fleet/FLEET-AND-CI.md), then return here for CI runner selection, Xcode pins, repository variables, and the Blacksmith overflow switch. This file is the owner for those CI procedures.

## Rules that must never be broken

- CI is minis first. Blacksmith is overflow; do not disable an owned lane to
make room for a dev build.
- Every mini must have every pinned Xcode from
`scripts/ci/xcode-pins.txt` and the matching `CMUX_CI_XCODE_APP_*` variable
before it receives that pool's job.
- A dead Blacksmith pool uses the one probe-and-switch path below. Queued runs
move only by force-cancel plus rerun; GitHub cannot rerun one job until its
whole run finishes.

Every CI/CD job picks its runner from a repository variable instead of a
hardcoded label. Changing a runner type is a single repository-variable update
that takes effect on the next workflow run.
Expand Down Expand Up @@ -857,6 +870,14 @@ admission or is draining, pressured, or unavailable.

## Blacksmith outage: the overflow switch

This is the only Blacksmith-outage lever. The 2026-09-29 manual variable
changes are recorded in the hq
[lever log](https://github.com/manaflow-ai/cmuxterm-hq/blob/main/build-fleet/observations/2026-09-29-ci-lever-log.txt),
but the probe-and-switch workflow below owns current recovery. Do not lower
`CI_PR_POOL_OWNED`, `CI_E2E_OWNED_UI`, `CI_IOS_OWNED`, or another lane switch.
GitHub cannot rerun one job while its parent run is still active; a run stuck
on a dead pool needs force-cancel and then rerun.

Blacksmith is overflow, so when it stops starting jobs the fix is to stop
sending overflow there, not to reroute a lane. `ci-cloud-overflow-probe.yml`
does that automatically (`scripts/ci/cloud_overflow_switch.py`). Every 10
Expand Down
Loading