ci: run unsigned iOS jobs on owned minis with counted simulator capacity

[teamleaderleo]

Lets the unsigned iOS CI jobs run on the owned Mac mini pool. They pick it with the same owned-pool picker that pull request and E2E runs use (#14205, #14225), and fall back to Blacksmith. Routing is off until CI_IOS_OWNED=1 is set. That variable is new, and this PR does not set it.

What changes

New scripts/ci/ios_runner_pool.py. It applies the E2E rule (e2e_runner_pool.decide, which calls pr_runner_pool.decide) and accepts only an owned pool. If the picker would pick a Blacksmith pool instead (for example the 12vcpu overflow), iOS stays on its own variable, so MACOS_RUNNER_IOS keeps its current meaning. An auto run goes to the minis only when all of these hold:

• CI_PR_POOL_OWNED=1 and CI_IOS_OWNED=1
• the default resolves to blacksmith-6vcpu-macos-26, as for E2E
• no ios_version input, no App Store upload, no seed_cache (it runs in the ci-cache-writer environment with the R2 write keys), and not called from a release
• the owned pool has 2 machines free
• simulator capacity: the glaeda-ios-sim entry of CI_OWNED_POOL_SLOTS (for example {"glaeda-ios-sim": 2}) leaves one simulator mini free for each requested device family. Without that entry the lane never routes on its own. A swift_package run holds one machine and no simulator, so it skips this check.

Every other case keeps the default. No job is sent to wait in an owned queue.

Labels. Jobs that need the iOS runtime (ios-simulator-build, ios-simulator, screenshots) request [glaeda-std-xcode-<pin>, glaeda-ios-sim]. They never request the pool label alone. mobile-core-package runs host SwiftPM tests and needs no simulator, so it takes the pool label alone. glaeda classifies these jobs itself (isolated or simulator token), so they never take the root label. Both labels come from the picker's JSON output. No glaeda- label appears in any workflow runs-on: line, so the self-hosted guard still holds.

Capacity accounting.

• pr_runner_pool gains CAPABILITY_LABELS and capability_slots(). A glaeda-ios-sim entry in CI_OWNED_POOL_SLOTS is not reported as a slot problem, and slots() still returns only pools.
• queue_janitor counts queued and running jobs that carry the capability label.
• queue_janitor also reads a per-run capability marker (macos-pool-persistent-<run>-<attempt>-<sim_jobs>-glaeda-ios-sim), so a run's simulator jobs count as taken before they exist.
• Every in-flight test-ios or ios-screenshots run created since the snapshot costs 2 simulators.
• The build job carries the label too, so the count errs toward Blacksmith.

test-ios.yml

• New runner job (actions: read). The three macOS jobs read fromJSON(run_attempt > 1 && retry_runs_on || runs_on).
• The fork branch, the tart-ios Tart identity checks and the GitHub-hosted check all still work. They now read the runner job's label.
• runner input gains owned, which forces the owned pool for a proof run without reading the queue. It is an explicit request, so it fails the runner job with an error, never a fallback, unless CI_PR_POOL_OWNED=1 (otherwise the rescue never watches the run and a queued job would wait for good) and CI_OWNED_POOL_SLOTS gives glaeda-ios-sim at least one mini. It also refuses what auto refuses. CI_IOS_OWNED is not required, so the proof run can come first.
• Jobs on a glaeda- label set CMUX_CI_XCODE_APP to CMUX_CI_XCODE_APP_PR, as test-e2e.yml does, so the Xcode matches the pool label; a Blacksmith re-run keeps it.
• runner is added to the ios-tests gate's needs.
• A simulator this run created is removed on reused machines too.

ios-screenshots.yml: screenshots job.

• runner input with owned.
• release.yml calls this workflow with contents: read only (release.yml fails at startup: generate-ios-screenshots requests 'actions: write' the caller does not grant (stable release blocked) #12149). Its runner job therefore cannot hold actions: read to read the queue, so auto keeps MACOS_RUNNER_IOS, and only a direct capture-only dispatch with runner: owned uses a mini.
• The GhosttyKit cache and fastlane snapshot's DerivedData (SNAPSHOT_DERIVED_DATA_PATH) move to $RUNNER_TEMP. The Xcode pin follows the owned label, as in test-ios.
• The ASC key is removed in an always() step.

Rescue. ci-owned-pool-rescue.yml / owned_pool_rescue.py now watch test-ios and ios-screenshots dispatches exactly like E2E. A job left queued or refused (a second simulator job on a mini) gets its failed jobs re-run on retry_runs_on, which is Blacksmith.

Kept on Blacksmith

• ios-streamed-validate (validate). Not routed. It binds ports 3000 and 13000, runs rm -rf /tmp/cmuxpg, uses shared ~/Library DerivedData, puts CLAUDE_CODE_OAUTH_TOKEN into the GUI session with launchctl setenv, and flips the global dark mode. It also writes agent credentials under $HOME and launches the app through launchd. The new secrets cleanup is general hygiene only; it is not a reason to route this job.
• Secrets hygiene. ios-streamed-validate and iroh-release-gate now write $HOME/.secrets/cmuxterm-dev.env under umask 077 and delete it in an if: always() step. The file cannot move to $RUNNER_TEMP because the Mac app's DebugDogfoodCredentialResolver, scripts/lib/dev-secrets.sh and web/scripts/load-dev-env.sh all read that fixed path.
• Signing workflows. ios-testflight, ios-app-store and ios-appstore-upload hold the distribution certificate, the provisioning profiles and the ASC keys. They are untouched.
• Runs that need another runtime or write credentials. Any ios_version run, any App Store upload, any seed_cache run, and the release call of ios-screenshots.

Next: nightly/beta

The TestFlight uploads (plain cmux, INTERNAL, BETA) move to a dedicated release mini whose label takes only signing jobs, with Blacksmith as the fallback. That's a follow-up PR. Decision and host choice: manaflow-ai/cmuxterm-hq#627.

Rollout

1. The glaeda-ios-sim label ships (glaeda#1218) on minis with the iOS simulator role and an iOS 26.x runtime.
2. CI_PR_POOL_OWNED=1 (already set), so the rescue watches the run.
3. Add "glaeda-ios-sim": <sim minis> to CI_OWNED_POOL_SLOTS.
4. Proof run: gh workflow run test-ios.yml --repo manaflow-ai/cmux -f runner=owned. Do this only after step 1 is live.
5. Set CI_IOS_OWNED=1.

Tests

• tests/test_ci_pr_runner_pool.py gains IOSRouting and IOSWiring. They cover capacity counting, package-only runs needing no simulator, the label shape, the switches, the blockers (including seed_cache), forced owned and its refusals without CI_PR_POOL_OWNED or simulator slots, the Xcode pin, the janitor's capability counting and markers, the workflow wiring, signing and streamed-validate staying off the fleet, and a guard that every $HOME/.secrets write has a later always() removal.
• tests/test_ci_owned_pool_rescue.py covers iOS dispatch targets and a job stuck waiting on glaeda-ios-sim.
• test_ci_self_hosted_guard.sh, test_ios_workflow_dispatch_ref.py and test_ci_workflow_run_sources.py are updated.
• Local results: every test above passes; also test_ci_queue_janitor, test_ci_reusable_workflow_permissions, test_run_e2e, test_runner_label_policy, test_ci_actionlint_covers_every_workflow, and actionlint 1.7.7 on all workflows. tests/test_ios_simulator_build_once.py fails the same way on unmodified main locally (the macOS /var symlink), and this PR does not touch it.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Lets the unsigned iOS CI jobs (test-ios.yml, ios-screenshots.yml) run on the owned Mac mini pool, picked by the same pool picker that E2E and pull request runs use, with simulator capacity counted so minis never oversubscribe. Routing is off until the new CI_IOS_OWNED variable is set to 1; nothing changes by default. Activation requires the minis to carry the glaeda-ios-sim label and a matching entry in CI_OWNED_POOL_SLOTS.

Routing

• test-ios.yml and ios-screenshots.yml gain a runner job running the new scripts/ci/ios_runner_pool.py; all macOS jobs read its choice and re-runs fall back to Blacksmith.
• An auto run routes to a mini only when CI_IOS_OWNED and CI_PR_POOL_OWNED are 1, the pool has two machines free, and the glaeda-ios-sim slot entry leaves a simulator mini free per requested device family; otherwise MACOS_RUNNER_IOS keeps its meaning. ios_version, seed_cache, App Store upload, and release calls stay on Blacksmith. A swift_package run takes one mini and needs no simulator, so it skips the capacity check.
• Simulator jobs request the pool label together with glaeda-ios-sim; mobile-core-package takes the pool label alone. runner: owned forces the fleet for a proof run, failing unless CI_PR_POOL_OWNED is 1 and glaeda-ios-sim has minis (the rescue would not watch it otherwise).
• Jobs on a glaeda- label take CMUX_CI_XCODE_APP_PR; the queue janitor and a per-run marker count simulator capacity, and the rescue workflow now watches iOS dispatches and re-runs stuck or refused jobs on Blacksmith.

Secrets hygiene

• ios-streamed-validate and iroh-release-gate write $HOME/.secrets/cmuxterm-dev.env under umask 077 and remove it in an always() step.

^{Written for commit d08b033. Summary will update on new commits.}

Summary by CodeRabbit

• New Features

  • iOS simulator tests and screenshot runs can now be routed to supported Mac runner pools, with automatic selection based on availability and capacity.
  • Manual screenshot runs offer runner choices, including automatic and owned-pool options.
  • Owned-pool rescue now covers iOS simulator and screenshot workflow runs.

• Bug Fixes

  • Improved cleanup of temporary credentials and simulator resources after workflow runs.
  • Runner selection and retry routing account for simulator capacity and ongoing iOS jobs.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 078f20a1-8b94-443d-9fb8-7f5dfd1595de

📥 Commits

Reviewing files that changed from the base of the PR and between 28147df and 45834e9.

📒 Files selected for processing (15)

• .github/workflows/ci-owned-pool-rescue.yml
• .github/workflows/ios-screenshots.yml
• .github/workflows/ios-streamed-validate.yml
• .github/workflows/iroh-release-gate.yml
• .github/workflows/test-ios.yml
• scripts/ci/e2e_runner_pool.py
• scripts/ci/ios_runner_pool.py
• scripts/ci/owned_pool_rescue.py
• scripts/ci/pr_runner_pool.py
• scripts/ci/queue_janitor.py
• tests/test_ci_owned_pool_rescue.py
• tests/test_ci_pr_runner_pool.py
• tests/test_ci_self_hosted_guard.sh
• tests/test_ci_workflow_run_sources.py
• tests/test_ios_workflow_dispatch_ref.py

 ___________________________________________________
< Brace yourself. Winter is coming...for your bugs. >
 ---------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}