ci: re-run by cause: host faults to Blacksmith, code failures back to the minis

[teamleaderleo]

Leo picked "split by cause". This builds on #15035 and changes only who goes where.

Re-run after	Who re-runs it	Goes to
host fault on a mini (refusal, lost runner, timeout, restore failure)	github-actions[bot]: owned-pool rescue, failure attribution (classify_failures.py, all-machine verdict)	retry_runner on Blacksmith, as #15035 does
code or test failure only	a person or agent ("Re-run failed jobs", "Re-run all jobs")	back to the owned label attempt 1 placed the job on. A full re-run gets the normal pick, without queueing.
Linux/guard failure, cancel	anyone	unchanged

Why the actor carries the cause. A failed-jobs re-run does not re-run changes, so runs-on can only see attempt 1's outputs plus github.run_attempt and github.triggering_actor. The classifiers already exist: the rescue re-runs only refusals and stuck queues, and the attribution re-runs only when every failed job is a machine failure. So a bot re-run is a host-fault re-run by construction.

Measured over 7 days (ci.yml, same-repo, runs with 2 attempts):

Who re-ran	Host fault	Code failure only	Other
bot (138 re-runs)	135	0	3
person or agent (217 re-runs)	23	139	55

Bound. If a mini fails a person's re-run, the attribution now re-runs that attempt as the bot, onto Blacksmith. Before, it re-ran attempt 1 only. It never re-runs its own re-run, so there is no loop.

Changes:

• runs-on (ci-macos.yml, ci.yml claude-wrapper, remote-daemon PR lane): github.run_attempt > 1 becomes github.run_attempt > 1 && github.triggering_actor == 'github-actions[bot]'. swift-package also admits a person's retry.
• Picker: only a bot retry drops the owned pools (host_fault_retry()). Any other retry picks every pool, with queue_rounds=0.
• Picker and janitor may_hold_owned_pool: they count a person's re-run as possibly owned.
• classify_failures.rerun_decision: also re-runs a person-started attempt.

Tests: I ran the targeted files locally (picker, janitor, rescue, classify, self-hosted guard .sh, release-sdk .sh, seed routing subset, change-areas routing subset). The full suite runs on CI.

Supersedes #15032.

🤖 Generated with Claude Code

---

Summary by cubic

Routes CI re-runs by cause: github-actions[bot] re-runs a pull request run only after a host fault on a mini (refusal, lost runner, timeout, restore failure) and sends it to Blacksmith's retry_runner, while anyone else's re-run follows a code or test failure and goes back to the owned pool attempt 1 placed the job on. This changes the old behavior, which sent every re-run to Blacksmith regardless of who started it.

Changes

• runs-on gates the Blacksmith retry lane on github.run_attempt > 1 && github.triggering_actor == 'github-actions[bot]'; the picker's host_fault_retry() drops the owned pools only for bot retries, and a person's full re-run picks like attempt 1 without queueing.
• classify_failures.py now re-runs a machine failure on a person-started attempt to Blacksmith; it never re-runs the bot's own re-run, so the flow cannot loop.
• The rescue sweeper lists in-progress CI runs a person re-ran and watches them on any attempt, and the picker and janitor read the newest marker up to a run's attempt; main's dispatch keeps every retry on Blacksmith.

Measured over 7 days the classifier matches the intent: 135 of 138 bot re-runs followed a host fault, and 139 of 217 other re-runs a code failure only.

^{Written for commit 808eae7. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 207b6f9d-af8e-4384-9661-a71b03c4cb80

📥 Commits

Reviewing files that changed from the base of the PR and between 0bc5145 and 808eae7.

📒 Files selected for processing (16)

• .github/workflows/ci-macos.yml
• .github/workflows/ci.yml
• .github/workflows/remote-daemon.yml
• docs/ci-runners.md
• scripts/ci/classify_failures.py
• scripts/ci/owned_pool_rescue.py
• scripts/ci/pr_runner_pool.py
• scripts/ci/queue_janitor.py
• tests/test_ci_change_areas.py
• tests/test_ci_classify_failures.py
• tests/test_ci_owned_pool_rescue.py
• tests/test_ci_pr_runner_pool.py
• tests/test_ci_queue_janitor.py
• tests/test_ci_release_sdk_lane.sh
• tests/test_ci_self_hosted_guard.sh
• tests/test_seed_derived_data.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[github-actions]

Merge receipt for 808eae7e16: every check was green at merge (24 verified; 19 skipped by policy). Full suite runs on main after merge.

[teamleaderleo]

Toolbox g1 🔔 reviewed the cause-based retry routing. The bot-only host-fault path is bounded, person-triggered reruns preserve attempt-1 ownership, and the classifier/rescue/janitor changes avoid retry loops. Full required CI, macOS, remote-daemon, guard, and classifier checks are green. Enabling ordinary squash auto-merge subject to human approval.