ci: give a refused owned job one more try on the fleet before Blacksmith

[teamleaderleo]

Why

Leo's rule: a job an owned runner refuses goes back to the fleet. It goes to Blacksmith only when the fleet keeps refusing or is busy. The rescue already re-runs a refused run's failed jobs (#14299), but only ever watched attempt 1.

GitHub sends no workflow_run requested event for a re-run. Run 36059281883 was re-run to attempt 2, and there is no owned-pool-rescue-36059281883-2; all 200 recent rescue runs are for attempt 1. So attempt 2 can't be watched from a new event.

What (rescue side, scripts/ci/owned_pool_rescue.py)

• After re-running the failed jobs of attempt 1, the same watch follows attempt 2 (LAST_OWNED_ATTEMPT = 2).
  • It only acts once a job of attempt 2 asks for an owned label; changes isn't re-run, so there's no marker.
  • It stops when no owned job appears, or once every owned job has run past the 120-second refusal window ("the fleet accepted the retry"). It doesn't wait for the whole run, so it stays within the job's 70-minute timeout.
• A job refused, or queued past the budget, on attempt 2 gets its failed jobs re-run once more, keeping what passed. Attempt 3 and later never take an owned pool, so it can't loop.

Wiring left to the per-job placement PR

Whether attempt 2 actually lands on the fleet depends on each owned-capable job's runs-on reading github.run_attempt == 2 && inputs.pr_refused_retry_runner || first, from a new picker output refused_retry_runner (the owned label) that is separate from retry_runner. The "Place each PR macOS job on a free owned mini" session is restructuring the picker and those runs-on expressions, and it knows which jobs may run owned (GUI jobs may not yet). So the wiring belongs there. Until it lands, attempt 2 takes retry_runner as today and the extra watch ends at its first look.

Verification

tests/test_ci_owned_pool_rescue.py passes locally (32 tests). The fake API now tracks attempts. New cases:

• the fleet refuses the retry again → second re-run, attempt 3 never watched
• the fleet accepts the retry → the watch stops once the job is past the refusal window
• the retry is stuck queued → failed jobs re-run again and nothing that passed is lost

🤖 Generated with Claude Code

---

Summary by cubic

Gives a refused owned job one more try on the fleet before sending it to Blacksmith.

• The rescue now follows the re-run (attempt 2) it triggered, watches for owned-label jobs, and re-runs failed jobs again if the fleet refuses or keeps them queued past the budget; attempt 3 and later always take retry_runner on Blacksmith. One deadline spans both attempts, so a rescue that can't fit its cancel and re-run inside it is skipped rather than risk killing a job.
• Per-job runs-on wiring so attempt 2 can take the owned pool via a new pr_refused_retry_runner picker output is left to the placement change; until that lands, attempt 2 uses retry_runner and the extra watch ends at its first look.

^{Written for commit 25cdf79. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: afd1c7d7-67d0-44cf-8186-d1577e70d764

📥 Commits

Reviewing files that changed from the base of the PR and between 0cc5be3 and 25cdf79.

📒 Files selected for processing (3)

• docs/ci-runners.md
• scripts/ci/owned_pool_rescue.py
• tests/test_ci_owned_pool_rescue.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

Note: pr_refused_retry_runner is described here and in owned_pool_rescue.py, but nothing on main sets it yet, so a refused job still goes straight to Blacksmith on attempt 2. The wiring is in #14325, which is stacked on #14318.