ci: never abandon a run the owned pool rescue cancelled

[teamleaderleo]

Summary

The owned-pool rescue cancels a PR run and re-runs it when an owned Mac is busy or refuses a job. Today it gives up 180 s after the cancel if the run hasn't finished cancelling. The PR's run is then left cancelled with nobody to re-run it.

Rescue run 36074563873 did exactly that to CI 36074561333 (PR #14314):

• look 5: claude-wrapper and cli refused by glaeda-std-xcode-26.6 ("a fleet build holds the host lock")
• cancelled run, then force-cancelled run
• gave up: run 36074561333 did not finish 180s after cancel; not re-run

The in-flight macos / macOS compile admission on a Mac settled about 5 minutes after the cancel. The PR had to be closed and reopened to get CI.

Change

• CANCEL_WAIT_SECONDS goes from 180 to 20 minutes. The force-cancel is re-sent every 5 minutes while waiting, not just once. If GitHub refuses a force-cancel (the run settled between the read and the POST), the rescue logs it and keeps waiting instead of aborting.
• A rescue may now run up to 25 minutes (RESCUE_GRACE_SECONDS) past the end of the watch: 60 minutes for a PR run, 150 for an E2E run (ci: let E2E runs take an owned Mac with a free slot #14311). Before, a refusal found in the watch's last few minutes was left alone ("too little of the watch left"). It is now re-run too. No cancel starts without 21 minutes of that grace left.
• ci-owned-pool-rescue.yml timeout-minutes goes from 160 to 180 (the E2E watch, plus the grace, plus 5 minutes for checkout). A test pins it to JOB_TIMEOUT_SECONDS.

What's unchanged: the watch itself (60 minutes, one deadline across attempts), the pushed-head check before the re-run, and the "someone else re-ran it" check.

Not changed here

For a refused cheap job, the rescue still cancels the whole run, including a compile already running on another Mac. That throws away minutes of compile to retry a 6-second job. Letting the run finish and then re-running failed jobs would keep the compile, but the refused job would retry later. That tradeoff belongs with the pool-routing work in #14310 and #14318, which also edit this file.

Testing

• tests/test_ci_owned_pool_rescue.py, 42 tests:
  • new: a cancel that settles after 330 s is waited out and re-run
  • a cancel that never settles force-cancels repeatedly and gives up only at the 20-minute limit
  • a late refusal is now rescued, still within the job timeout
  • a refused (409) force-cancel keeps waiting and still re-runs
  • no cancel starts when fewer than 21 minutes of the job remain
• actionlint and validate_test_execution_registry.py are clean.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • CI rescue runs now allow more time for cancellation to settle before rerunning a job, with repeated force-cancellation attempts if needed.
  • Rescue runs can proceed when a job is refused near the end of its watch period, provided enough time remains for cancellation and rerun.
  • Watched attempts retain their original time limit, while the overall job timeout includes additional time for rescue handling.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 9805fdd6-4e44-43dc-81d3-4f85ea56c3bb

📥 Commits

Reviewing files that changed from the base of the PR and between fc7f80d and 2218f0e.

📒 Files selected for processing (3)

• .github/workflows/ci-owned-pool-rescue.yml
• scripts/ci/owned_pool_rescue.py
• tests/test_ci_owned_pool_rescue.py

 __________________________________________________
< I ran the tests. They filed a restraining order. >
 --------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}