ci: land the fleet review fixes #14159 merged without

[teamleaderleo]

Summary

#14159 auto-merged at 43f2c06. The commit that addressed its CodeRabbit review (40ccf9196a) arrived a minute later and never reached main. That PR's review threads say "Fixed in 40ccf91", so this PR makes that true:

• Xcode pin behind an unreadable org endpoint. read_variables() read the organization variables first, and a failure there stopped it before it reached the repository variables that hold the pin. up then checked the mini against the fallback path, not the pin. The organization read is now optional; the repository read is still required.
• Empty pilot cohort. The doctor marked it as a problem but then printed "Next: nothing; the fleet is routing" even though nothing routes. It now says to run scripts/persistent-compile pilot <your PR number>.
• quarantine without a runner. When the Glaeda step failed, the error said "the runner is stopped" even on a machine with no runner configured. It now says "no runner is configured".
• Drain wait. This answers ci: guard the fleet routing switch, Xcode pin and quarantine #14159's "no hacky sleeps" pre-merge error. drain and quarantine polled pgrep every 2 seconds. They now wait on the job process's exit through kqueue NOTE_EXIT, bounded by the drain deadline, so the runner stops the moment the job ends.

One fix is new here. It covers the gap #14159's last comment left open. Running quarantine <reason> on a mini that is already quarantined keeps the old reason, because Glaeda has no quarantined → quarantined transition. It used to do this silently; it now says which reason stays.

Testing

• Commit 1 adds the tests only; they fail on main (3 failures, 5 errors).
• Commit 2: python3 tests/test_ci_persistent_compile_fleet.py passes, 69 tests. That includes the non-admin Xcode pin test that ci: guard the fleet routing switch, Xcode pin and quarantine #14159 added after the review.
• tests/test_ci_change_areas.py passes. Both files are control plane only (ci: route the persistent compile fleet command as control plane only #14157), so no macOS compile runs.
• Not verified on a Mac: the kqueue wait. The tests drive it through a fake select module. Python's select.kqueue API is macOS-only, and no mini has drained yet.

Checklist

• I added or updated tests for behavior changes

🤖 Generated with Claude Code

https://claude.ai/code/session_01RSxoKWiT1D35DDmEYxk59f

---

Generated by Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Lands the CodeRabbit review fixes from #14159 that auto-merged before they could reach main.

Fixes

• Organization variables are now optional; the Xcode pin at the repository level is still required.
• An empty pilot cohort now tells the operator to run scripts/persistent-compile pilot <PR number>.
• drain and quarantine now wait on the worker process exit via kqueue NOTE_EXIT instead of polling pgrep.
• Errors no longer report a stopped runner when none is configured.
• Re-quarantining with a new reason prints that the previous reason is kept instead of silently ignoring the change.

^{Written for commit c92b115. Summary will update on new commits.}

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ed4ab38c-a374-4f49-ab73-e7e2a665dfc0

📥 Commits

Reviewing files that changed from the base of the PR and between adddb59 and c92b115.

📒 Files selected for processing (2)

• scripts/ci/persistent_compile_fleet.py
• tests/test_ci_persistent_compile_fleet.py

 __________________________________________________________________________________________
< Recursion is the root of computation since it trades description for time. - Alan Perlis >
 ------------------------------------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

Rebased onto main after #14164 merged. That PR already landed three of the fixes described above: the optional organization-variables read, the next step for an empty pilot, and the "no runner is configured" error. The conflicts kept main's version of each fix and its tests.

This PR now adds only two things:

• The kqueue NOTE_EXIT drain wait.
• The message saying the old reason stays when a quarantined mini is quarantined again.

69 fleet tests pass, and so does tests/test_ci_change_areas.py. Auto-merge (squash) is on.