[teamleaderleo]

The queue janitor's stall-aware failover (category e: a job GitHub never assigned, runner_id 0, queued 30+ minutes while newer jobs on exactly its labels started) skipped two live cases on 2026-10-31. This fixes the selection logic for both. Each case has a test built from the jobs API's recorded history of its run. The tests fail on main and pass here.

Case 1: run 41158893861 (PR #28749, CI)

GhosttyKit release check (job 124413283263, blacksmith-4vcpu-ubuntu-2404, needs changes) was created at 02:18:12Z. It stayed queued with runner_id 0 until 02:59:09Z. In that time a newer job on its label started (job 124415812116, created 02:28:45Z, started 02:42:40Z).

Cause. The jobs API names no needs, so find_lost() treated every sibling as a possible need:

• _gated() left the run alone while any sibling was unfinished. In this run, web / react-* jobs were created at 02:30Z, 02:43Z and 02:57Z and ran in waves.
• lost_wait_start() restarted the 30 minutes at every sibling's completion. web / runners finished at 02:43:49Z, and react-apps-check (4) finished at 02:57:37Z.

As a result, the job was never 30 minutes overdue with no sibling left unfinished.

Fix. GitHub creates a job's record only after its needs have finished, so a need always predates its dependent. In this run, ci-status, which needs every job, had no record yet. A new function, may_need(job, other), is true only for a sibling created before the job, or one with no created_at. lost_wait_start() and _gated() now count only those siblings.

A sibling created before the job still gates it whether it is running or queued, as before. A later sibling, running or not, does not gate it: the run cannot finish without the lost job, and rerun-failed-jobs after the cancel brings the cancelled siblings back too.

Case 2: run 41150189497 (PR #28544, cmux-next)

cmux-next fast tier (job 124387425580, glaeda-side-std-xcode-26.6) was created at 00:36:11Z and queued with runner_id 0 for 2.5 hours, while newer jobs on that label ran. Every sweep found it lost. The 02:28Z sweep logged: "lost runner assignment: cmux-next fast tier queued 48m with no runner while 1 newer job(s) on glaeda-side-std-xcode-26.6 started; at the head of open PR #28544 (the run already failed, and a re-run of its failed jobs would repeat that red)". It kept the job anyway. The failover was finally done by hand at 03:12Z.

Cause. cmux-next swift test had failed at 01:35Z, and open_head_lost() refuses any run that already failed.

Fix. When the lost job is the run's only unfinished job, the sweep cancels the run and re-runs that job alone: POST /actions/jobs/{id}/rerun, as cmux-bot, never on GITHUB_TOKEN. The red stays as it is. The pieces:

• LostJob.job_id
• LostVerdict.rerun_job / Candidate.rerun_job
• the pending entry's job_id
• GitHub.rerun_job()
• rerun_lost(), which posts the job re-run when the entry names one

A failed run with any other unfinished job, or one whose lost job has no id, keeps OPEN_HEAD_RULE as before.

Unchanged

• Nightly, DEV and promote runs never fail over: _lost_scope(), plus the protected_reason() and is_delivery_run() checks in open_head_lost().
• A bot's re-run is never cancelled again.
• A newer run at the head takes over.
• With no cmux-bot token or no re-run state, nothing at an open head is cancelled.

Functions touched: may_need() (new), lost_wait_start(), _gated(), find_lost(), LostJob, LostVerdict, Candidate, open_head_lost(), classify_lost(), build_plan() (passes rerun_job), new_lost_reruns(), rerun_lost(), GitHub.rerun_job() (new), and the module docstring for category e. Nothing in GitHub.request, jobs, artifact_names, read_run_jobs, the rate-limit exit or the workflow file.

Red/green

tests/test_ci_queue_janitor.py, run as a single file:

• On this branch: Ran 300 tests ... OK.
• The new tests against main's scripts/ci/queue_janitor.py: 3 failures and 5 errors in LostBesideLaterSiblingsTests, LostInAFailedRunTests and LostJobDetectionTests.test_wait_starts_when_the_last_job_it_may_need_finished. For example, the wait start computed at 02:48:30Z was 02:43:49Z instead of 02:18:12Z, and a failed run's lost job was never cancelled.

The other suites that import the janitor pass as single files: test_ci_owned_pool_rescue.py, test_ci_pr_runner_pool.py, test_ci_runner_pools.py, test_ci_starved_labels.py, test_ci_health_report.py, test_ci_owned_warm_state.py, test_ci_change_areas.py and test_reuse_app_host_products.py.

Not fixed here, found on the way

In case 1, no sweep judged the job between 02:48Z and 02:59Z:

• The 02:46Z and 02:55Z cron sweeps were cancelled while pending in the concurrency group.
• The sweep queued at 02:45:58Z waited 16 minutes for a runner on the same blacksmith-4vcpu-ubuntu-2404 label. Its runs-on reads LINUX_RUNNER, which holds 4 vCPU.
• The 02:42Z and 03:02Z sweeps listed 2,420 and 2,376 in-flight runs, past MAX_JOB_LISTINGS (2,000). Each logged "the pool snapshot says truncated, and this sweep decided nothing", so category e did not act either.

These are in the workflow file and in the listing and truncation path, which another change is editing now. They are follow-ups.

🤖 Generated with Claude Code

Agent session: agent session

[teamleaderleo]

Owned by an agent session · session

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Merge receipt for 10f006dc08: every check was green at merge (9 verified; 17 skipped by policy). Full suite runs on main after merge.