ci: stop counting queue time on cancelled jobs as runner minutes

[teamleaderleo]

Summary

The CI health report (run 35948904357) listed iroh-v2 client, cli-pipe-regressions, Claude wrapper regressions and macOS compile admission as the top "cancelled after ≥5 min of macOS work" offenders. Each of those sampled jobs never got a runner. They waited in the macOS queue and were cancelled when a newer push superseded the run.

The Actions API stamps started_at = created_at on a job cancelled while queued, with runner_id: 0, an empty runner name and no steps. The report took completed_at - started_at as runner minutes, so queue wait showed up as burned runner time. Now such a job has zero minutes, and its wait until the cancel counts as queue wait. That also makes the queue percentiles more honest: those jobs used to show 0 minutes of queue.

What the data says (report window 2026-09-23 20:37 to 2026-09-24 02:37 UTC)

Real runner minutes on cancelled jobs, versus queue-only time that the old report counted as minutes:

Job	cancelled jobs	runner min	queue-only min
iroh-v2 / client	7	0	334
ci / cli / cli-pipe-regressions	52	21	1022
ci / Claude wrapper regressions	41	3	728
ci / macOS compile admission	89	321	2648

• The iroh-v2 cancels are concurrency supersession on the same PR (the group is already keyed per PR). Path filters are correct: every run in the window came from a branch that touched workers/iroh-v2/**, CmuxIrxTransport, or the workflow file.
• The compile admission minutes that did run were all pull_request supersessions, 1 to 28 min in. ci: gate macOS admission on fast Linux jobs through needs instead of a fixed sleep #14134 (merged 04:31 UTC, after this window) replaced the fixed 180 s debounce with a needs-gate, so the next snapshots are its before/after.
• cmux-tui-testbox-warmup.yml is not waste: the job is the Testbox session, and its cleanup cancels the run on purpose. Documented in docs/ci/health-report.md.
• test-depot.yml no longer exists on main (ci: route focused tests to one front door, and half of them to 12 vCPU #14067). 4 of its 5 cancelled tests jobs in the window also never got a runner.

Test plan

• python3 -m unittest tests.test_ci_health_report (78 tests), including a new case built from the real run 35941162262 job payload
• Re-scored the window's 197 cancelled ci.yml runs plus the iroh-v2 and testbox runs through the patched job_rows (table above)

🤖 Generated with Claude Code

---

Summary by cubic

Stops counting queue time on cancelled jobs as runner minutes in the CI health report.

Jobs cancelled before a runner picked them up now get zero minutes; GitHub stamps their started_at at creation, so the old report counted the entire queue wait as burned runner time. Their wait still counts as queue wait, and skipped jobs are unaffected.

Bug Fixes

• Detects never-runner jobs via runner_id: 0, an empty runner name, and no steps.
• Re-scoring the sampled macOS cancelled jobs shows most of their time was queue-only, not runner minutes.
• Documents the Testbox warmup exception and the matching queue p90 bump, and adds a test based on a real queued-cancel job.

^{Written for commit 446a0e5. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes
  • Jobs cancelled before a runner starts now show zero runner minutes, with time spent waiting counted as queue wait instead.
• Documentation
  • Clarified that queue-wait figures for cancelled jobs are lower bounds and may raise the reported macOS p90 without indicating a real regression.
  • Explained that cancelled minutes for Testbox warmup runs reflect the VM session, not runner work.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 9441890f-2999-449c-9dbc-93cb583b1677

📥 Commits

Reviewing files that changed from the base of the PR and between 23c0ce2 and 446a0e5.

📒 Files selected for processing (3)

• docs/ci/health-report.md
• scripts/ci/ci_health_report.py
• tests/test_ci_health_report.py

 ___________________________________________________________________________________________________________________________________
< Don't use manual procedures. A shell script or batch file will execute the same instructions, in the same order, time after time. >
 -----------------------------------------------------------------------------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}