ci: self-calibrating warm-distance compile estimates

[teamleaderleo]

Recommendation 3 of the owned-fleet warmth audit: make the compile estimates correct themselves.

Why

scripts/ci/warm-distance-model.json's tier p50s (near 140 / far 267 / rebuild 401 s) were fitted by hand once, and nothing fed the actual compile seconds back. Every admission records predicted_seconds against compile_seconds in each mini's admissions.jsonl. Over the 243 scored owned compiles in those logs (2026-09-26 03:03 to 09-27 09:08 UTC), the model's median absolute error is 58 s, with a bias of −28 s, and only 22% of near compiles land within 25%. The tiers miss the start kind. A near compile from a kept build ran ~91 s and one from a seed ran ~157 s, but both were predicted at 140 s.

What

• fit() adds tiers_by_start. It stores p50/p90/mean/n and the app-rebuilt count per tier × start kind (kept, seed, ...), with counts. predict(feature, model, start) uses the cell once it has ≥5 compiles, and otherwise falls back to the tier. A p50 that is missing, zero, negative, NaN or a bool is ignored. Models without tiers_by_start predict exactly as before, so the schema is additive and version stays 1. admission now records the prediction for its own start kind. Medians throughout, so one hung 3,000 s compile moves nothing (tested).
• warm_distance.py refit. It refits only the tiers and cells from the last 14 days of admissions under the committed near threshold and hot files. Hot files, start_classes and job_seconds are left as committed, because they need git history or glaeda's job log, and the picker work owns start_classes. Before fitting it dedupes admissions and drops implausible compiles (<5 s or >2 h). A tier with fewer than 5 rows keeps its committed entry. drift() flags any tier or cell p50 with n ≥ 20 that moved more than 20% from what the committed model predicts for it. backtest is the time-ordered replay below.
• scripts/ci/warm_model_refit.py is the daily job, meant for mini-6. It reads admissions (ci-dash's estimates.jsonl, whose compact lines read_rows now accepts, or --hosts over SSH), refits, and stops unless something drifted. On drift it writes warm-distance-model.json, a git apply-able refit.patch, summary.md (the drift, the cells, errors before and after, and the replay) and status.json. With --token-file it force-resets ci/warm-model-refit, and only that branch (the name is hard-coded and asserted not to be main), to the checkout's HEAD. It commits the model there through the contents API and opens or updates one PR against main. It never writes main. Without a token, the patch and summary in the output directory are the result.
• Collector bug fixed. warm_distance.py collect read nothing from any mini without a per-root cmux-ci-*/admissions.jsonl. The minis' login shell is zsh, and an unmatched glob fails the whole cat. It returned 32 of the 251 lines on the fleet. It uses find now.
• The committed model is refit with exactly what the job would propose (refit on the 243 admissions). Tiers: near 140→105, far 267→206, rebuild 401→373. Cells: near kept 91 / seed 157, far kept 121 / seed 247, rebuild kept 355 / seed 442. Only tiers, tiers_by_start, misclassified, rows, fitted_at and the new calibrated change. start_classes and job_seconds are untouched.
• glaeda's hook reads only tiers[*].p50, so it keeps working unchanged. Hook: cost a root's kept build with the model's kept-start cell teamleaderleo/glaeda#1304 makes it cost a root's kept build with the (tier, kept) cell under the same ≥5 rule, with a parity test.

Backtest: committed model vs self-calibrated, time-ordered replay

Data: every mini's admissions.jsonl, read over SSH (read-only) and deduped, 243 successful compiles. "Model" is the committed tier p50s, which predicted all of these. The self-calibrated prediction for each compile is refit() from only the admissions before it: its cell once that has 5 earlier compiles, else the (refit) tier. Within 25% means |actual − predicted| ≤ 25% of predicted.

Refit before every admission (continuous):

compiles	n	model MAE s	model bias s	model within 25%	self-calibrated MAE s	bias s	within 25%
all	243	57.8	-28.1	53%	43.0	-7.1	60%
tier near	98	54.5	-31.2	22%	43.2	-9.5	38%
tier far	22	69.0	-56.7	41%	60.2	-40.5	50%
tier rebuild	123	58.6	-27.3	80%	40.9	-1.0	80%
start kept	130	57.8	-52.2	55%	33.5	-7.1	65%
start seed	113	57.8	22.5	51%	64.2	-8.3	55%
near from kept	44	55.2	-49.9	14%	29.4	-5.0	41%
near from seed	54	53.4	17.9	30%	71.1	-23.8	35%
far from kept	10	116.4	-116.4	20%	84.2	-60.2	40%
far from seed	12	43.4	-22.7	58%	40.5	-12.7	58%
rebuild from kept	76	57.0	-48.7	83%	31.5	-4.7	83%
rebuild from seed	47	62.0	41.0	74%	52.0	6.7	77%

Refit once a day (the job's cadence), scored on the 65 compiles of 09-27, each predicted by the refit from 09-26's admissions (out of sample):

compiles	n	model MAE s	model bias s	model within 25%	refit daily MAE s	bias s	within 25%
all	65	55.6	-28.0	52%	38.4	12.4	63%
tier near	27	46.7	-22.4	22%	36.7	16.5	44%
tier far	2	118.4	-118.4	0%	64.7	-64.7	0%
tier rebuild	36	60.0	-29.2	78%	40.8	14.5	81%
start kept	37	54.0	-40.4	62%	31.9	16.5	70%
start seed	28	59.1	17.6	39%	61.6	-14.0	54%
near from kept	13	53.3	-36.2	23%	25.1	16.5	46%
near from seed	14	45.4	21.9	21%	48.4	2.5	43%
far from kept	1	145.6	-145.6	0%	58.3	-58.3	0%
far from seed	1	91.1	-91.1	0%	71.1	-71.1	0%
rebuild from kept	23	56.5	-40.4	87%	31.9	16.6	87%
rebuild from seed	13	68.1	32.7	62%	71.5	-8.3	69%

• Kept starts gain the most. Near-from-kept MAE falls 55→29 s and within-25% rises 14%→41%. Rebuild-from-kept falls 57→32 s.
• Near-from-seed does not improve. That cell is wide (50–450 s), and its first rows (09-26 morning) ran high, so early cells over-predicted. Recency windows and shrinkage toward the tier did no better (tried windows 10–50 and blends k = 5/10: near-from-seed MAE stayed 58–69 s). Its full-window p50 (157 s) sits within 20% of the old 140 s, so the drift gate would not have proposed it alone.
• Far is thin. Far has only 22 compiles, 10 of them from kept builds, so the far cells are the least settled. The kept cell still predicts from 5.
• Limited history. The admission logs start on 09-26, so the daily replay has one refit boundary. The job's summary will carry the same table with more days as they accrue.

Where the refit runs, and why

• GitHub-hosted runners: cannot reach the minis or the tailnet.
• An owned trusted runner (CI_SEED_TRUSTED_POOL, cmux15): holds only its own mini's log (cmux15 has none), and owned runners run PR code.
• ci-dash (mini-6): already collects every mini's admission lines (estimates.jsonl, 30 days, the fields the refit needs) and runs no GitHub runners (hq#595). A token there is out of reach of PR code, so the refit runs there as _cidash from a checkout of main.
• Opening PRs: GITHUB_TOKEN cannot open PRs here (the repo's "Allow GitHub Actions to create and approve pull requests" is off). The job therefore takes an explicit token, and without one it leaves the patch and summary as its artifact.
• Untrusted inputs: admission lines are written by jobs, pull requests' included. The refit uses medians and plausible times only, and a person merges every change.

Setup on mini-6

manaflow-ai/cmuxterm-hq#785 installs the refit with ci-dash's own install.sh, and hq-autodeploy runs it. It covers the _cidash sparse https checkout of this repo's main and the daily LaunchDaemon ai.manaflow.ci-dash-warm-refit, and it only syncs until this PR is on main. After both PRs merge:

1. Probe. Run fleet push-probe so ci-dash ingests admission lines. It needs no sudo; /estimates.json currently reports n = 0.
2. Optional token. For automatic PRs, create a fine-grained token limited to manaflow-ai/cmux with Contents: write and Pull requests: write, and install it as state/warm-model-refit.token (_cidash, 600; the command is in the ci-dash README). Without it, the job only writes refit.patch and summary.md. A PAT-opened PR triggers CI, where a GITHUB_TOKEN one would not.

Tests

tests/test_ci_warm_distance.py (29 tests) covers:

• fit cells and counts, and robustness to an outlier
• predict with a full cell, a sparse cell, bad cells and old models
• the admission recording the kept cell
• refit's window, dedupe, implausible rows and kept-as-committed parts
• the drift threshold and minimum n
• backtest time order, with no future rows
• ci-dash line normalization and the collect command
• the refit job: no-drift, drift patch/summary (git apply --check against the tree), and PR calls that touch only ci/warm-model-refit and refuse main

test_seed_decide, test_ci_pr_runner_pool, test_ci_owned_build_state and test_ci_admission_placement pass.

The edits in warm_distance.py stay in predict/fit/the model schema, plus new refit/backtest functions and collect. route_admission, the candidate scoring and start_classes are untouched.

🤖 Generated with Claude Code

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[coderabbitai]

Warning

Review limit reached

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 859655e3-aab1-4d25-8e10-fb88407befd5

📥 Commits

Reviewing files that changed from the base of the PR and between 6431ac2 and b484034.

📒 Files selected for processing (5)

• docs/ci-runners.md
• scripts/ci/warm-distance-model.json
• scripts/ci/warm_distance.py
• scripts/ci/warm_model_refit.py
• tests/test_ci_warm_distance.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[github-actions]

Merge receipt for b4840343b9: every check was green at merge (13 verified; 17 skipped by policy). Full suite runs on main after merge.