Skip to content

ci: park each pull request's build on its root and route its re-push back to it - #15020

Merged
teamleaderleo merged 4 commits into
mainfrom
ci/park-pr-builds
Sep 27, 2026
Merged

teamleaderleo merged 4 commits into
mainfrom
ci/park-pr-builds

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Compile admission p50 was 334 s over 290 owned admissions (2026-09-26 13Z to 09-27 14Z, every mini's admissions.jsonl). 165 of them (57%) rebuilt the app, at p50 396 s against 123 s for those that did not. Split by cause:

Rebuild cause Count p50
The PR's own package interface change 90 408 s
The PR's own app files (more than 5) 16 391 s
The kept build's distance (another PR, older main) 58 360 s

For a PR that changes a package interface, every start except its own previous build rebuilds the app. 69 re-pushes rebuilt for their own changes: 53 on another mini than their previous push, and 16 on the same mini after another PR's admission replaced the kept build. Starts from the same PR's build skipped the rebuild 11 times in 19 (7 of 12 with a package interface change).

The distance picker (#14949) cannot find those builds, since a root publishes only its current stamp.

Change

  • Park. When keep replaces another PR's build, it now moves that build to pr-builds/pr-<n> beside the root's store. It's a rename, not a copy. A root keeps at most 2 parked builds, for 6 h, and parks only with 140 GiB free. The slots also sweep leftovers from a killed park.
  • Unpark. check with CMUX_OWNED_PR (compile admission passes the PR number) swaps that PR's parked build back in and parks the current one. It never drops a main build (idle warming's) or a build it could not park: then the job starts from the kept build. keep clears the job's own stale slot.
  • Publish. Each root's roots entry in warm-keys carries its parked stamps (owned_warm_state.py checks the fields). They stay out of keys, so they never crowd out other roots' keys.
  • Route. distance_route() costs a root by its parked build of this PR whenever check would swap it in. hook_root_cost() ranks a same-PR start far, not rebuild, when the only package change is the PR's own. Main's diff from a parked base still makes it rebuild.

glaeda's hook reads the parked stamp when it ranks roots on the mini (teamleaderleo/glaeda#1284).

Ported from #14804 without its age pricing, jobs-API lookup, contention multiplier or SwiftPM scratch, and rebuilt on main's distance routing.

Expected effect

If every one of the 87 re-push rebuilds had started from its own build at the observed same-PR hit rate (40-58%), compile p50 would have been 220-246 s instead of 334 s, and p90 452-462 s instead of 478 s. That's an upper bound: a busy mini's parked build is only used when the picker's wait limit allows it. ci-dash /estimates.json by_start and samepr starts will show the change.

Tests

tests/test_ci_owned_build_state.py (park, unpark, fingerprint mismatch, disk floor, count and age caps), tests/test_ci_owned_warm_state.py (parked fields), tests/test_ci_warm_distance.py (a parked build draws a package-changing re-push to its mini; hook parity, also run against glaeda#1284's hook). Targeted -k runs pass locally; CI runs the files.

🤖 Generated with Claude Code


Summary by cubic

Prevents a pull request's re-push from rebuilding the app when its previous build is still on a root. When keep replaces another PR's build, it now parks that build in pr-builds/pr-<n> (a rename, not a copy) instead of dropping it; admission's check swaps the PR's parked build back in, never dropping a main build, and also replaces a kept build no router can read (stale or missing stamp). Distance routing sends the re-push to the mini holding it. A same-PR package interface change ranks far instead of rebuild, since every other start rebuilds.

  • Parking is capped at 2 builds per root for 6 h, and only with 140 GiB free; a park killed after its stamp is written is finished rather than dropped, and expired slots are skipped, not published.
  • warm-keys publishes parked stamps as parked on each root's roots entry, which glaeda's hook reads; parked builds don't count against the keys list.
  • Re-push compile admission p50 is expected to drop from 334 s to 220-246 s.

Written for commit 752934d. Summary will update on new commits.

Review in cubic

…back to it

A root's kept DerivedData is replaced by the next admission on it, so a
re-push usually found its previous build gone. `keep` now parks another pull
request's build in pr-builds/pr-<n> (a rename; 2 per root, 6 h, only with
140 GiB free), `check` swaps it back in for that pull request, and
warm-keys publishes it as `parked` so distance routing sends the re-push to
the mini that holds it. A same-PR start with the PR's own package interface
change ranks far rather than rebuild, since every other start rebuilds.

Ported from #14804 without its age pricing, jobs-API lookup, contention
multiplier or SwiftPM scratch, onto main's distance routing (#14949).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: d421cf93-5872-4bf1-9ff1-983728f3c0d4

📥 Commits

Reviewing files that changed from the base of the PR and between 82c26b3 and 752934d.

📒 Files selected for processing (8)
  • .github/workflows/ci-macos.yml
  • docs/ci-runners.md
  • scripts/ci/owned_build_state.py
  • scripts/ci/owned_warm_state.py
  • scripts/ci/warm_distance.py
  • tests/test_ci_owned_build_state.py
  • tests/test_ci_owned_warm_state.py
  • tests/test_ci_warm_distance.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

CI failure attribution

CI passes on 752934de36 (run 36337659699 attempt 1).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Toolbox g1 🔔 reviewed the failed run. The implementation checks passed, but the CLI product job was refused on cmux7s-Mac-mini because glaeda-cmux-runner-hook reported capacity: the gui token is taken; ci-status failed as a consequence. This is fleet capacity contention rather than a code regression. I’m leaving the branch unchanged until a clean rerun has an available GUI token.

teamleaderleo and others added 3 commits September 27, 2026 13:31
…ed keys out of warm-keys

From review: unpark only over another pull request's build (or none), and
routing costs the parked build exactly when check would swap it in; keep
clears the job's own stale slot; a park killed after its stamp is renamed
into place; expired slots are not published; parked pr- keys no longer
crowd MAX_WARM_KEYS (roots[].parked carries them).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the stamp on a failed park

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 27, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@teamleaderleo
teamleaderleo merged commit b78e8aa into main Sep 27, 2026
72 of 73 checks passed
@teamleaderleo
teamleaderleo deleted the ci/park-pr-builds branch September 27, 2026 17:45
@github-actions

Copy link
Copy Markdown
Contributor

Merge receipt for 752934de36: every check was green at merge (19 verified; 22 skipped by policy). Full suite runs on main after merge.

teamleaderleo added a commit that referenced this pull request Sep 27, 2026
…isk floor (#15039)

* ci: evict parked pull request builds oldest first instead of a free-disk floor

#15020 parked a pull request's build only with 140 GiB free. There is no
floor now: parking is bounded by 2 per root and 6 h, and when space runs
out parked builds go first. `keep` that hits ENOSPC evicts every parked
build on the mini, oldest first, and retries the clone once;
`owned_build_state.py evict-parked STORE [COUNT]` does the same for disk
tooling. Main builds are never parked, so eviction never touches one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: clean a partial clone on failure, strict evict-parked args (review)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
rustybret pushed a commit to rustybret/bmux that referenced this pull request Sep 27, 2026
c490fb4 Add an opt-in subtle sidebar selection highlight (manaflow-ai#14890)
b78dae5 ci: evict parked pull request builds oldest first instead of a free-disk floor (manaflow-ai#15039)
31b6fc9 test: close the TabManagers the session snapshot tests create (manaflow-ai#15038)
e64eb93 ci: send every retry of an owned-mini job to Blacksmith, once (manaflow-ai#15035)
17f0f90 Test: pin persisted Kimi custom kind on exact remote restore (manaflow-ai#15034)
f9d5bb5 Switch between cmux and cmux NIGHTLY from inside the app (manaflow-ai#14995)
756847a sidebar: keep the selected row clear of the footer (manaflow-ai#15013)
1431a1e One dialog per close, and Feed banner clicks open the agent (manaflow-ai#14960)
74d4719 ci: route cli-product-tests to the gui runners like the shards (manaflow-ai#15011)
b78e8aa ci: park each pull request's build on its root and route its re-push back to it (manaflow-ai#15020)
60e45e7 web tests: run every child process asynchronously (manaflow-ai#15001)
55724f0 ci: side lanes take the light minis' side runners first (manaflow-ai#15019)
3be8bdd Restoring a remote workspace terminal hands it this machine's working directory (manaflow-ai#8634)
a9e555c Reattach 0.64.25 tmux-profile SSH workspaces after upgrading (manaflow-ai#14938)
fe20e20 fix: mark workspace-level notifications read on workspace focus (manaflow-ai#12427)

# Conflicts:
#	.github/workflows/app-host-test-rerun.yml
#	.github/workflows/auth-refresh-tests.yml
#	.github/workflows/ci-macos.yml
#	.github/workflows/ci.yml
#	.github/workflows/cloud-command-deadlines.yml
#	.github/workflows/cloud-machine-tests.yml
#	.github/workflows/cloud-task-local-tests.yml
#	.github/workflows/cmux-tui.yml
#	.github/workflows/iroh-v2.yml
#	.github/workflows/relay-tls.yml
#	.github/workflows/reload-build.yml
#	.github/workflows/remote-daemon.yml
#	.github/workflows/terminal-hang-diagnostics.yml
teamleaderleo added a commit that referenced this pull request Sep 28, 2026
Main landed this pull request's parking (#15020, #15039, #15056), distance
routing (#14949) and a refit (#15117) in its own form, so the conflicted
files take main's side; what still adds value comes back in the next
commits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant