ci: give PR builds one darwin aarch64 test lane that any mac agent can take - #39195
Conversation
…n take PRs lost darwin tests entirely in #39191 because the version-specific lanes queue for hours. Capacity is not actually short, it is fragmented: the 26 lane's agents sit idle on PR-heavy days while the 14 lane backs up. So PR builds now get a single untiered aarch64 lane (no release-tier selector) that every arm64 mac agent serves; main, manual and [macos tests] builds keep the 26 / 14 / x64 lanes for version coverage. No-Verification-Needed: pipeline generator only; checked with --dry-run
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
Comment |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — small, self-contained follow-up to #39191 that re-enables one untiered darwin aarch64 test lane on PR builds.
What was reviewed:
- Traced the new
{os: "darwin", arch: "aarch64", release: "any"}platform throughgetTestBunStep:depends_on/--stepresolve todarwin-aarch64-build-bun(matches the cross-compiled build lane),getTestAgentnow omitsrelease-tierwhentieris unset, andEXPECTED_PLATFORM_RELEASEis not emitted (its condition requirestier === "latest"). - Confirmed
.concatonly adds the untiered lane when!darwinTestsEnabled, so main/manual/[macos tests]keep the three tiered lanes unchanged and the release step'stestStepKeysnever picks up the PR-only lane. - Checked
getImageKeyon the new platform — darwin is filtered fromimagePlatforms, so no spurious build-image dependency; step/group keys (darwin-aarch64-any) don't collide with anything on PR builds.
Extended reasoning...
Overview
This PR modifies .buildkite/ci.mjs only. It is a direct follow-up to #39191, which took all darwin test lanes off PR builds because version-tiered agent targeting was fragmenting the mac pool. This change re-introduces exactly one darwin aarch64 test lane on PR builds, but with no release-tier agent selector so any arm64 mac agent can pick it up. Main branch, manual builds, and [macos tests] opt-in commits keep the existing three version-specific lanes unchanged.
The concrete edits: (1) getTestAgent now spreads release-tier only when tier is truthy (previously unconditionally for aarch64), (2) a new prDarwinTestPlatforms list with one untiered entry, (3) relevantTestPlatforms concats that list when darwinTestsEnabled is false, and (4) the comment above testPlatforms is updated to describe the new arrangement.
Security risks
None. This is CI pipeline-generation code that produces Buildkite YAML; no user-controlled input, no auth/crypto/permissions surface. The commit-message regex parsing already existed and is unchanged.
Level of scrutiny
Low-to-medium. It's a config-shape change to pipeline generation, verified by the author via --dry-run (output pasted in the PR description matches what tracing the code produces). The blast radius is limited to which test lanes appear on PR builds; main-branch behavior is provably unchanged because darwinTestsEnabled is true whenever isMainBranch(), so the concat contributes nothing and the tiered-lane filter keeps all three darwin entries.
I traced every consumer of the new platform object: getTargetKey → darwin-aarch64 correctly matches the existing build step; getPlatformKey → darwin-aarch64-any is unique; getTestAgent emits {queue, os, arch} with no tier; EXPECTED_PLATFORM_RELEASE is gated on tier === "latest" so it's omitted and the runner's platform assert won't reject a mismatched macOS version; getImageKey produces a darwin key that imagePlatforms never contains (darwin is filtered out), so no build-image dep is added. The getTestAgent guard change is backward-compatible — every existing aarch64 darwin entry in testPlatforms has tier set.
Other factors
The author also authored #39191 that this reverses/refines. No outstanding reviewer comments (only bot noise in the timeline). The change is small (~15 lines), mechanical, and the dry-run output demonstrates the intended pipeline shape.
### Problem The darwin lanes on PR builds skip every test file whose expected duration is 10s or more (`--skip-slower-than=10000`, from #39215). That is about 1% of the files but about 60% of a shard's time. A PR can break one of those tests and still go green. `main` still runs them. The skip was a stopgap while the mac pool was small. ### Fix - Remove the skip from the darwin PR lanes and the beta lane. - Remove the `--skip-slower-than` runner flag. It had no other caller. - Raise the beta lane's timeout from 45 to 60 minutes. That lane runs the whole suite as one shard on one box. `test/expected-durations.json` stays. Shard packing and the parallel allowlist read it. ### Background The mac pool is back to full size. On 2026-09-02, over about 19 hours: - Darwin test jobs waited a median of 6s for an agent (p90 11-17s). - arm64: about 3 of 15 slots busy on average. - x64: about 3 of 9 slots busy on average. A PR darwin shard should go from about 9 minutes to about 14, which is what the `main` shards take now. PR builds keep the one "any macOS" lane per arch (#39195). `main` still runs the three version-specific lanes.
2 participants
#39191 took darwin tests off PR builds because the version-specific lanes were queueing for hours. The arm64 pool is not actually short of capacity for PR volume; it is fragmented: the agents behind the
26lane sit mostly idle (that lane is main-only) while the14lane backs up.This gives PR builds one untiered
darwin aarch64test lane whose agent selector is justqueue=test-darwin, os=darwin, arch=aarch64, so every arm64 mac agent serves it regardless of macOS version.main, manual builds and[macos tests]commits keep the three version-specific lanes.getTestAgentonly emitsrelease-tierwhen a tier is set, and the untiered lane sets noEXPECTED_PLATFORM_RELEASE, so the runner's platform assert checks os/arch only.--dry-run: