ci: run darwin tests on Buildkite-hosted agents while the mac fleet is offline - #37354
Conversation
…s offline The bare-metal macOS fleet behind the `test-darwin` queue is offline being reimaged. Until it returns: - darwin test steps target a `test-darwin-hosted` queue (Buildkite-hosted M4 agents) instead of the fleet's tag-selected agents - only the aarch64 `latest` (macOS 26) lane runs there: hosted agents are Apple Silicon only and the queue is pinned to one image, so the aarch64 `previous` and x64 lanes are dropped for now - darwin tests run on main, or on a PR when the commit subject contains `[macos tests]`; other PR pushes skip them to bound hosted-agent spend. Darwin build lanes are unaffected (they cross-compile on Linux) Revert this commit when the fleet is back. No-Verification-Needed: CI pipeline config only; checked with ci.mjs --dry-run
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughDarwin CI scheduling now uses the temporary ChangesHosted Darwin test scheduling
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.buildkite/ci.mjs:
- Around line 435-437: Verify that the CI cluster contains the
test-darwin-hosted queue and that it is populated with Apple Silicon macOS 26
agents before merging the darwinHostedQueue selector. If the queue is absent or
contains non-Apple-Silicon macOS agents, update the queue configuration or
selector so the Darwin step schedules correctly and satisfies the
EXPECTED_PLATFORM_* checks.
- Around line 401-408: Update the temporary cleanup note near getTestAgent() to
explicitly instruct restoring the previous platform-tagged test-darwin selector
there, alongside deleting darwinHostedQueue, darwinTestsEnabled, and the
getPipeline() filter.
- Around line 414-416: Update darwinTestsEnabled to extract only the first line
of getCommitMessage() before applying the macos/darwin opt-in regular
expression, while preserving the isMainBranch behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 65b5f32f-5770-4cda-882a-129b6993c7dd
📒 Files selected for processing (1)
.buildkite/ci.mjs
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — temporary CI-only reroute of darwin test lanes to a hosted queue, well-commented with a clear revert path.
What was reviewed:
- Filter logic in
getPipeline(): both new predicates pass non-darwin platforms through unchanged; on main onlydarwin-aarch64-26survives, matching the hosted queue's arm64/macOS-26 constraint. getTestAgentchange is safe for the other darwin caller (getTraceOrderStepviatraceOrderTargets) — that lane is also aarch64/latest and soft-fails.EXPECTED_PLATFORM_RELEASEfor the surviving lane is "26", which matches the hosted image; canary release gating on main still sees the remaining darwin test key.
Extended reasoning...
Overview
This PR touches only .buildkite/ci.mjs, the pipeline generator. It adds a hosted-queue constant, two small predicates (darwinTestsEnabled, isRunnableDarwinTestPlatform), swaps the darwin branch of getTestAgent to return { queue: 'test-darwin-hosted' }, and chains two .filter() calls onto relevantTestPlatforms so darwin test lanes are (a) dropped on PRs unless the commit subject opts in, and (b) narrowed to the single aarch64/latest lane the hosted agents can run.
Security risks
None. This is CI orchestration YAML generation — no product code, no auth, no user input handling. The new regex runs against the commit subject (already used the same way for [build images] etc.).
Level of scrutiny
Low. It is a self-described temporary infra stopgap, confined to one CI script, with detailed comments explaining the constraint and explicit deletion instructions for the revert. Worst-case failure mode is a mis-scheduled Buildkite step, which is immediately visible and does not affect the shipped product. The author verified with --dry-run across the three relevant scenarios.
Other factors
I traced the knock-on effects: the traceOrderTargets darwin entry also flows through getTestAgent and will land on the hosted queue — that's fine since it targets aarch64/latest and is soft_fail: true. The release step's testStepKeys gate on main now includes one darwin key instead of three, which is the intended reduction. getTestBunStep sets EXPECTED_PLATFORM_RELEASE: '26' for the surviving lane, matching the hosted image. Merge-queue builds fall under the PR path (no darwin tests unless opted in), consistent with the cost-bounding intent. Nothing here touches build lanes, so darwin still proves it compiles on every PR.
There was a problem hiding this comment.
Beyond the inline nit, I also checked: the darwin trace-order step (traceOrderTargets) — its host is aarch64/latest, so it lands on the hosted queue correctly on main; EXPECTED_PLATFORM_RELEASE: "26" on the surviving lane matches the hosted queue's pinned macOS 26 image; and getCommitMessage() matching the full body — on Buildkite it reads BUILDKITE_MESSAGE (subject-only), same as every other [tag] opt-in in this file, so that's consistent.
Extended reasoning...
This is a small, clearly-temporary CI-only change with one minor nit filed inline. I checked a few adjacent concerns the diff touches indirectly (trace-order routing, EXPECTED_PLATFORM_* env, commit-message matching semantics) and none are affected. Leaving the approval decision to a human because the change reduces what gates canary releases on main and depends on an out-of-band Buildkite queue being provisioned — both things a maintainer should acknowledge rather than a bot.
…ert note - darwinTestsEnabled() also returns true for isBuildManual(), so picking a darwin lane in the New Build options form is not silently dropped for lack of a [macos tests] tag in the message - the temporary-block comment now says to revert the commit rather than delete pieces by hand, since getTestAgent()'s old selector must come back with it No-Verification-Needed: CI pipeline config only; checked with ci.mjs --dry-run
There was a problem hiding this comment.
LGTM — the manual-build opt-in and revert-note concerns from the earlier round are addressed in a16b111.
Checked that the surviving darwin aarch64 26 / latest lane is also the one traceOrderTargets uses, so the darwin trace-order step on main still resolves to a runnable agent. The [macos tests] regex over getCommitMessage() matches how every other commit-message option in this file already works (BUILDKITE_MESSAGE is subject-only in CI), so not splitting on newline is consistent. Canary release on main now gates on one darwin lane instead of three, which is the point — the other two have no agents to run on.
Extended reasoning...
Overview
Temporary CI pipeline change in .buildkite/ci.mjs only: routes darwin test steps to a Buildkite-hosted test-darwin-hosted queue while the bare-metal mac fleet is offline, restricts darwin testing to the single aarch64/macOS-26 lane the hosted queue can serve, and gates darwin tests on main / manual UI builds / a [macos tests] commit-subject opt-in to bound per-minute hosted spend. No runtime, bindings, or test code touched.
Security risks
None. This is pipeline-generation JS that emits Buildkite YAML. No new secrets, no external input parsing beyond the existing getCommitMessage() regex pattern already used for every other [tag] option in the file.
Level of scrutiny
Low-medium. It's infrastructure config, explicitly temporary, with a one-commit revert plan. The two things that would normally warrant a human — the external test-darwin-hosted queue prerequisite and the change to what gates canary — are both called out in the PR description, and the fallback if the queue is missing (step waits for an agent) is stated as no worse than the current state (fleet offline, steps already wait forever).
Other factors
- My earlier review flagged that a manual UI build selecting darwin in the block-step form would be silently dropped; the author added
isBuildManual()todarwinTestsEnabled()and documented it in the block comment. Addressed. - CodeRabbit's rollback-note concern is addressed: the comment now says "revert the commit that added this block" and names both
getTestAgent()and thegetPipeline()filter. - CodeRabbit's "read only the subject" suggestion was correctly not taken —
getCommitMessage()readsBUILDKITE_MESSAGE(subject-only) in CI, and every otherparseOptionregex ingetPipelineOptions()matches against the same value without splitting. - Traced
traceOrderTargets: the darwin entry targets{aarch64, release: 26, tier: latest}, which is exactly the laneisRunnableDarwinTestPlatformkeeps, so the darwin trace-order step on main still gets a valid{queue: test-darwin-hosted}agent. - No CODEOWNERS entry for
.buildkite/. - Author verified the three cases via
--dry-run.
…7364) ### What does this PR do? #37344 commented out the three darwin `testPlatforms` entries and the darwin `trace-order` target while the mac fleet was offline. #37354 (merged after it, but branched before it) routes darwin tests to Buildkite-hosted agents instead; with the entries gone there is nothing left to route, so **main currently runs no darwin tests at all** (see build 91730: 0 darwin test steps). This puts the entries back. #37354's filter still drops the two lanes hosted agents can't run (aarch64 `previous`, x64) and gates the rest to `main` / `[macos tests]`, so the effective result is one aarch64 macOS 26 lane on `test-darwin-hosted`, which build 91720 already showed passing in ~13 min. Reverting #37354 when the fleet is back brings all three lanes back with no further edit here. ### How did you verify your code works? `node .buildkite/ci.mjs --dry-run` on top of current main: - `main`: `darwin-aarch64-26-test-bun` + `darwin-aarch64-trace-order`, both `queue: test-darwin-hosted` - PR, plain subject: no darwin test steps - PR with `[macos tests]`: the single darwin test step
### What does this PR do? Darwin test steps were pinned to `parallelism: 2` because the bare-metal fleet only had a couple of boxes per lane. On the Buildkite-hosted queue from #37354 every shard gets its own M4 agent, so widen darwin to 10 shards. Expected effect per main build: the ~26 min darwin suite (2 × 13 min on hosted today, build 91720) becomes ~26/10 + ~2 min per-shard setup ≈ 4-5 min wall clock, for roughly +30% billed minutes (each shard repeats checkout / artifact download / `bun install`). Not going wider than 10 because beyond that the fixed setup dominates. Revisit together with the #37354 revert once the fleet is back (the fleet can still absorb 10 queued shards per lane, it just serializes them). ### How did you verify your code works? `node --check`; the value only feeds the step's `parallelism` field, and `runner.node.mjs` already shards by `BUILDKITE_PARALLEL_JOB`/`_COUNT` (linux runs 20-wide the same way).
…7633) The macOS test agents are back on `test-darwin`, so this undoes the hosted-queue detour from #37354 / #37369: - darwin aarch64 tests (`latest` and `previous` lanes) run on every build again, not just main / `[macos tests]` - tag-based agent selector (`queue=test-darwin, os, arch, release-tier`) restored, hosted queue and its helpers removed - sharding back to 2 per lane - darwin symbol-order trace target restored The x64 test lane stays commented out until the Intel agents are back; that is the only difference from the pre-outage file. `--dry-run` for a PR build: ``` darwin-aarch64-26-test-bun {queue: test-darwin, os: darwin, arch: aarch64, release-tier: latest} parallelism=2 darwin-aarch64-14-test-bun {queue: test-darwin, os: darwin, arch: aarch64, release-tier: previous} parallelism=2 ```
2 participants
What does this PR do?
Stopgap while the bare-metal macOS test fleet (
test-darwinqueue) is offline being reimaged.test-darwin-hostedqueue backed by Buildkite-hosted M4 agents, instead of selecting fleet agents byos/arch/release-tiertags.latest(macOS 26) lane runs there. Hosted agents are Apple Silicon only and the queue is pinned to a single image, so the aarch64previousand the x64 lanes are dropped until the fleet returns.main, or on a PR when the commit subject contains[macos tests]. Other PR pushes skip them, to bound hosted-agent spend. Darwin build lanes are unaffected (they cross-compile on Linux), so PRs still prove darwin compiles.Needs the
test-darwin-hostedqueue to exist in theCIcluster before merge (Buildkite hosted,MACOS_ARM64_M4_6X28, macOS 26 image). Until it does, darwin test jobs on main will wait for an agent, which is no worse than today.Revert this PR when the fleet is back.
How did you verify your code works?
node .buildkite/ci.mjs --dry-runwith Buildkite env for three cases:main: onedarwin-aarch64-26-test-bunstep,agents: { queue: test-darwin-hosted }[macos tests]in the subject: same single darwin test step as main