Skip to content

darwin-ci: leave the checkout's .git out of the guest sync - #40693

Open
robobun wants to merge 2 commits into
mainfrom
farm/47d6f10a/darwin-ci-sync-skip-git
Open

robobun wants to merge 2 commits into
mainfrom
farm/47d6f10a/darwin-ci-sync-skip-git

Conversation

@robobun

@robobun robobun commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • A darwin-aarch64 test job on a Tart host fails with step failed outside runner before any test runs. The sync step prints ShellError: Failed with exit code 23 and rsync(45694): error: .../.git/gc.pid: open (2) ...: No such file or directory (build 107284, agent darwin-arm64-baguette-tart-26-1).
  • The command hook rsyncs the whole checkout into the guest, .git included (scripts/darwin-ci/lib/guest.ts:53). The agent's git fetch leaves a detached git gc --auto running. It removes .git/gc.pid and rewrites packs while rsync reads them. A vanished file is exit 23, fatal to the hook.

Fix

  • syncTo passes --exclude=/.git. The pattern is anchored to the checkout root, so a nested repository still copies whole.
  • Correct because nothing in the guest reads the checkout's .git. The runner takes commit, branch, and repository from the forwarded BUILDKITE_* env (scripts/utils.mjs). Vendor tests clone their own repositories, and tests that need one create it in a temp dir. Nothing else writes to the checkout during the sync.
  • main.ts install-self exposes the existing installSelf(), and the README gets an "Updating the scripts on a host" section. See Rollout.
  • Verified: test/internal/darwin-ci-guest-sync.test.ts runs the real syncTo through a fake ssh that runs the rsync server side locally. It fails on main and passes here. Also test/internal/runner-junit.test.ts.

Rollout

  • The hosts run the hooks from the copy installed in /usr/local/share/darwin-ci at provision time. This PR's own darwin lanes ran that old copy, so a green tart lane here is not evidence the new sync ran. The unit test is.
  • After the merge, the five tart hosts (challah, baguette, ciabatta, focaccia, sourdough) need the refresh from the README: rsync the directory over, then main.ts install-self. No agent restart. cc @alii

Background

  • A test-darwin Tart job boots a fresh macOS guest, rsyncs the checkout to ~/work, and runs guest/job.sh over ssh.
  • git gc --auto runs after a fetch when a repository has too many packs or loose objects. It detaches by default, so it outlives the checkout step.
  • rsync exit 23 is "partial transfer due to error". The macOS rsync reports a vanished source file this way. Upstream rsync uses exit 24.
Notes
  • The guest also receives about 1 GB less per job, the size of a full bun .git.
  • The Tart agent config (lib/agent.ts) sets no git-clone-flags and no git-mirrors-path, so the checkout is a full clone that is fetched on every job and wiped nightly. Each fetch adds a pack or loose objects. Once the count crosses gc.autoPackLimit (50) or gc.auto (6700), the next fetch triggers the background repack. On a multi-GB repository that takes minutes, so the race window covers the whole sync.
  • Excluding only gc.pid is not enough. The repack also writes objects/pack/tmp_pack_*, then renames them and deletes the old packs, so other files vanish too.
  • A retry on exit 23 or 24 was the other option. A repack that is still running can fail the retry the same way, and each attempt copies the whole .git again. Leaving .git out removes the only concurrent writer to the checkout.
  • --delete is kept. An excluded path is not deleted on the receiver without --delete-excluded, and the baked image ships an empty ~/work anyway.
  • Local check of the failure class: rsync 3.4.1 copying a tree whose .git/gc.pid is removed mid-copy exits 24 ("some files vanished") without the exclude and 0 with it.
  • Possible follow-ups, each its own PR: let hooks/command run the checkout's scripts/darwin-ci/hooks/command.ts when present, so a hook fix ships with its commit and a PR's own darwin lane exercises it. Give the Tart agent a git-mirrors-path like scripts/agent.mjs does.
  • darwin-ci: address review of the tart host scripts #37771 rewrites the same syncTo call to an argv array. Whichever lands second needs a one-line rebase.
  • The test skips when rsync is not on PATH (Windows). The fake ssh handles the -l user form that rsync uses for user@host.
  • Also ran test/internal/build-download-retry.test.ts.

no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/internal/darwin-ci-guest-sync.test.ts

The tart command hook rsyncs the whole agent checkout into the guest,
.git included. The agent's fetch leaves a detached `git gc --auto`
running, which rewrites .git while rsync reads it. A file that vanishes
mid-transfer (.git/gc.pid) fails the sync with exit 23, and the job
dies before any test runs.

Nothing in the guest reads the checkout's .git: the runner takes the
commit and branch from the forwarded BUILDKITE_* env, and vendor tests
clone their own repositories. Exclude it, anchored to the checkout root
so a nested repository still copies whole. The guest also receives
about 1 GB less per job.
@coderabbitai

coderabbitai Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

On-demand reviews are free for the next 23 days. After that, they cost $0.25 per reviewed file.

Or wait 20 minutes for your next included review.

View limit details

Limit details: You’ve used all 5 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6c0bb684-848f-433e-8e26-40f48faebe0c

📥 Commits

Reviewing files that changed from the base of the PR and between a92d84e and ce08238.

📒 Files selected for processing (4)
  • scripts/darwin-ci/README.md
  • scripts/darwin-ci/lib/guest.ts
  • scripts/darwin-ci/main.ts
  • test/internal/darwin-ci-guest-sync.test.ts

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — a single anchored rsync exclude in CI tooling, with a test that runs the real command.

What was reviewed:

  • rsync semantics: --exclude=/.git is anchored to the transfer root, so nested vendor/*/.git still copies and .gitignore is unaffected; --delete without --delete-excluded won't touch a pre-existing .git on the receiver (and the baked image's ~/work is empty).
  • Bun shell array interpolation of checkoutExcludes expands as one argv entry, not a quoted string.
  • The test's fake ssh shim correctly consumes -i/-o/-l pairs and the host before exec'ing the rsync server side in the temp guest home; 192.0.2.1 is TEST-NET-1 and never reaches the network. PATH is restored in finally.
Extended reasoning...

Overview

This PR touches two files: scripts/darwin-ci/lib/guest.ts adds an exported checkoutExcludes = ["--exclude=/.git"] and interpolates it into the syncTo() rsync invocation, and test/internal/darwin-ci-guest-sync.test.ts is a new test that drives the real guest().syncTo() through a fake ssh on PATH so the rsync server side runs against a local temp directory. The change is pure CI infrastructure for the macOS Tart hosts — it does not touch any code that ships in the Bun binary.

Security risks

None. This is internal CI tooling that assembles an rsync command. No user-controlled input flows into the new exclude flag (it's a literal), and the test is hermetic: the fake ssh shim intercepts before any network is contacted, the target IP is in the RFC 5737 documentation range, and everything is confined to a tempDir. The PATH mutation is process-local to the test runner and restored in a finally.

Level of scrutiny

Low. The functional change is a one-flag addition to an rsync command line whose semantics are well-known: a leading / in an rsync filter anchors it to the transfer root, so only the checkout-root .git is skipped while nested repositories and .gitignore still transfer. The Bun $ shell expands an array argument as separate argv entries, so ${checkoutExcludes} becomes exactly --exclude=/.git. The PR description's claim that nothing in the guest reads the checkout's .git is plausible and, more importantly, the blast radius of being wrong is a red CI job on a Tart host — not user-facing behavior.

Other factors

The test follows repo conventions (tempDir from harness, skipIf gating on Windows/missing rsync, resource restoration in finally, module-scope imports) and lives alongside sibling test/internal/ infrastructure tests like runner-junit.test.ts and build-download-retry.test.ts, so the new-file placement is consistent. No CODEOWNERS entry covers scripts/darwin-ci/ or test/internal/. The PR timeline shows no outstanding reviewer objections. The bug-hunt exit reason was dry_streak with zero findings.

@robobun

robobun commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review.

Reproduced the failure class locally with rsync 3.4.1: a copy of a tree whose .git/gc.pid is removed mid-transfer exits 24 ("some files vanished") without the exclude and 0 with it. The macOS rsync on the Tart hosts reports the same case as exit 23, which is what build 107284 showed.

Second commit (ce08238): main.ts install-self and a README section on refreshing the installed scripts. The hosts run the hooks from /usr/local/share/darwin-ci, so the fix takes effect on a host once that copy is refreshed (see Rollout in the description).

CI on ce08238 (build 107624) and on 235da7c (build 107304): every lane is green except test/js/web/url/url.test.ts on darwin x64 (TypeError: Invalid URL for the Unicode 16 IDNA case). That test fails on main too and is unrelated to this change. The other reported tests passed on retry. Nothing more to do here on my side.

…host

The agent runs the hooks from the copy installed in
/usr/local/share/darwin-ci at provision time. A change under
scripts/darwin-ci reaches a host only once that copy is refreshed.
Expose installSelf() as `main.ts install-self` and write the steps down
in the README.
@robobun

robobun commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 3:35 AM PT - Aug 28th, 2026

❌ @robobun, your commit ce08238 has 1 failures in Build #107624 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 40693

That installs a local version of the PR into your bun-40693 executable, so you can run:

bun-40693 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — internal CI tooling change, small and well-contained.

Checked that --exclude=/.git is anchored (nested vendor/*/.git still copies) and that Bun.$ array interpolation expands checkoutExcludes as separate argv entries in the rsync command. The new test drives the real syncTo through a fake ssh shim, uses tempDir, restores PATH in finally, and skipIfs on Windows/missing rsync. For the follow-up commit: installSelf was already imported and used by provision(), so the new install-self case just exposes it as a standalone subcommand; config.installDir renders in the usage string. README addition is ops docs only.

Extended reasoning...

Overview

This PR touches only macOS Tart CI orchestration tooling under scripts/darwin-ci/ plus a test in test/internal/. The core fix is a single rsync flag: checkoutExcludes = ["--exclude=/.git"] spliced into guest().syncTo() so the agent checkout's .git is not copied into the guest VM, avoiding rsync exit-23 when a background git gc --auto rewrites .git mid-transfer. The second commit adds an install-self subcommand to main.ts (wiring the already-imported installSelf() from lib/host.ts) and a README section documenting how script changes reach a host. A new test in test/internal/darwin-ci-guest-sync.test.ts runs the real syncTo through a fake ssh on PATH that executes the rsync server side against a local tempDir.

Security risks

None. This is host-side CI orchestration that runs on trusted BuildKite macOS hosts, not shipped runtime code. No user input parsing, no auth/crypto/permissions changes. The fake ssh shim in the test is scoped to a tempDir prepended to PATH and restored in finally. The install-self subcommand rsyncs the script directory into /usr/local/share/darwin-ci under sudo — the same call provision already makes.

Level of scrutiny

Low. The functional change is one rsync argument with a well-documented failure mode (vanished .git/gc.pid → exit 23) and a clear proof that nothing in the guest reads the checkout's .git (runner uses forwarded BUILDKITE_* env for commit metadata). The anchored /-prefixed exclude pattern is the standard rsync idiom for "root-level only", and the test explicitly asserts a nested vendor/elysia/.git still transfers. The install-self addition is a three-line dispatch to an existing helper.

Other factors

The test follows harness conventions: tempDir with using, test.skipIf(isWindows || !rsync), module-scope imports, PATH mutation guarded by try/finally, content assertions before implicit exit-code check (rsync failure would throw from Bun.$). No CODEOWNERS entries cover scripts/darwin-ci/ or test/internal/. The bug hunt exited on dry_streak with no findings and no ruled-out candidates. The PR description notes a one-line rebase interaction with #37771 — informational only, not a blocker for this review.

@robobun

robobun commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator Author

Another occurrence on 2026-09-18: build 117649, job darwin any aarch64 - test-bun, agent darwin-arm64-focaccia-tart-15-1. The sync step failed with ShellError: Failed with exit code 23 on .git/gc.log.lock and .git/gc.pid, right after the agent printed Auto packing the repository in background. No test ran in that job.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant