Skip to content

Retry bootstrap.sh in the Linux base image to ride out apt.llvm.org flakes - #21

Open
dylan-conway wants to merge 2 commits into
masterfrom
retry-linux-bootstrap
Open

dylan-conway wants to merge 2 commits into
masterfrom
retry-linux-bootstrap

Conversation

@dylan-conway

Copy link
Copy Markdown
Member

Summary

About half of the recent Daily Docker Build failures (Aug 31, Sep 1, 2, 4, 8) were the base stage dying inside bun's scripts/bootstrap.sh because a single HEAD request to apt.llvm.org failed. llvm.sh reports that either as "GPG key not reachable" or as the misleading "Distribution 'debian' in version '13 (trixie)' is not supported" (its wget --method=HEAD https://apt.llvm.org/trixie/ probe returned non-zero in ~0.2s, e.g. run 33876381969). One failed request there takes out that arch's base -> prebuilt -> heavy chain and the multi-platform manifest jobs.

bootstrap.sh is safe to re-run, so this wraps it in a 4-attempt loop with a 60s pause between attempts instead of failing the multi-hour pipeline on one flaky request.

Out of scope: the Sep 6-7 arm64 :heavy failures were an upstream bun build:release binary-verify check (init_have_sme static initializer) and cleared on their own by Sep 9.

Test plan

Exercised the loop under sh in debian:trixie-slim with a stub scripts/bootstrap.sh that fails twice then succeeds (exits 0 after attempt 3) and one that always fails (exits 1 after attempt 4). Did not run a full image build.

…lakes

About half of the recent Daily Docker Build failures were the base stage
dying inside bun's bootstrap.sh because a single HEAD request to
apt.llvm.org failed (reported by llvm.sh as "GPG key not reachable" or
the misleading "Distribution 'debian' in version '13 (trixie)' is not
supported"). One failed request there takes out the whole arch's
base -> prebuilt -> heavy chain and the multi-platform manifests.

bootstrap.sh is safe to re-run, so wrap it in a 4-attempt loop with a
60s pause between attempts.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — straightforward retry wrapper around an existing bootstrap step.

What was reviewed:

  • Traced the for attempt in 1 2 3 4 loop under Debian's /bin/sh (dash): POSIX-only constructs, && break yields exit 0 on success, exit 1 fires only after the 4th failure.
  • Confirmed no set -e is active in this RUN, so an intermediate scripts/bootstrap.sh failure falls through to the retry rather than aborting the shell.
  • git pull still gates the whole step via &&, matching the previous behavior.
Extended reasoning...

Overview

This PR touches a single RUN instruction in Dockerfile (the base stage). It replaces a one-shot sh -c "git pull && scripts/bootstrap.sh" with git pull && followed by a 4-attempt retry loop around scripts/bootstrap.sh, sleeping 60s between attempts and emitting progress/failure messages to stderr. A comment documents why (intermittent apt.llvm.org HEAD-probe failures inside llvm.sh) and notes bootstrap.sh is idempotent. Nothing else in the file changed.

Security risks

None. The change does not introduce new network sources, credentials, or executable content — it re-runs an already-trusted script from the cloned bun repo. No injection surface (no user-controlled interpolation), no auth or crypto paths touched.

Level of scrutiny

Low. This is a CI-resilience tweak in a build image, not runtime code. The main correctness questions are shell semantics: the loop uses only POSIX constructs (for … in, [ … -eq … ], break, exit) so it runs correctly under dash, Debian's default /bin/sh. On success break leaves the loop with status 0; on the final failure the explicit exit 1 fails the layer. This RUN does not use set -e, so a non-zero from scripts/bootstrap.sh correctly falls through to the retry logic rather than aborting. git pull remains the leading && guard, preserving prior behavior if the pull itself fails.

Other factors

No CODEOWNERS/ANTOWNERS files exist in the repo, no prior reviews or objections are on the PR timeline, and the bug hunt exited on dry_streak with no findings. The author's test plan (stubbed bootstrap in debian:trixie-slim) matches the shell environment Docker will use. Change is small, self-contained, and easy to revert.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — straightforward retry wrapper around an idempotent bootstrap step.

What was reviewed:

  • Traced the for loop under /bin/sh -c (Docker's default RUN shell): && break on success, explicit exit 1 on the 4th failure, so the layer's exit status is correct in both paths.
  • Confirmed $attempt is shell-expanded (not a Docker ARG/ENV), and git pull still gates the whole step via && as before.
  • The follow-up commit dropped the stderr redirects on the echo lines; loop semantics are unchanged.
Extended reasoning...

Overview

This PR touches a single RUN instruction in /home/claude/bun-development-docker-image/Dockerfile. It replaces a one-shot sh -c "git pull && scripts/bootstrap.sh" with git pull && followed by a POSIX for loop that retries scripts/bootstrap.sh up to 4 times with 60-second sleeps, plus an explanatory comment about apt.llvm.org flakiness. A second commit (dc735ed) removed >&2 redirects from the two echo diagnostics after the initial review; no other behavior changed.

Security risks

None. The change is purely control-flow around an existing script invocation in a build-image Dockerfile. No new sources, credentials, URLs, or packages are introduced; the same scripts/bootstrap.sh runs, just possibly more than once. There is no injection surface — $attempt is a loop variable over literal integers.

Level of scrutiny

Low. This is a small, self-contained CI reliability tweak in infrastructure tooling. The shell is well-formed for Debian's /bin/sh (dash): scripts/bootstrap.sh && break short-circuits on success (loop exits with status 0), and the [ "$attempt" -eq 4 ] guard issues an explicit exit 1 so the Docker layer fails when all attempts are exhausted — the RUN cannot silently succeed after four failures. Removing the redundant outer sh -c wrapper is a no-op since shell-form RUN already runs under /bin/sh -c.

Other factors

No CODEOWNERS or ANTOWNERS files exist in the repo. The bug hunt exited on dry_streak with no findings. The only prior review on the timeline was a COMMENTED state (not CHANGES_REQUESTED) and was followed by a responsive commit, so there are no outstanding third-party objections. The PR author validated the loop with a stub script per the description; the logic is simple enough to verify by inspection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant