Skip to content

Install the toolchain with the script bun generates for its CI machines - #24

Open
robobun wants to merge 2 commits into
oven-sh:masterfrom
robobun:robobun/1e86f878/generated-toolchain
Open

robobun wants to merge 2 commits into
oven-sh:masterfrom
robobun:robobun/1e86f878/generated-toolchain

Conversation

@robobun

@robobun robobun commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Fix

  • install-toolchain.sh generates the script that oven-sh/bun bakes its Debian 13 CI machines with, and runs it up to the prefetch section.
  • A second commit makes curl and wget use IPv4 and a new lookup for each retry.
  • Verified in a fork, workflow unchanged: all 9 jobs green on bun main (run 36206871186) and with bun at 29d9638da3 (run 36203662359). Notes lists every run.

Background

  • scripts/build/ci-images/spec.ts in oven-sh/bun renders one bake script per CI image (bun run ci:images). Its documentation says to cut before prefetch in a container.
  • The script exports PATH in /etc/profile.d/bun-ci.sh. Docker shells do not read it, so the llvm-current step stays.
  • Considered: install the toolchain in the Dockerfile with versions from spec.ts (duplicates the package list and install steps), and a section filter (no use: su and sudo work after ulimits).

Downsides

  • bun on PATH is the version that CI pins (1.4.2), not releases/latest. qemu-user is gone. tailscale and buildkite-agent are new. Compressed base: 4092 MB on amd64 (was 4823), 3728 MB on arm64 (was 4295).
  • The build depends on three names in oven-sh/bun (images, generateImage, the prefetch banner). A rename stops it with a message.
  • A BUN_REF older than ci: content-addressed machine images bun#43608 cannot build.
Notes

This PR covers Linux only. The Windows image has the same break and an older one. It needs a maintainer's decision: #25.

Runs in the fork (workflow_dispatch of daily-build.yml, images go to the fork's own registry). The runs used commits af03e04 and b5a4fd6. The two commits of this PR differ from them in one comment of install-toolchain.sh.

Run Content Result
36201653087 first commit, bun at 29d9638da3 all 9 jobs green
36203662359 both commits, bun at 29d9638da3 all 9 jobs green
36203659404 both commits, bun main 8 jobs green. arm64 heavy failed, see below
36206871186 both commits, bun main, no other run at the same time all 9 jobs green
  • Two runs build bun at 29d9638da3 through one more commit that is not in this PR. bun main did not compile on Linux from 754ea34261 to 4dbe5e04aa (2026-09-25 23:13 to 2026-09-26 00:12 UTC).
  • The arm64 heavy failure of run 36203659404 is an error of the test setup. Three runs ran at the same time and pushed the same tag. The job pulled prebuilt-arm64@sha256:4ac9d009..., the image of run 36203662359, whose bun checkout is a detached HEAD. git pull --ff-only then said You are not currently on a branch. The image of its own run is sha256:8469c5d1.... Run 36206871186 repeated the job alone, and it passed.
  • The heavy jobs cover su postgres -c initdb and the debug, fuzzilli and release builds. Their logs show bun install v1.4.2.

What the generated script does in docker build (measured on amd64 and arm64):

  • Every section before prefetch exits 0. Its own packages section installs systemd, so systemctl enable and systemctl mask work.
  • The script cannot run twice: the second run exits 9 at groupadd --system buildkite-agent. The retry loop of Retry bootstrap.sh in the Linux base image to ride out apt.llvm.org flakes #21 cannot wrap it.
  • role: "test" leaves out the cross-compile SDKs and sysroots that only CI's build machine needs. The spec's arm64 Debian image is that build machine, so the script here is generated from it with the role changed.
  • The script writes the node-gyp header cache for the buildkite-agent user. install-toolchain.sh copies it to /root/.cache, where bootstrap.sh put it.

apt.llvm.org (requests from build steps on GitHub runners):

  • 3 of 10 runs of the script failed in the LLVM install before the second commit: curl: (7) Failed to connect to apt.llvm.org port 443, or llvm.sh said Distribution 'debian' in version '13 (trixie)' is not supported by this script.
  • The cause in each checked case: Trying [2a04:4e42:...]:443... Immediate connect fail ... Network is unreachable. The retries of curl and wget reuse that lookup.
  • With the tools' defaults 40 of 1200 requests failed. With the two configuration files 0 of 1200 failed, and 5 passed after a retry (run 36201763637). deb.debian.org and github.com had 0 failures of 600 each.
  • base passed 6 of 6 times on the first attempt with the second commit.

Image size. base on arm64 was 3725 to 3728 MB in three runs and 4749 MB in run 36206871186. In that run git pull rewrote the pack of /workspace/bun/.git (1037 MB) inside the toolchain layer. The git pull is the one of the old line. base on amd64 was 4092 or 4093 MB in all four runs.

  • /etc/hosts is read-only in a build step, so a pinned address is not possible from the Dockerfile.
  • Not covered: a lookup that fails for longer than the tools retry.

More differences from the last published image: curl-h3 and age are in /usr/local/bin (were in /usr/bin). /opt/rust holds the pinned nightly (2.4 GB, was 4.8 GB with stable). The Dockerfile's own Rust step still installs a second copy in /root/.rustup, as before. A buildkite-agent user exists. ENV BUN_NO_CORE_DUMP=1 has no reader left in oven-sh/bun.

For oven-sh/bun, not part of this PR: spec.ts has no way to render a script for a container. A container image in the spec (the macOS machines have a toolchain-only list), or a filter for tools, removes the cut by banner here. That is a decision for the maintainers of the spec.

Review. I started a second review of this diff. It did not finish, so it has no result. The verification is the runs above.

Helper branches. The fork keeps the branches that the runs above built (robobun/1e86f878/probe, verify-pinned, verify-pinned-2). They hold the probe scripts and the commit that pins bun. They can go when this PR merges.

…I machines

oven-sh/bun#43608 removed scripts/bootstrap.sh. The base stage ran it, so
every daily build fails with 'scripts/bootstrap.sh: not found'.

install-toolchain.sh generates the bake script for Debian 13 with
scripts/build/ci-images/spec.ts, leaves out the cross-compile tools of
CI's build machine, cuts the script before its prefetch section and runs
it. Root gets a copy of the node-gyp header cache, as bootstrap.sh gave
it.
…ails

Some lookups of apt.llvm.org from a GitHub runner return one IPv6 address
and nothing else. A build container has no IPv6 route, so the curl that
fetches llvm.sh exits 7, or the wget check inside llvm.sh fails and llvm.sh
reports that the distribution is not supported. The retries of both tools
reuse the failed lookup.

Configuration files for curl and wget, applied to the generated script
through CURL_HOME and WGETRC, make them ask for IPv4 only and look the name
up again for each retry.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@robobun

robobun commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Status

How to reproduce the failure on master:

gh run view 36139096809 -R oven-sh/bun-development-docker-image --log-failed | grep 'bootstrap.sh'
# sh: 1: scripts/bootstrap.sh: not found   (build-amd64-base and build-arm64-base)

curl -s -o /dev/null -w '%{http_code}\n' https://raw.githubusercontent.com/oven-sh/bun/main/scripts/bootstrap.sh
# 404

State of this PR: open, ready for review. The unchanged daily-build.yml is green in all 9 jobs on bun main, in the fork: https://github.com/robobun/bun-development-docker-image/actions/runs/36206871186

@Jarred-Sumner

Copy link
Copy Markdown
Contributor

@robobun wake up!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants