install: let libarchive gunzip buffered tarballs instead of inflating into memory

[robobun]

Repro

W=$(mktemp -d); export BUN_INSTALL_CACHE_DIR=$W/cache
# build a ~21 MB .tgz whose decompressed tar is 2050 MiB of zeros
bun -e '
  import {createWriteStream} from "node:fs"; import {createGzip} from "node:zlib";
  function oct(n,w){return n.toString(8).padStart(w-1,"0")+"\0"}
  function hdr(name,size){const b=Buffer.alloc(512,0);b.write(name,0,100);b.write(oct(0o644,8),100);b.write(oct(0,8),108);b.write(oct(0,8),116);b.write(oct(size,12),124);b.write(oct(0,12),136);b.fill(" ",148,156);b.write("0",156);b.write("ustar\0",257);b.write("00",263);let s=0;for(let i=0;i<512;i++)s+=b[i];b.write(oct(s,8),148);return b}
  const gz=createGzip({level:1}),out=createWriteStream(process.argv[1]);gz.pipe(out);
  const w=b=>new Promise(r=>gz.write(b)?r():gz.once("drain",r));
  const pj=Buffer.from(JSON.stringify({name:"pkg",version:"1.0.0"}));
  await w(hdr("package/package.json",pj.length));await w(pj);await w(Buffer.alloc(512-pj.length%512));
  const mb=2050;await w(hdr("package/big.bin",mb<<20));const z=Buffer.alloc(1<<20);for(let i=0;i<mb;i++)await w(z);
  await w(Buffer.alloc(1024));await new Promise(r=>gz.end(()=>out.once("close",r)));
' $W/pkg.tgz
printf '{"name":"root","dependencies":{"pkg":"file:./pkg.tgz"}}' > $W/package.json
(cd $W && bun install)

On 1.4.0-canary and main:

error: ZlibError decompressing "pkg" to ".xxx-1.pkg"
error: InstallFailed extracting tarball from pkg
error: pkg@file:./pkg.tgz failed to resolve

Cause

ExtractTarball::extract() (the buffered path used for every file: tarball and for the HTTP fallback when streaming is disabled) first decompresses the entire gzip stream into a Vec<u8> via ZlibReaderArrayList, capped at MAX_DECOMPRESSED_TARBALL_SIZE = 2 GiB, then hands the decompressed tar to Archiver::extract_to_dir. Hitting the cap surfaces as a bare ZlibError. Below the cap, peak RSS is roughly the decompressed size plus growth-doubling slack: a 256 MiB package costs ~780 MB RSS just for the intermediate buffer.

libarchive already supports reading gzipped input directly: BufferReadStream::open_read registers archive_read_support_filter_gzip, backed by the same vendored zlib-ng the removed pre-decompress used.

Fix

Hand the compressed .tgz bytes straight to Archiver::extract_to_dir and let libarchive gunzip block-by-block. The 2 GiB cap and intermediate buffer are gone.

The libdeflate fast path is kept for tarballs whose gzip ISIZE trailer reports under 64 MB (unchanged threshold), so small npm packages keep the faster decoder. The trailer is now read as little-endian per RFC 1952 (was native-endian); the same one-token change is applied at the two sibling sites in src/http/InternalState.rs and src/runtime/api/BunObject.rs.

Dead code removed with the pre-decompress: bun_install::Error::Zlib and the bun_zlib workspace dependency in bun_install; ZlibReaderArrayList::max_output_size narrowed to pub(crate) now its external writer is gone.

Verification

$ bun bd test test/cli/install/bun-install-streaming-extract.test.ts
 10 pass  0 fail

buffered extract does not hold the decompressed local tarball in memory replaces the previous buffered extract rejects ... test (which asserted the cap). It installs a 256 MiB local tarball and asserts the child's maxRSS stays under 2 * PAYLOAD_SIZE. On the released bun it peaks at ~780 MB release / ~1 GB debug+ASAN and fails; with this change it stays at baseline (~40 MB release / ~240 MB debug+ASAN) and passes.

Also ran bun-install-tarball-integrity.test.ts (16 pass) and bun-install-registry.test.ts (229 pass) to confirm the common paths are unchanged.

---

no test proof · iteration 1 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/install/bun-install-streaming-extract.test.ts

[coderabbitai]

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 17 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: fb32eded-b980-4f9a-961b-be57bf283b00

📥 Commits

Reviewing files that changed from the base of the PR and between 45ccba4 and 2f74de4.

📒 Files selected for processing (3)

• src/install/extract_tarball.rs
• src/sys/lib.rs
• test/cli/install/bun-install-streaming-extract.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

Beyond the inline findings, I confirmed BufferReadStream::open_read() does register archive_read_support_filter_gzip (src/libarchive/lib.rs:944), so handing compressed bytes straight to Archiver::extract_to_dir is sound; and time_started_for_verbose_logs is still consumed by the later "Extracted to" log so it isn't dead after the first verbose block dropped its use.

Extended reasoning...

The core Rust change is correct: libarchive's buffered reader already registers the gzip filter, so passing the raw .tgz works, and the from_ne_bytes→from_le_bytes fix matches RFC 1952. The 2.25 GiB-to-disk test cost flagged inline is the item that warrants a maintainer decision before merge; the cap removal itself is a design call that will be visible to whoever looks at that.

[claude]

All prior review feedback is addressed and the current pass found nothing new, but this rewrites the buffered tarball decompression path in bun install and removes the 2 GiB MAX_DECOMPRESSED_TARBALL_SIZE guard — worth a maintainer's eyes on the design call rather than an auto-approve.

What was reviewed:

• Confirmed BufferReadStream::open_read registers archive_read_support_filter_gzip (src/libarchive/lib.rs:944), so passing compressed bytes through is sound.
• The libdeflate fast path is behaviour-preserving for ISIZE < 64 MB; on failure it now falls through to libarchive instead of the removed zlib pre-inflate.
• maxRSS unit handling in the test checked against spawn_process.rs — Windows normalises PeakWorkingSetSize / 1024, so the isMacOS ? 1 : 1024 branch is correct.
• Dead-code cleanup (Error::Zlib, bun_zlib dep, max_output_size visibility) and the two sibling from_ne_bytes → from_le_bytes sites are all applied.

Extended reasoning...

Overview

The PR changes ExtractTarball::extract (src/install/extract_tarball.rs) to stop pre-inflating gzipped tarballs into a Vec<u8> via ZlibReaderArrayList and instead hand the compressed bytes directly to Archiver::extract_to_dir, relying on libarchive's built-in gzip filter to gunzip block-by-block. The libdeflate one-shot fast path for small tarballs (ISIZE < 64 MB) is preserved. Dead code orphaned by the removal is deleted (Error::Zlib, the bun_zlib workspace dep, Cargo.lock entry), ZlibReaderArrayList::max_output_size is narrowed to pub(crate), and the from_ne_bytes → from_le_bytes ISIZE fix is applied at the two sibling sites in InternalState.rs and BunObject.rs. The oversized-tarball test is rewritten to install a 256 MiB local .tgz and assert child maxRSS < 2 * PAYLOAD_SIZE.

Security risks

The change removes MAX_DECOMPRESSED_TARBALL_SIZE = 2 GiB. I believe this is net-neutral to net-positive: the cap bounded an in-memory intermediate buffer that no longer exists, disk output was never capped (libarchive writes entry-by-entry), and the streaming HTTP path — the default for registry packages — already goes through libarchive gunzip without this cap. The buffered path applies to file: tarballs (local, user-controlled) and the BUN_FEATURE_FLAG_DISABLE_STREAMING_INSTALL fallback. Still, deleting an explicit resource bound in a package manager is the kind of judgment I'd rather a maintainer confirm than sign off on autonomously.

Level of scrutiny

High. ExtractTarball::extract is on the critical path for every bun install that touches a local or buffered tarball, and ~60 lines of its decompression logic are rewritten. This is not a mechanical change; it's a deliberate architectural swap of one decompressor for another with a user-visible behaviour change (previously-failing >2 GiB local tarballs now install).

Other factors

Four prior review rounds on this PR raised test cost (2.25 GiB → 256 MiB + maxRSS assertion), missing gzip error wiring, orphaned dead code, sibling from_ne_bytes sites, a stale comment, and a dropped stderr destructure — all addressed and marked resolved. I verified open_read() at src/libarchive/lib.rs:944 does call read_support_filter_gzip(), and that the test's Windows maxRSS unit assumption matches spawn_process.rs:138 (PeakWorkingSetSize / 1024). The verbose-install log line loses its per-decompress timing (decompressed X in Y → X tgz file); the extraction-elapsed line further down still prints. No CODEOWNERS entry covers these files. The bug hunter found nothing this run.

[robobun]

^{Updated 5:59 AM PT - Jul 31st, 2026}

❌ @robobun, your commit 755febc has 1 failures in Build #86226 (All Failures):

• test/js/node/test/parallel/test-worker-message-port-transfer-terminate.js - SIGABRT on 🐧 13 x64-asan

---

🧪   To try this PR locally:

bunx bun-pr 36541

That installs a local version of the PR into your bun-36541 executable, so you can run:

bun-36541 --bun

[robobun]

CI on build 86226: 193 lanes passed, 1 failed. The one hard failure is test/js/node/test/parallel/test-worker-message-port-transfer-terminate.js SIGABRT on debian x64-asan (JSC ExceptionScope::assertNoException()), which this PR does not touch; reported separately for main triage. Everything else is flaky and passed on retry or when run alone. test/cli/install/bun-install-streaming-extract.test.ts passed on every lane.

[robobun]

Heads up: the buffered extract does not hold the decompressed local tarball in memory test this added is red on every non-macOS lane on main because it scales resourceUsage().maxRSS by 1024 on Linux/Windows; #36087 had already normalised that value to bytes everywhere before this merged. Fixed in #36586.