Skip to content

Replace libdeflate with safe Rust - #47072

Draft
dylan-conway wants to merge 22 commits into
mainfrom
claude/libdeflate-rust-replacement-0hfpor
Draft

dylan-conway wants to merge 22 commits into
mainfrom
claude/libdeflate-rust-replacement-0hfpor

Conversation

@dylan-conway

@dylan-conway dylan-conway commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

What does this PR do?

Removes the dependency on libdeflate. src/libdeflate (bun_libdeflate) is a port of it to safe Rust.

Draft. It builds and the existing tests of every consumer pass. Still to come in this PR: the performance work with numbers (see the end).

  • src/libdeflate (new, #[forbid(unsafe_code)]). It also denies indexing, unwrap, expect, panic! and unreachable!, because streams come from anywhere and release builds abort on a panic. It holds the compressor at levels 0 to 12 (DEFLATE, zlib, gzip) and the decompressor (DEFLATE, gzip). zlib_decompress.c is not ported: nothing in Bun gives libdeflate a zlib stream, fetch leaves those to zlib-ng on purpose. README.md there explains the decode tables, the bit buffer and how output memory is handled.
  • The rule: no observable change. The compressor writes the bytes libdeflate writes, at every level and for every input. The decompressor accepts what libdeflate accepts and reports the same error, bytes read and bytes written.
  • Memory that is only reserved stays untouched. C wrote into uninitialized memory. Safe Rust cannot, and a match copies from the output. Zeroing a whole reservation first would let a lying gzip trailer cost 32 to 64 MiB, cost 128 KiB for each small WebSocket message, and make Bun.gzipSync of a large buffer resident at twice its size. So a Vec is lengthened as the result grows: the decompressor asks for as much again as there is, the compressor for one block at a time, and the compressor's large tables are Vecs filled as they are used.
  • Bun.gunzipSync and inflateSync with library: "libdeflate" no longer start over from the beginning each time the output buffer has to grow.
  • uWS is C++, so it reaches the decompressor through src/uws_sys/inflate.rs. That file holds the only new unsafe in shipped code apart from one FFI call: it dereferences the pointers C++ passes and frees the object. The logic is a safe method.
  • CRC-32 and Adler-32 come from zlib through bun_zlib. For CRC-32 that is what bun_tar and Bun.hash.crc32 already do. bun_zlib::adler32_bytes is new.
  • Debug builds compile this one crate with opt-level = 3 (debug assertions and overflow checks stay on). Unoptimized it is many times slower, and every bun install goes through it.
  • bun build --compile --aot --trim removes the compressor from executables that do not use it. main defined that feature (LibdeflateCompressor in src/trim/features/bun.def) by the names of the C functions. It now names Compressor::new, compress and compress_to_vec with a cxx-function pattern, which matches demangled Rust names too, and the three are #[inline(never)]. The macros that mark Rust entry points expand to unsafe, which this crate forbids. A minimum makes the link fail if the pattern stops matching.
  • A fix that is not about libdeflate, needed for CI to pass on this branch. The scan in test/bundler/what-an-executable-can-call.ts kept following the register of an arm64 adrp after another instruction had overwritten it. With this branch's layout that made compile-trim-capabilities.test.ts report a call of fork on Linux arm64 where there is none. Fix false alarms in the scan behind compile-trim-capabilities.test.ts #47326 has since fixed this on main for an ldr or add that reads the register and overwrites it. What is left here is five lines on top of it: the scan also stops at a second adrp or adr into the register, which is the case this branch hit, and at an ldr or add that overwrites it from another register.
  • A change taken from test: one shape of build-scaling grows 2.58 times in memory #47329, also needed for CI and also not about libdeflate: its two files under test/aot/build-scaling, as they are there. Since the mimalloc upgrade in Remove the address space ballast and the mimalloc patches, upgrade mimalloc to v3.6.0 and ICU to 79.1 #47281, one shape of build-scaling-2.test.ts grows more than the allowed 2.5 times in memory on macOS 26 arm64, on main and on other branches too. Nothing is left of it here once test: one shape of build-scaling grows 2.58 times in memory #47329 is on main.
  • Deleted: the dependency, its patch, its update workflow, src/libdeflate_sys, its entries in the baseline CPU allowlists, an unsafe impl Zeroable in the HTTP thread, and process.versions.libdeflate. The licence stays in LICENSE.md, and its text is in src/libdeflate/LICENSE.

Changes I know of:

Change Why
process.versions.libdeflate is gone As with libarchive
Failing to allocate the 13 KB of a decompressor aborts It used to fall back to zlib or throw. The large allocations of a compressor still fail gracefully
BUN_FEATURE_FLAG_NO_LIBDEFLATE=1 now also covers messages to Bun.serve and request bodies that fetch compresses Both have a zlib path to fall back to, and the flag is the way out if the port has a bug
Each uWS ZlibContext has a second 16 KiB buffer The one-shot inflater writes into initialized memory of its own, not into the malloced inflationBuffer
fetch with a custom compress level, or a body over the bound, no longer creates the cached level 6 compressor It was not used
Bun.Archive with gzip: no memory for the compressor is an out of memory error It was GzipInitFailed, as for a bad level, because the C wrapper could not tell the two apart
bun audit: a refused compression would be an error It would have sent an empty body. Neither can happen: the bound is reserved first
An input that another thread rewrites during compression can get "insufficient space" where C returned some garbage stream A block's output is bounded by the size computed for it, not by the whole buffer. Bun.gzipSync and deflateSync copy shared inputs first, as before

The first commit only removes the library from the build system, and does not build.

How did you verify your code works?

New tests, which pin what the C library did. Every expected value was recorded from a build of the merge base, and the tests pass on that build too:

  • zlib.test.js, "libdeflate": the output for nineteen inputs at levels 0 to 12 (several blocks, stored blocks, runs, changing statistics, repeats 32767 to 32769 bytes back, few symbols, so many matches that the sequence stores fill up, so many that the match cache does, no match for over 65535 bytes, only literals at levels 10 to 12, blocks that end before the position reached, a last block shorter than the minimum, and three limits of the compressor from both sides), and for every length from 0 to 79.
  • The same block: what becomes of 77 damaged or unusual DEFLATE and gzip streams. Some pin quirks: length symbols 286 and 287 decode as 258, a match decodes at distance 1 when no distance has a code, and up to 288 literal and 32 distance codes are accepted.
  • The gzip wrapper at each level there, and the zlib wrapper at each level in fetch-compress.test.ts.
  • Streams found by mutation testing: the largest decode tables there can be (2342 and 402 entries), distance symbols 30 and 31, and items that use the most bits one round of the decoder's fast loop can. Line coverage was complete without the last kind, and still a refill could be deleted from that loop with nothing failing. Each of its six refills is now needed by some test.
  • Which inflater Bun.serve uses: a stream that only libdeflate accepts (length symbol 286) gets through up to 16384 bytes, not at 16385, and not at all under BUN_FEATURE_FLAG_NO_LIBDEFLATE=1. The same flag for fetch with compress.
  • Matches of 113 lengths from 50 distances, checked against a byte by byte copy. Changing any branch or constant of the code that copies a match makes it fail (22 such changes tried).
  • "Compressor and Decompressor", 17 tests over libdeflate_test.bind.ts, a binding that only builds with assertions have (debug, CI's ASAN lane): output of the exact size and one byte less, a Vec's spare capacity, how far it grows, how far past the result it is written to, what an error leaves, how much input was read (gzip members too), and reuse of a compressor or decompressor, also after a call that had no room. Miri is not an option for these: creating 13 compressors takes it over 100 seconds, and the crate has no unsafe for it to check.

Against the C library, in one process (outside the repo: libdeflate at Bun's pin and patch, linked next to the crate):

  • Compressor: about 1.2 million compressions, 22 GB of input, at all 13 levels, in all three encodings, into buffers of the bound, of the exact size, one byte less, and random smaller sizes. Every result is byte for byte the same, refusals included. Compressors are reused across inputs and replaced at random. Inputs go up to 4 MB: random, few symbols, runs, periodic, words, DNA, real files, and self-copies with offsets around the 32 KiB window.
  • Decompressor: about 3.5 million streams, each through the slice, spare-capacity and growing APIs. Status, bytes read, bytes written and content are the same. The streams are libdeflate's own output, mutated output, and streams from a generator that writes what libdeflate never does: 15-bit codes, incomplete and overfull codes, every run-length coding of the code lengths, junk after the end, and gzip headers with every optional field.
  • Part of both ran on a build with overflow checks and debug assertions.
  • A Vec reserved at exactly the right size is never reallocated (61,000 streams).

Old binary against new binary (debug builds of the merge base and of this branch):

  • gzipSync and deflateSync at levels 0 to 12 on eight kinds of input: identical bytes.
  • Error messages for bad levels, garbage, cut-off streams and a bad CRC: identical.
  • fetch of gzip, zlib, raw deflate, two-member and corrupt bodies, below and above the 512 KiB shared buffer: identical.
  • fetch with compress, both encodings at nine levels: identical bytes on the wire.
  • WebSocket echo with permessage-deflate between Bun's client and Bun.serve, from 0 to 300,000 bytes.
  • Bun.Archive with gzip, and bun install of a tarball: identical trees.

Existing tests, on a debug build on Linux x64:

  • zlib.test.js, zstd.test.ts, fetch-gzip, fetch-compress, fetch.stream, archive.test.ts, archive-golden, bun-audit, bun-install-streaming-extract, websocket-server.test.ts, and the four permessage-deflate suites.
  • They pass, with two exceptions that fail the same way on the merge base in my container. Four tests of process.test.js depend on the host. The "multiple parts" group of fetch.stream.test.ts needs more than the default 5 s, without compression too, and passes on both builds with a longer timeout.

--trim, in a trimmable release build of this branch merged with main (local WebKit, without LTO, and with the step for ICU's data switched off, because the system's ICU is a shared library):

Program, built with --compile --aot --trim Compressor code kept Decompressor code kept
console.log("hello") 99 of 65,407 bytes 0 of 11,425
only Bun.gunzipSync 99 11,280
only new WebSocket(...) 457 11,425
the same, with the feature still defined by the C names 65,407 11,425
Bun.gzipSync and Bun.deflateSync 65,407 11,280
  • All of them run, and the last writes the bytes that the whole Bun writes.
  • With the pattern changed to match nothing, the link fails: FeatureBelowMinimum: 0.00 MB can be removed with LibdeflateCompressor.

libdeflate's history. There is no CVE or advisory for libdeflate (NVD, OSV, Debian's tracker, the repository's advisories), and nothing upstream is newer than Bun's pin. I read the subjects of all 796 commits, NEWS.md and all 174 issues. About 50 commits change the behaviour of code that is ported.

  • libdeflate's six regression programs (test_incomplete_codes, test_invalid_streams, test_litrunlen_overflow, test_overread, test_trailing_bytes, test_slow_decompression) pass unchanged against the port, linked through a C shim.
  • The reproducers written out in issues 279, 420 and 429 give what C gives.
  • I put 15 of the old bugs back into the crate, one at a time, built Bun and ran the "libdeflate" tests of zlib.test.js. Those that nothing in this repo noticed at first got a test.
Bug put back Fixed upstream in Tests that fail
No limit to reading past the end of the input (issue 157) 3f21ec9 6, among them "a stream cut off where zeros are literals" (new); then the run hangs
Too many code lengths accepted (issue 288) 65da376 "too many code lengths, accepted until 1.15" (new)
A code of one symbol decodes as symbol 0 446e226, 480e031 3, two of them new
A match one byte too long for the output accepted d5e5933 11
A run of over 65535 literals corrupts the output (0.6) f2f0df7 input noMatches (new)
Incomplete Huffman codes written, which Windows Explorer refuses (issue 323) f52567f 8, among them "writes complete Huffman codes" (new)
Levels 10 to 12 can exceed the bound (1.16) ea2c695 4, among them input skewed (new)
The match cache overflows 4f7d134 input coins
Level 1's sequence store overflows d352945 inputs coins and shortMatches
A match can be chosen that reaches past its block 4d81654 1 or 5, depending on how it is put back; see below
Output of the exact size refused (issue 102) b8f2e78 19
The bound too small 7e19d17, 5f4da4b "the bound is 5 bytes for each 5000, and the wrapper"
gzip with unusual XFL bits refused 0b6d0f4 "FTEXT, a time, every XFL bit and another OS" (new)
gzip trailer read as big endian 19e1688 11
Static tables rebuilt for every block (issue 33) 57cab07 none: it only costs time. libdeflate's test_slow_decompression notices
  • The fuzzing session went through the history on its own (214 commits touch ported files) and compared the fragile places with the pinned C statement by statement. It found no difference, and four changes of behaviour under subjects that do not say "fix". Put back the same way: 7380364 (junk code lengths priced) fails 10 tests, 5a9d25a (gzip footer taken from the end of the input) 2, 55f9f70 (all statistics cleared after going back) changes the output for inputs mixed and shortMatches (seen outside Bun: that edit leaves dead code, which Bun's build refuses), and bfa6ddb (shortest match worked out from the wrong position after going back) failed none, so input alternating is new.
  • One weakness came out of it. An optimization of mine left the previous block's nodes in place at levels 10 to 12. Its output was right, but with it the 4d81654 bug could be put back and no test here noticed. Each block starts from fresh nodes again.
  • Not applicable: bugs in the CRC-32 and Adler-32 kernels (zlib-ng computes both here), in zlib_decompress, the custom allocator and the gzip program, and fixes for platforms, ABIs and build systems.
  • Not checked: files attached to issues, which I could not download.

Fuzzing, in a separate session against this branch, everything replayed on 2955608a (since then the crate only got three #[inline(never)]): 446 million coverage-guided executions (360 million of them on the code since 2955608a), about 1.3 billion directed decompressions and 50 million directed compressions against the C library, 7 million calls with an input that other threads rewrite, inputs over 4 GiB, an allocation failure injected at every allocation in turn, and the C library with its own assertions on. No difference, panic or hang. Exhaustive runs, also without a difference: every DEFLATE stream of 0 to 4 bytes; every input of 0 to 20 symbols over 2 letters (shorter over 3 and 4), alone and after six prefixes, at all 13 levels (1.09 billion compressions into a slice of the bound, of the exact size and of one byte less; 518 million more on slightly shorter inputs that also go through compress_to_vec; 821 million at every output size from 0 to nine bytes over the bound); every 5-byte stream that starts a dynamic block (275 billion calls, but none of them decodes, so this shows that no such header panics or hangs, not that both accept the same codes); and for that, every assignment of code lengths to chosen sets of symbols, each followed by streams that are valid under it (35 billion calls, 122 million of which decode: up to 9 symbols of the code for the code lengths, and 8 symbols of the literal/length and offset codes with lengths up to 15 bits, subtables included; four bugs planted in the checks of the code lengths were each caught first; codes with many different long lengths at once are not enumerated). Every 5-byte stream that starts a static block: 1.1 trillion calls, 126 billion of which decode. Stored and reserved block types are not enumerated at 5 bytes. After change (4) below had slipped through, the kind of input that shows it was run on the crate itself, at all 13 levels, byte for byte against C: 29.9 million windows of 5,001 to 700,000 bytes over ordinary files and 780,000 compressions of generated texts whose vocabulary drifts. No difference. The decoder also got streams that zlib wrote, since all valid streams until then came from libdeflate or from generators: 60,000 streams at every level, strategy, memLevel, window size and kind of flush, each whole, with bytes after it, cut short and with single bits flipped (1.36 million variants, 657,000 of which decode). No difference. Mutation testing of the whole crate, 4,096 mutants (single-token edits and deleted lines): 229 do not compile, 3,552 are killed, and 315 give equal results, meaning the same return value and bytes for an input that stands still; time and memory may differ. None is known to be alive. An earlier version of this text said 3,548 and 319, all equivalent. Four of those 319 were real changes, found while writing down a reason for each. The crate is right in all four places; the checks were too weak. (1) Without the out.truncate(start) in the failure branch of compress_to_vec, the Vec keeps partial bytes after an error. The branch only runs when another thread rewrites the input during the call. (2, 3) bitcount >= 8 weakened, in two spellings, in the byte-wise flush near the end of the room: it trips the crate's own debug_assert, only with such an input and only with debug assertions on. (4) num_new_observations * 200 / 512 changed to / 511 in the decision to end a block. It was taken for equivalent because a decision was thought to come at exactly 512 new observations, but at levels 5 to 9 it also comes at 513 to 516, and at 514 the quotient differs. Nothing had caught it: in a build of Bun with that change all 140 "libdeflate" tests pass, while 30,000 bytes of bun.d.ts compress to 8,565 bytes at level 6 instead of C's 8,580. All 319 have since been run with inputs that are rewritten during the call, those outside the decoder over about 9,300 windows of ordinary files and 2,400 generated texts, and those of the decoder and of the handling of room through the enumerations above. Each of the 315 has a written reason, by one reader. Three that had survived a thinned set of checks were real changes (the limits of levels 11 and 12, and the 5000 bytes that must be left for a block to end). Nothing in this repo noticed them either, so zlib.test.js now pins those limits from both sides: with a limit moved by one either way, in a build of Bun, exactly the input meant for it fails. Eighteen other off-by-one changes to constants of the heuristics are caught by the fuzzing suites only, not by this repo's tests: they need inputs built for an exact boundary. With (4) that makes nineteen that this repo does not pin. (1) to (3) need an input that changes during the call, so no deterministic test can pin them. Ten mutants needed an exact numeric tie in the block-splitting and cost heuristics, so inputs that land exactly on those ties were constructed (about 10,900 compressions, no difference from C). It also read every caller against the merge base for changes that the table above does not list, and found the two about Bun.Archive and bun audit.

Performance so far

Speed relative to the C library, 1.0 being the same. Both are in one process, built with the flags of a release build (-march=nehalem, frame pointers kept), on a Cascade Lake Xeon. The C library picks its BMI2 decompressor at run time, which is worth 10% to it. Safe Rust compiled for Nehalem cannot.

On this family of CPUs (Skylake to Cascade Lake) a jump that crosses a 32-byte boundary is costly, and the bounds checks give the Rust loops more jumps than C's have. The same decoder ran at 0.74 to 0.93 depending on nothing but its address. So there are two columns: "as built" is this build on this CPU, and "padded" is built so that no jump crosses such a boundary, which is about what other CPUs see. The decoder's loop is aligned to 64 bytes (#[feature(fn_align)]) so that other code cannot move it. Where its jumps fall has to be looked at again in a release build of Bun. src/libdeflate/README.md has the details.

Decompressing first port as built padded
Rust and TypeScript sources 0.55 0.86 0.94
an executable 0.67 0.87 0.90
base64 0.80 0.97 0.98
zeros 1.2 1.17 1.15
long matches from far back 0.70 0.95 0.96
repeats every 8 to 41 bytes 0.72 1.02 1.07
runs of one byte 0.81 0.85 0.89
16-byte records 0.51 0.72 0.75
repeats every 2 to 7 bytes 0.42 0.69 0.73
inputs of 128 to 8192 bytes 0.85 to 0.90
inputs of 8 and 32 bytes 0.78
Compressing sources and an executable first port as built padded
level 1 0.76 0.83 0.93
levels 2 to 6 0.77 to 0.79 0.79 to 0.82 0.86 to 0.89
levels 8 and 9 0.90 and 0.93 0.90 and 0.93 0.93 and 0.96
levels 10 to 12 0.79 to 0.81 0.81 0.92 to 0.93

Still to do here

  • More of the compressor's performance, the decoder's weak cases (short periods, records), and numbers from release builds of Bun.

Generated by Claude Code

The C library is no longer fetched, patched, compiled or linked. Its update
workflow, its patch, its version macro and its entries in the baseline CPU
allowlists are gone too.

The build fails after this commit: uWS still includes libdeflate.h,
BunProcess.cpp still reads BUN_VERSION_LIBDEFLATE_HASH, and bun_libdeflate_sys
still declares the C functions. The next commits replace them with Rust.
`src/libdeflate` (`bun_libdeflate`) replaces the C library and its bindings in
`src/libdeflate_sys`. It forbids `unsafe` and denies indexing, `unwrap`,
`expect`, `panic!` and `unreachable!`.

The compressor writes the bytes libdeflate writes at every level, and the
decompressor accepts what libdeflate accepts, with the same error, bytes read
and bytes written.

- Output buffers are lengthened as they fill, so memory that is only reserved
  stays untouched, as it did when C wrote into uninitialized memory.
- uWS reaches the decompressor through `src/uws_sys/inflate.rs`.
- CRC-32 and Adler-32 come from zlib through `bun_zlib`.
- The crate is optimized in debug builds too.
- `process.versions.libdeflate` is gone.
@robobun

robobun commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator
Updated 10:19 AM PT - Oct 29th, 2026

✅ @claude, your commit cbf095e1daa62ea8832745ed0bdfd263407b3cc1 passed in Build #135188! 🎉


🧪   To try this PR locally:

bunx bun-pr 47072

That installs a local version of the PR into your bun-47072 executable, so you can run:

bun-47072 --bun

autofix-ci Bot and others added 6 commits October 27, 2026 12:30
The tests record what the C library wrote for seven kinds of input at
levels 0 to 12, and what it made of 70 damaged or unusual DEFLATE and
gzip streams. They pass on a build that links the C library.

Nothing in Bun gives libdeflate a zlib stream to decompress, so that
decoder goes, and Decompressor takes a Decoding (Deflate or Gzip).
…ding

libdeflate_test.bind.ts reaches what Bun.gzipSync and the like do not
show: output of an exact size, a Vec's spare capacity, how far it grows,
what an error leaves, how much input was read, and reuse of a compressor
or decompressor. It is only in builds with assertions.

Also pins streams that need the largest decode tables, distance symbols
30 and 31, and items of the most bits there can be. Without the last,
deleting a refill from the decoder's fast loop went unnoticed.
The test keeps a list of every .bind.ts and what it is published as.
This one is published as nothing, like the other two test-only ones.
On text it ran at 0.55 of the speed of the C library, built with the
flags of a release. Now it runs at 0.92, and at 0.95 to 1.0 where jumps
on 32-byte boundaries cost nothing.

- The loop is a function of its own, so that its state stays in
  registers, with a copy for a litlen table of the full size.
- A match is copied through registers, 40 bytes with one bounds check.
  One that overlaps itself is stored as a repeated pattern and never
  read back.
- Bit masks are looked up.
- The function is aligned to 64 bytes. On Skylake to Cascade Lake its
  speed varied by 20% with its address alone.

README.md has the reasons and how to measure. New tests copy matches of
113 lengths from 50 distances, and need each refill of the loop.
@dylan-conway
dylan-conway force-pushed the claude/libdeflate-rust-replacement-0hfpor branch from 1436dbe to 8d44b1d Compare October 27, 2026 17:08
No change in what is compressed or decompressed.

- The decompressor grows its output in place. It used to stop, hand its
  state back and be resumed in a longer output, decoding one item twice.
- One `Output` type serves the compressor and the decompressor.
- Files, constants and functions carry libdeflate's names where the
  thing is libdeflate's: deflate_compress.rs, deflate_decompress.rs,
  DEFLATE_*, GZIP_*, ZLIB_*, fastloop.
- `Level` is a type, so `Compressor::new` only fails for lack of memory
  and says so. `decompress_to_vec_grow` appends like its sibling.
- The first step of a growing output is 512 bytes, not 4096: a short
  WebSocket message costs half as much.
- skip_bytes checks its bounds once, not per position, which makes
  levels 1 to 9 faster.
- BUN_FEATURE_FLAG_NO_LIBDEFLATE now also covers messages to Bun.serve
  and request bodies that fetch compresses.

Tests: inputs that fill the sequence stores and the match cache, which
inflater Bun.serve uses on either side of 16 KiB and under the flag, how
far past its result a Vec's spare capacity is written to, and the zlib
wrapper at all 13 levels. The test binding reports a broken invariant as
an error of one test and no longer aborts the file.
Every bug fix in libdeflate's history that touches ported code was
checked against the port. None is present. Those that had no test here
get one:

- The streams of libdeflate's own regression tests: codes of one symbol,
  too many code lengths, and a stream cut off where the zeros that stand
  in for missing input decode to literals for ever.
- Its input without a match for over 65535 bytes, which 0.6 corrupted.
- An input that levels 10 to 12 write as literals only, which in 1.16
  could exceed the bound.
- Every Huffman code written is complete, which Windows Explorer insists
  on and 1.18 did not guarantee.
- A gzip header with every XFL bit, a time and another OS.

Levels 10 to 12 start each block from nodes that are all too expensive
again. Leaving the previous block's nodes in place made the result rest
on one fill that no test could see failing.
… back

When those levels end a block before the position they have reached, the
shortest match worth counting is worked out from the start of the next
block. libdeflate worked it out from the position reached until 2022.
Doing that again in the port changed no test's result. It does now.
At levels 10 to 12 a call that runs out of room can leave matches in the
cache. The next call clears it, and nothing failed without that: the
leftovers only matter when the next input's first block fills the cache.

- Input `fourSymbols` does that. Pinning its output also covers a last
  block shorter than the minimum, which only a full cache leads to.
- The test binding can give the first input too little room.

Copy link
Copy Markdown
Member Author

Build #134048 on 9d96ac3 is marked as failing because its two darwin any x64 - test-bun jobs expired: they waited 78 minutes for an agent and never started, so no test ran. The other 179 jobs passed. BuildKite has queued both again. Nothing to fix in this PR; the same happened on 1436dbe, which went green once the retries got an agent.


Generated by Claude Code

Each could be moved by one in either direction with no test failing:

- Levels 11 and 12 also parse for the static codes, up to 1000 and 10000
  bytes. That shows only on data that those codes suit best.
- A block can end where the statistics change only if 5000 bytes are left.

Six inputs, each of which fails for exactly one of the six changes.
main defines the removable feature LibdeflateCompressor by the names of
the C functions, which are gone. A program that keeps the HTTP client but
compresses nothing, such as one that only opens a WebSocket, would keep
all 65 KB of the compressor.

- The feature now names Compressor::new, compress and compress_to_vec,
  which stay out of line so that there is something to cut.
- It has a minimum, so the link fails if the pattern stops matching.
- libdeflate leaves the list of C libraries built with SafeStack, and
  LIBDEFLATE_DISABLE_CPU_FEATURES the list of variables Bun reads.
Its 380,880 comparisons take Miri hours, so `cargo miri test` has not
finished on any pull request since the test landed. The same change is
in #47181; this one goes away when that lands.

Copy link
Copy Markdown
Member Author

cargo miri test never finished on 4ab6113 (over 3 hours, GitHub cancels it at 6). That is not from this PR: #45631 added a bun_sema test whose 380,880 comparisons take Miri hours, and the job runs on the merge with main. #47181 fixes it. I merged main and ported its two lines in src/sema/check/regexp_scanner.rs (b2adb51); they become a no-op once #47181 lands. With them, bun run rust:miri -p bun_sema finishes here in 10 minutes (226 passed, 250 ignored).

The merge also showed that main's --trim still named the C functions of libdeflate; 971e000 adapts that, and the description has the details.


Generated by Claude Code

The scan for places that call a function follows the register of an
`adrp` through the next four instructions, also after it was written to:

    adrp x0, page
    add  x0, x0, 0x530
    ldr  x1, [x0]

The last line read as a use of `page + 0`. Where an import's slot lies
at such an offset of a page that also holds globals, each of these
counted as a call. In the released 1.4.4 for Linux arm64 that gives 47
calls of `pipe` and 18 of five other functions. The other 59,146 places
found there stay the same.

Which import is hit depends on the layout. On this branch it is `fork`,
so compile-trim-capabilities.test.ts failed on Linux arm64 only. That
test now also says which features keep a place, as its neighbour does.

dylan-conway commented Oct 28, 2026 •

Copy link
Copy Markdown
Member Author

CI status (I keep this comment up to date; last edit for cbf095e)

Now: cbf095e takes the two files of #47329 as they are there, to get macOS arm64 green. Nothing else changes, and nothing is left of the commit once #47329 is on main.

CI on cbf095e: green. #135188 passed at 17:19 UTC: 188 of 188 jobs, on every platform, no failure of any kind, soft ones included. GitHub checks: all green. macOS arm64 got agents at the third attempt (macOS 26.4, 26.6.2 and 15.7.7), macOS x64 at the fourth.

  • The change from test: one shape of build-scaling grows 2.58 times in memory #47329 does what it should: build-scaling-2.test.ts passes on macOS 26.4 arm64, 4 pass and 0 fail, with the shape at 2.66 times. That is over the old bound of 2.5 and under the new one of 2.84.
  • This PR's suites, with the same counts on macOS arm64 and x64: zlib.test.js 581 pass, archive.test.ts 170, archive-golden 52, bun-tar 98, bun-install-streaming-extract 134, websocket-server-permessage-deflate 65, zstd.test.ts 95, test/internal/trim/features.test.ts 7. fetch-compress 35 on x64. compile-trim-capabilities.test.ts, fetch-gzip and the three client permessage-deflate suites ran in batches without a failure.
  • compile-trim-capabilities.test.ts now passes on macOS x64 too, where "holds no place that calls dlopen" had failed in #134483 (last table below).

The merge before it, 5ff31b2: main had come to conflict with this branch again, so it merges da867c6. One conflict, in test/bundler/what-an-executable-can-call.ts: #47326 fixed on main part of what this branch had fixed in the arm64 scan.

  • The file is now main's with five lines changed. main stops following the register of an adrp at an ldr or add that reads and overwrites it. It does not stop at a second adrp into the register, which is the case this branch hit (under "Solved" below), so that stays.
  • describePlace is gone, because main now reports the bytes around a place. compile-trim-capabilities.test.ts is main's again.
  • Checked on an aarch64 executable made for it, with one call of fork, the sequence below and the one from Fix false alarms in the scan behind compile-trim-capabilities.test.ts #47326. The scan before both fixes reports all three, main's reports the call and the sequence below, the merged one reports only the call. On the released 1.3.0 for Linux aarch64 and 1.4.4 for Linux x64 the merged scan finds the same places as the one in 1db0a6c (17,936 and 26,288, for 27 names).
  • src/libdeflate and src/zlib are unchanged. Before the push, on a debug build of the merged tree on Linux x64 (with the new WebKit and mimalloc): zlib.test.js 598 pass, archive.test.ts 170, archive-golden 52, bun-tar 98, fetch-gzip 89, fetch-compress 35, the four permessage-deflate suites 97, bun-install-streaming-extract 134, zstd.test.ts 99, test/internal/trim/features.test.ts 7, no failure. Six inputs at levels 0 to 12, as gzip and as raw DEFLATE, give the bytes that 1.4.4 with the C library gives.

CI on 5ff31b2: one failure, the one above. #135058: all builds, all of Linux and Windows, and darwin 27 aarch64 - test-bun (one job that runs the whole suite on macOS arm64, 7,773 entries) passed. GitHub checks: all green.

  • macOS 26 arm64 (darwin any aarch64) got agents at the fourth attempt: 3 of its 4 jobs passed, the fourth has the failure above and no other. This PR's suites pass there: zlib.test.js 581, archive.test.ts 170, archive-golden 52, bun-tar 98, bun-install-streaming-extract 134, zstd.test.ts 95, test/internal/trim/features.test.ts 7, and compile-trim-capabilities.test.ts, fetch-gzip and the three client permessage-deflate suites in batches without a failure.
  • macOS x64 never got an agent (three expiries, a fourth round was queued when the push cancelled the build), so it has not run since 73ffb98.
  • compile-trim-capabilities.test.ts, with the merged scan, ran on all six Linux platforms that can trim (Alpine, Debian, Ubuntu; aarch64 and x64), in batches that report 0 failures. The runner now runs several files in one batch, so there is no count for this file alone.
  • In the macOS 27 job, 4 tests of 3 files failed in a batch of 1,747 files run in parallel (serve-direct-readable-stream.test.ts, node-http.test.ts, native-source-onclose-leak.test.ts). The runner ran the three again alone and they passed: 171, 314 and 4 tests. None is about compression.

CI on 1db0a6c, the first merge: no test failed. #134890: 180 of 180 jobs on Linux and Windows passed, and all 4 test-bun jobs on macOS arm64, at the third attempt to get agents. The 4 on macOS x64 had expired three times without starting when this push cancelled the build. Those expiries are the only reason it was marked as failed. compile-trim-capabilities.test.ts passed on macOS arm64 and on all six Linux platforms that can trim (Alpine, Debian, Ubuntu; aarch64 and x64). With main's new pipeline there are no test-aot and test-aot-drawn steps any more: those tests run in the test-bun shards.

Before the merges: green on 73ffb98. #134483 passed: 308 jobs, on every platform, macOS included (arm64 and x64: all of test-bun and test-aot, at the fourth attempt to get agents). 74 soft failures, all in test-aot-drawn. GitHub checks: all green.

Solved: compile-trim-capabilities.test.ts passes on Debian 13 aarch64 with 73ffb98 (#134483: 3 pass, 0 fail). Before, it found one place that calls fork in an executable trimmed of everything that starts programs. It was the only hard failure of #134476, in which 247 jobs had passed when the next push cancelled the rest.

It is a mistake of the test's scan on arm64, not a call. With e9aa197 the failure prints the words at the place:

76a6ee8  adrp x8, 0x8702000
76a6eec  str  x0, [x8, #0xa68]
76a6ef0  adrp x8, <another page>
76a6ef4  add  x8, x8, #0x160

The scan added the 0x160 to the first page, although the second adrp had replaced x8. That gives 0x8702160, where this build's slot for fork happens to lie. So it depends on the layout, which is why only this branch hit it.

  • 73ffb98 ends the scan at an adr, adrp, ldr or add that writes to the register.
  • Checked by writing sequences into the released arm64 1.4.4. The one above and two more with an overwritten register were reported and no longer are. Five real uses are still reported, among them one with a store of the register in between and one with an adrp of another register in between.
  • The 59,146 places the scan finds in that binary stay the same.
  • My first explanation was incomplete. 13fbef1 fixed the same kind of mistake for a register overwritten by an ldr or add that reads it (65 false places in 1.4.4), but not this case. The message of that commit claims more than it should.

Not from this PR:

Failure Cause
compile-trim-capabilities.test.ts, "holds no place that calls dlopen", on macOS x64 (#134483, a soft test-aot-drawn job) Two calls of dlopen in code that goes with WorkerConstructor. Another test than the one above, and on x64, where the scan is unchanged. The same failure with the same keptBy is on macOS arm64 in #134328, a branch without this PR. This PR touches neither workers nor anything that calls dlopen. It passes on macOS arm64 in #134890, and on macOS arm64 and x64 in #135188
lol-html cargo test (e9aa197) "Setup Bun" downloaded at about 2 KB/s and ended with curl: (56) Recv failure: Connection reset by peer. cargo test never ran
test/regression/issue/ctrl-c.test.ts on six Linux platforms (#134463) It installs vite from the real registry, which answered 404 for rollup-4.66.0.tgz
One shard of windows 11 aarch64 - test-bun cancelled (#134463) "Failed to create agent"
test_init_order and test_symbol (N-API) on two Windows shards (#134446) node-gyp timed out 4 times between 12:54 and 13:05 UTC. From 13:07 the next N-API tests of the same shard built in seconds, with the same Bun
TypeScript types cancelled after 30 minutes (b2adb51) It hung in "Setup Bun", 12:32 to 13:02 UTC, and checked nothing
test-aot-drawn jobs (soft failures) They also fail on main (4 of 14 in #134092) and on other branches. No log of the 72 failed jobs of #134446 mentions libdeflate or a removed feature. The two crashes reported in #134483 are aborts of the AoT compiler and of its validator (Stopped while compiling … in GenerateCode, AOT validation: ValueType … the empty value)
macOS test jobs expire No agent takes them, on other branches too. In #134483 three rounds expired without starting before the fourth ran

Generated by Claude Code

compile-trim-capabilities.test.ts still finds one place that calls `fork`
on Linux arm64 on this branch, at the same address, in bytes that go with
no feature. So the scan's mistake that the previous commit fixed is real,
but it is not what fails here, contrary to what that commit's message says.

An address alone does not tell what it is. The failure now shows whether
the place is code or data, the words around it, where the import's slot
and stub are, and how many branches lead to it.
compile-trim-capabilities.test.ts reported a call of `fork` on Linux arm64
on this branch. The words the previous commit made it print are:

    adrp x8, 0x8702000
    str  x0, [x8, #0xa68]
    adrp x8, <another page>
    add  x8, x8, #0x160

The scan added the 0x160 to the first page, which gives 0x8702160, where
this build's slot for `fork` happens to lie. The second `adrp` had already
replaced the register, so the code has nothing to do with `fork`.

The scan now stops at an `adr`, `adrp`, `ldr` or `add` that writes to the
register, whichever register it reads. Only instructions that are certain
to write end it, so no real use is passed over: a store of the register,
or an `adrp` of another register, in between still leads to a report.

The 59,146 places it finds in the published Linux arm64 build of 1.4.4
stay the same.
Conflicts:
- Cargo.toml: both `[profile.dev.package.*]` sections stay, this branch's for
  bun_libdeflate and main's for bun_sema_standalone.
- src/sema/check/regexp_scanner.rs: main's version. It skips the test under
  Miri itself, so the skip this branch had taken over is no longer needed.
- test/bundler/compile-trim-capabilities.test.ts: main's rewrite, with the
  failure of the `fork` assertion again saying what lies at the place.
One conflict, in test/bundler/what-an-executable-can-call.ts: #47326 fixed
on main part of what this branch had fixed in the arm64 scan.

- The file is main's, with five lines changed. main stops following the
  register of an `adrp` at an `ldr` or `add` that reads and overwrites it.
  The scan here also stops at a second `adrp` or `adr` into the register,
  which is the case this branch hit on Linux arm64, and at an `ldr` or
  `add` that overwrites it from another register.
- `describePlace` is gone: main now reports the bytes around a place, so
  compile-trim-capabilities.test.ts is main's again.
`test/aot/build-scaling/build-scaling-2.test.ts` fails on macOS 26 arm64
since the merge of main: one shape grows 2.51 to 2.64 times in memory and
2.5 is allowed. It comes from the mimalloc upgrade in #47281, not from this
branch: main's own build after #47281 failed it twice before it passed, and
another branch fails it four times of four with the same numbers.

These are the two files of #47329 as they are there, which records 2.58 for
the shape, so the bound becomes 2.84. Nothing is left of this commit once
#47329 is on main.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants