Repository navigation
Replace libdeflate with safe Rust - #47072
dylan-conway wants to merge 22 commits into
Conversation
The C library is no longer fetched, patched, compiled or linked. Its update workflow, its patch, its version macro and its entries in the baseline CPU allowlists are gone too. The build fails after this commit: uWS still includes libdeflate.h, BunProcess.cpp still reads BUN_VERSION_LIBDEFLATE_HASH, and bun_libdeflate_sys still declares the C functions. The next commits replace them with Rust.
`src/libdeflate` (`bun_libdeflate`) replaces the C library and its bindings in `src/libdeflate_sys`. It forbids `unsafe` and denies indexing, `unwrap`, `expect`, `panic!` and `unreachable!`. The compressor writes the bytes libdeflate writes at every level, and the decompressor accepts what libdeflate accepts, with the same error, bytes read and bytes written. - Output buffers are lengthened as they fill, so memory that is only reserved stays untouched, as it did when C wrote into uninitialized memory. - uWS reaches the decompressor through `src/uws_sys/inflate.rs`. - CRC-32 and Adler-32 come from zlib through `bun_zlib`. - The crate is optimized in debug builds too. - `process.versions.libdeflate` is gone.
|
Updated 10:19 AM PT - Oct 29th, 2026
✅ @claude, your commit cbf095e1daa62ea8832745ed0bdfd263407b3cc1 passed in 🧪 To try this PR locally: bunx bun-pr 47072That installs a local version of the PR into your bun-47072 --bun |
The tests record what the C library wrote for seven kinds of input at levels 0 to 12, and what it made of 70 damaged or unusual DEFLATE and gzip streams. They pass on a build that links the C library. Nothing in Bun gives libdeflate a zlib stream to decompress, so that decoder goes, and Decompressor takes a Decoding (Deflate or Gzip).
…t-replacement-0hfpor
…ding libdeflate_test.bind.ts reaches what Bun.gzipSync and the like do not show: output of an exact size, a Vec's spare capacity, how far it grows, what an error leaves, how much input was read, and reuse of a compressor or decompressor. It is only in builds with assertions. Also pins streams that need the largest decode tables, distance symbols 30 and 31, and items of the most bits there can be. Without the last, deleting a refill from the decoder's fast loop went unnoticed.
The test keeps a list of every .bind.ts and what it is published as. This one is published as nothing, like the other two test-only ones.
On text it ran at 0.55 of the speed of the C library, built with the flags of a release. Now it runs at 0.92, and at 0.95 to 1.0 where jumps on 32-byte boundaries cost nothing. - The loop is a function of its own, so that its state stays in registers, with a copy for a litlen table of the full size. - A match is copied through registers, 40 bytes with one bounds check. One that overlaps itself is stored as a repeated pattern and never read back. - Bit masks are looked up. - The function is aligned to 64 bytes. On Skylake to Cascade Lake its speed varied by 20% with its address alone. README.md has the reasons and how to measure. New tests copy matches of 113 lengths from 50 distances, and need each refill of the loop.
1436dbe to
8d44b1d
Compare
No change in what is compressed or decompressed. - The decompressor grows its output in place. It used to stop, hand its state back and be resumed in a longer output, decoding one item twice. - One `Output` type serves the compressor and the decompressor. - Files, constants and functions carry libdeflate's names where the thing is libdeflate's: deflate_compress.rs, deflate_decompress.rs, DEFLATE_*, GZIP_*, ZLIB_*, fastloop. - `Level` is a type, so `Compressor::new` only fails for lack of memory and says so. `decompress_to_vec_grow` appends like its sibling. - The first step of a growing output is 512 bytes, not 4096: a short WebSocket message costs half as much. - skip_bytes checks its bounds once, not per position, which makes levels 1 to 9 faster. - BUN_FEATURE_FLAG_NO_LIBDEFLATE now also covers messages to Bun.serve and request bodies that fetch compresses. Tests: inputs that fill the sequence stores and the match cache, which inflater Bun.serve uses on either side of 16 KiB and under the flag, how far past its result a Vec's spare capacity is written to, and the zlib wrapper at all 13 levels. The test binding reports a broken invariant as an error of one test and no longer aborts the file.
Every bug fix in libdeflate's history that touches ported code was checked against the port. None is present. Those that had no test here get one: - The streams of libdeflate's own regression tests: codes of one symbol, too many code lengths, and a stream cut off where the zeros that stand in for missing input decode to literals for ever. - Its input without a match for over 65535 bytes, which 0.6 corrupted. - An input that levels 10 to 12 write as literals only, which in 1.16 could exceed the bound. - Every Huffman code written is complete, which Windows Explorer insists on and 1.18 did not guarantee. - A gzip header with every XFL bit, a time and another OS. Levels 10 to 12 start each block from nodes that are all too expensive again. Leaving the previous block's nodes in place made the result rest on one fill that no test could see failing.
… back When those levels end a block before the position they have reached, the shortest match worth counting is worked out from the start of the next block. libdeflate worked it out from the position reached until 2022. Doing that again in the port changed no test's result. It does now.
At levels 10 to 12 a call that runs out of room can leave matches in the cache. The next call clears it, and nothing failed without that: the leftovers only matter when the next input's first block fills the cache. - Input `fourSymbols` does that. Pinning its output also covers a last block shorter than the minimum, which only a full cache leads to. - The test binding can give the first input too little room.
|
Build #134048 on 9d96ac3 is marked as failing because its two Generated by Claude Code |
Each could be moved by one in either direction with no test failing: - Levels 11 and 12 also parse for the static codes, up to 1000 and 10000 bytes. That shows only on data that those codes suit best. - A block can end where the statistics change only if 5000 bytes are left. Six inputs, each of which fails for exactly one of the six changes.
…t-replacement-0hfpor
main defines the removable feature LibdeflateCompressor by the names of the C functions, which are gone. A program that keeps the HTTP client but compresses nothing, such as one that only opens a WebSocket, would keep all 65 KB of the compressor. - The feature now names Compressor::new, compress and compress_to_vec, which stay out of line so that there is something to cut. - It has a minimum, so the link fails if the pattern stops matching. - libdeflate leaves the list of C libraries built with SafeStack, and LIBDEFLATE_DISABLE_CPU_FEATURES the list of variables Bun reads.
Its 380,880 comparisons take Miri hours, so `cargo miri test` has not finished on any pull request since the test landed. The same change is in #47181; this one goes away when that lands.
|
The merge also showed that Generated by Claude Code |
The scan for places that call a function follows the register of an
`adrp` through the next four instructions, also after it was written to:
adrp x0, page
add x0, x0, 0x530
ldr x1, [x0]
The last line read as a use of `page + 0`. Where an import's slot lies
at such an offset of a page that also holds globals, each of these
counted as a call. In the released 1.4.4 for Linux arm64 that gives 47
calls of `pipe` and 18 of five other functions. The other 59,146 places
found there stay the same.
Which import is hit depends on the layout. On this branch it is `fork`,
so compile-trim-capabilities.test.ts failed on Linux arm64 only. That
test now also says which features keep a place, as its neighbour does.
|
CI status (I keep this comment up to date; last edit for cbf095e) Now: cbf095e takes the two files of #47329 as they are there, to get macOS arm64 green. Nothing else changes, and nothing is left of the commit once #47329 is on
CI on cbf095e: green. #135188 passed at 17:19 UTC: 188 of 188 jobs, on every platform, no failure of any kind, soft ones included. GitHub checks: all green. macOS arm64 got agents at the third attempt (macOS 26.4, 26.6.2 and 15.7.7), macOS x64 at the fourth.
The merge before it, 5ff31b2:
CI on 5ff31b2: one failure, the one above. #135058: all builds, all of Linux and Windows, and
CI on 1db0a6c, the first merge: no test failed. #134890: 180 of 180 jobs on Linux and Windows passed, and all 4 Before the merges: green on 73ffb98. #134483 passed: 308 jobs, on every platform, macOS included (arm64 and x64: all of Solved: It is a mistake of the test's scan on arm64, not a call. With e9aa197 the failure prints the words at the place: The scan added the
Not from this PR:
Generated by Claude Code |
compile-trim-capabilities.test.ts still finds one place that calls `fork` on Linux arm64 on this branch, at the same address, in bytes that go with no feature. So the scan's mistake that the previous commit fixed is real, but it is not what fails here, contrary to what that commit's message says. An address alone does not tell what it is. The failure now shows whether the place is code or data, the words around it, where the import's slot and stub are, and how many branches lead to it.
compile-trim-capabilities.test.ts reported a call of `fork` on Linux arm64
on this branch. The words the previous commit made it print are:
adrp x8, 0x8702000
str x0, [x8, #0xa68]
adrp x8, <another page>
add x8, x8, #0x160
The scan added the 0x160 to the first page, which gives 0x8702160, where
this build's slot for `fork` happens to lie. The second `adrp` had already
replaced the register, so the code has nothing to do with `fork`.
The scan now stops at an `adr`, `adrp`, `ldr` or `add` that writes to the
register, whichever register it reads. Only instructions that are certain
to write end it, so no real use is passed over: a store of the register,
or an `adrp` of another register, in between still leads to a report.
The 59,146 places it finds in the published Linux arm64 build of 1.4.4
stay the same.
Conflicts: - Cargo.toml: both `[profile.dev.package.*]` sections stay, this branch's for bun_libdeflate and main's for bun_sema_standalone. - src/sema/check/regexp_scanner.rs: main's version. It skips the test under Miri itself, so the skip this branch had taken over is no longer needed. - test/bundler/compile-trim-capabilities.test.ts: main's rewrite, with the failure of the `fork` assertion again saying what lies at the place.
One conflict, in test/bundler/what-an-executable-can-call.ts: #47326 fixed on main part of what this branch had fixed in the arm64 scan. - The file is main's, with five lines changed. main stops following the register of an `adrp` at an `ldr` or `add` that reads and overwrites it. The scan here also stops at a second `adrp` or `adr` into the register, which is the case this branch hit on Linux arm64, and at an `ldr` or `add` that overwrites it from another register. - `describePlace` is gone: main now reports the bytes around a place, so compile-trim-capabilities.test.ts is main's again.
`test/aot/build-scaling/build-scaling-2.test.ts` fails on macOS 26 arm64 since the merge of main: one shape grows 2.51 to 2.64 times in memory and 2.5 is allowed. It comes from the mimalloc upgrade in #47281, not from this branch: main's own build after #47281 failed it twice before it passed, and another branch fails it four times of four with the same numbers. These are the two files of #47329 as they are there, which records 2.58 for the shape, so the bound becomes 2.84. Nothing is left of this commit once #47329 is on main.
What does this PR do?
Removes the dependency on libdeflate.
src/libdeflate(bun_libdeflate) is a port of it to safe Rust.Draft. It builds and the existing tests of every consumer pass. Still to come in this PR: the performance work with numbers (see the end).
src/libdeflate(new,#[forbid(unsafe_code)]). It also denies indexing,unwrap,expect,panic!andunreachable!, because streams come from anywhere and release builds abort on a panic. It holds the compressor at levels 0 to 12 (DEFLATE, zlib, gzip) and the decompressor (DEFLATE, gzip).zlib_decompress.cis not ported: nothing in Bun gives libdeflate a zlib stream,fetchleaves those to zlib-ng on purpose.README.mdthere explains the decode tables, the bit buffer and how output memory is handled.Bun.gzipSyncof a large buffer resident at twice its size. So aVecis lengthened as the result grows: the decompressor asks for as much again as there is, the compressor for one block at a time, and the compressor's large tables areVecs filled as they are used.Bun.gunzipSyncandinflateSyncwithlibrary: "libdeflate"no longer start over from the beginning each time the output buffer has to grow.src/uws_sys/inflate.rs. That file holds the only newunsafein shipped code apart from one FFI call: it dereferences the pointers C++ passes and frees the object. The logic is a safe method.bun_zlib. For CRC-32 that is whatbun_tarandBun.hash.crc32already do.bun_zlib::adler32_bytesis new.opt-level = 3(debug assertions and overflow checks stay on). Unoptimized it is many times slower, and everybun installgoes through it.bun build --compile --aot --trimremoves the compressor from executables that do not use it.maindefined that feature (LibdeflateCompressorinsrc/trim/features/bun.def) by the names of the C functions. It now namesCompressor::new,compressandcompress_to_vecwith acxx-functionpattern, which matches demangled Rust names too, and the three are#[inline(never)]. The macros that mark Rust entry points expand tounsafe, which this crate forbids. Aminimummakes the link fail if the pattern stops matching.test/bundler/what-an-executable-can-call.tskept following the register of an arm64adrpafter another instruction had overwritten it. With this branch's layout that madecompile-trim-capabilities.test.tsreport a call offorkon Linux arm64 where there is none. Fix false alarms in the scan behindcompile-trim-capabilities.test.ts#47326 has since fixed this onmainfor anldroraddthat reads the register and overwrites it. What is left here is five lines on top of it: the scan also stops at a secondadrporadrinto the register, which is the case this branch hit, and at anldroraddthat overwrites it from another register.test/aot/build-scaling, as they are there. Since the mimalloc upgrade in Remove the address space ballast and the mimalloc patches, upgrade mimalloc to v3.6.0 and ICU to 79.1 #47281, one shape ofbuild-scaling-2.test.tsgrows more than the allowed 2.5 times in memory on macOS 26 arm64, onmainand on other branches too. Nothing is left of it here once test: one shape of build-scaling grows 2.58 times in memory #47329 is onmain.src/libdeflate_sys, its entries in the baseline CPU allowlists, anunsafe impl Zeroablein the HTTP thread, andprocess.versions.libdeflate. The licence stays inLICENSE.md, and its text is insrc/libdeflate/LICENSE.Changes I know of:
process.versions.libdeflateis goneBUN_FEATURE_FLAG_NO_LIBDEFLATE=1now also covers messages toBun.serveand request bodies thatfetchcompressesZlibContexthas a second 16 KiB buffermallocedinflationBufferfetchwith a customcompresslevel, or a body over the bound, no longer creates the cached level 6 compressorBun.Archivewith gzip: no memory for the compressor is an out of memory errorGzipInitFailed, as for a bad level, because the C wrapper could not tell the two apartbun audit: a refused compression would be an errorBun.gzipSyncanddeflateSynccopy shared inputs first, as beforeThe first commit only removes the library from the build system, and does not build.
How did you verify your code works?
New tests, which pin what the C library did. Every expected value was recorded from a build of the merge base, and the tests pass on that build too:
zlib.test.js, "libdeflate": the output for nineteen inputs at levels 0 to 12 (several blocks, stored blocks, runs, changing statistics, repeats 32767 to 32769 bytes back, few symbols, so many matches that the sequence stores fill up, so many that the match cache does, no match for over 65535 bytes, only literals at levels 10 to 12, blocks that end before the position reached, a last block shorter than the minimum, and three limits of the compressor from both sides), and for every length from 0 to 79.fetch-compress.test.ts.Bun.serveuses: a stream that only libdeflate accepts (length symbol 286) gets through up to 16384 bytes, not at 16385, and not at all underBUN_FEATURE_FLAG_NO_LIBDEFLATE=1. The same flag forfetchwithcompress.libdeflate_test.bind.ts, a binding that only builds with assertions have (debug, CI's ASAN lane): output of the exact size and one byte less, aVec's spare capacity, how far it grows, how far past the result it is written to, what an error leaves, how much input was read (gzip members too), and reuse of a compressor or decompressor, also after a call that had no room. Miri is not an option for these: creating 13 compressors takes it over 100 seconds, and the crate has nounsafefor it to check.Against the C library, in one process (outside the repo: libdeflate at Bun's pin and patch, linked next to the crate):
Vecreserved at exactly the right size is never reallocated (61,000 streams).Old binary against new binary (debug builds of the merge base and of this branch):
gzipSyncanddeflateSyncat levels 0 to 12 on eight kinds of input: identical bytes.fetchof gzip, zlib, raw deflate, two-member and corrupt bodies, below and above the 512 KiB shared buffer: identical.fetchwithcompress, both encodings at nine levels: identical bytes on the wire.Bun.serve, from 0 to 300,000 bytes.Bun.Archivewith gzip, andbun installof a tarball: identical trees.Existing tests, on a debug build on Linux x64:
zlib.test.js,zstd.test.ts,fetch-gzip,fetch-compress,fetch.stream,archive.test.ts,archive-golden,bun-audit,bun-install-streaming-extract,websocket-server.test.ts, and the four permessage-deflate suites.process.test.jsdepend on the host. The "multiple parts" group offetch.stream.test.tsneeds more than the default 5 s, without compression too, and passes on both builds with a longer timeout.--trim, in a trimmable release build of this branch merged withmain(local WebKit, without LTO, and with the step for ICU's data switched off, because the system's ICU is a shared library):--compile --aot --trimconsole.log("hello")Bun.gunzipSyncnew WebSocket(...)Bun.gzipSyncandBun.deflateSyncFeatureBelowMinimum: 0.00 MB can be removed with LibdeflateCompressor.libdeflate's history. There is no CVE or advisory for libdeflate (NVD, OSV, Debian's tracker, the repository's advisories), and nothing upstream is newer than Bun's pin. I read the subjects of all 796 commits,
NEWS.mdand all 174 issues. About 50 commits change the behaviour of code that is ported.test_incomplete_codes,test_invalid_streams,test_litrunlen_overflow,test_overread,test_trailing_bytes,test_slow_decompression) pass unchanged against the port, linked through a C shim.zlib.test.js. Those that nothing in this repo noticed at first got a test.3f21ec965da376446e226,480e031d5e5933f2f0df7noMatches(new)f52567fea2c695skewed(new)4f7d134coinsd352945coinsandshortMatches4d81654b8f2e787e19d17,5f4da4b0b6d0f419e168857cab07test_slow_decompressionnotices7380364(junk code lengths priced) fails 10 tests,5a9d25a(gzip footer taken from the end of the input) 2,55f9f70(all statistics cleared after going back) changes the output for inputsmixedandshortMatches(seen outside Bun: that edit leaves dead code, which Bun's build refuses), andbfa6ddb(shortest match worked out from the wrong position after going back) failed none, so inputalternatingis new.4d81654bug could be put back and no test here noticed. Each block starts from fresh nodes again.zlib_decompress, the custom allocator and thegzipprogram, and fixes for platforms, ABIs and build systems.Fuzzing, in a separate session against this branch, everything replayed on
2955608a(since then the crate only got three#[inline(never)]): 446 million coverage-guided executions (360 million of them on the code since2955608a), about 1.3 billion directed decompressions and 50 million directed compressions against the C library, 7 million calls with an input that other threads rewrite, inputs over 4 GiB, an allocation failure injected at every allocation in turn, and the C library with its own assertions on. No difference, panic or hang. Exhaustive runs, also without a difference: every DEFLATE stream of 0 to 4 bytes; every input of 0 to 20 symbols over 2 letters (shorter over 3 and 4), alone and after six prefixes, at all 13 levels (1.09 billion compressions into a slice of the bound, of the exact size and of one byte less; 518 million more on slightly shorter inputs that also go throughcompress_to_vec; 821 million at every output size from 0 to nine bytes over the bound); every 5-byte stream that starts a dynamic block (275 billion calls, but none of them decodes, so this shows that no such header panics or hangs, not that both accept the same codes); and for that, every assignment of code lengths to chosen sets of symbols, each followed by streams that are valid under it (35 billion calls, 122 million of which decode: up to 9 symbols of the code for the code lengths, and 8 symbols of the literal/length and offset codes with lengths up to 15 bits, subtables included; four bugs planted in the checks of the code lengths were each caught first; codes with many different long lengths at once are not enumerated). Every 5-byte stream that starts a static block: 1.1 trillion calls, 126 billion of which decode. Stored and reserved block types are not enumerated at 5 bytes. After change (4) below had slipped through, the kind of input that shows it was run on the crate itself, at all 13 levels, byte for byte against C: 29.9 million windows of 5,001 to 700,000 bytes over ordinary files and 780,000 compressions of generated texts whose vocabulary drifts. No difference. The decoder also got streams that zlib wrote, since all valid streams until then came from libdeflate or from generators: 60,000 streams at every level, strategy,memLevel, window size and kind of flush, each whole, with bytes after it, cut short and with single bits flipped (1.36 million variants, 657,000 of which decode). No difference. Mutation testing of the whole crate, 4,096 mutants (single-token edits and deleted lines): 229 do not compile, 3,552 are killed, and 315 give equal results, meaning the same return value and bytes for an input that stands still; time and memory may differ. None is known to be alive. An earlier version of this text said 3,548 and 319, all equivalent. Four of those 319 were real changes, found while writing down a reason for each. The crate is right in all four places; the checks were too weak. (1) Without theout.truncate(start)in the failure branch ofcompress_to_vec, theVeckeeps partial bytes after an error. The branch only runs when another thread rewrites the input during the call. (2, 3)bitcount >= 8weakened, in two spellings, in the byte-wise flush near the end of the room: it trips the crate's owndebug_assert, only with such an input and only with debug assertions on. (4)num_new_observations * 200 / 512changed to/ 511in the decision to end a block. It was taken for equivalent because a decision was thought to come at exactly 512 new observations, but at levels 5 to 9 it also comes at 513 to 516, and at 514 the quotient differs. Nothing had caught it: in a build of Bun with that change all 140 "libdeflate" tests pass, while 30,000 bytes ofbun.d.tscompress to 8,565 bytes at level 6 instead of C's 8,580. All 319 have since been run with inputs that are rewritten during the call, those outside the decoder over about 9,300 windows of ordinary files and 2,400 generated texts, and those of the decoder and of the handling of room through the enumerations above. Each of the 315 has a written reason, by one reader. Three that had survived a thinned set of checks were real changes (the limits of levels 11 and 12, and the 5000 bytes that must be left for a block to end). Nothing in this repo noticed them either, sozlib.test.jsnow pins those limits from both sides: with a limit moved by one either way, in a build of Bun, exactly the input meant for it fails. Eighteen other off-by-one changes to constants of the heuristics are caught by the fuzzing suites only, not by this repo's tests: they need inputs built for an exact boundary. With (4) that makes nineteen that this repo does not pin. (1) to (3) need an input that changes during the call, so no deterministic test can pin them. Ten mutants needed an exact numeric tie in the block-splitting and cost heuristics, so inputs that land exactly on those ties were constructed (about 10,900 compressions, no difference from C). It also read every caller against the merge base for changes that the table above does not list, and found the two aboutBun.Archiveandbun audit.Performance so far
Speed relative to the C library, 1.0 being the same. Both are in one process, built with the flags of a release build (
-march=nehalem, frame pointers kept), on a Cascade Lake Xeon. The C library picks its BMI2 decompressor at run time, which is worth 10% to it. Safe Rust compiled for Nehalem cannot.On this family of CPUs (Skylake to Cascade Lake) a jump that crosses a 32-byte boundary is costly, and the bounds checks give the Rust loops more jumps than C's have. The same decoder ran at 0.74 to 0.93 depending on nothing but its address. So there are two columns: "as built" is this build on this CPU, and "padded" is built so that no jump crosses such a boundary, which is about what other CPUs see. The decoder's loop is aligned to 64 bytes (
#[feature(fn_align)]) so that other code cannot move it. Where its jumps fall has to be looked at again in a release build of Bun.src/libdeflate/README.mdhas the details.Still to do here
Generated by Claude Code