Repository navigation
Replace the zstd C library with a port to safe Rust #47079
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 189 commits
Commits
Show all changes
227 commits
Select commit
Hold shift + click to select a range
564c4ff
Remove the zstd C library from the build
claude 8c197b5
zstd: add the decoder in safe Rust
claude 93bb106
zstd: add the compressor in safe Rust (work in progress)
claude 360684e
zstd: remove what nothing uses, and what clippy finds
claude 09ce132
zstd: the match finders and the entropy coders as verified against C
claude 43aa6d8
Bun.zstdCompress: a failed allocation is an out of memory error
claude 0f99288
zstd: optimize the crate in debug builds
claude 519fe26
zstd: test that the output is that of the C library
claude caaa573
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude 41d3dad
zstd: remove the scratch files, and keep one copy of a dictionary
claude 45523f4
zstd: zero output once, feed workers all of a write, drop what nothin…
claude de3e36b
zstd: the harness that compares the port with the C library
claude 42b3576
zstd: take the harness out again
claude f020d88
zstd: decode sequences as fast as the C library
claude a20882e
zstd: faster match finders and sequence encoder, and what mordant found
claude 11a8d5b
zstd: shorter prologues in the generic match finders, contexts are Sync
claude 9c7f6e0
zstd: a job with workers makes room for the long distance matches of …
claude 078890f
zstd: price the offset of a match once, not for each of its lengths
claude b37c7d4
zstd: a worker does not clear its tables for a job larger than those …
claude 20bd848
zstd: find where a match ends 16 bytes at a time, and where it starts…
claude 7bbf2a0
zstd: fast reads the input once per turn and indexes both positions o…
claude 9ca1713
zstd: fast keeps what it needs once per match out of the way of the s…
claude 2fd7bd8
zstd: the codes of a sequence come with its extra bits and are counte…
claude d43d2f4
zstd: dfast keeps what it needs once per match out of the way of the …
claude a9409b1
zstd: the tables of the predefined distributions are not made again f…
claude 7d42e1b
zstd: 8 equal bytes before a match take no detour if no more can be t…
claude c67b15a
zstd: a new context makes and clears less, and tables that cannot be …
claude 6c8cf31
zstd: the buffers of a compression stream are not cleared
claude 84d8c68
zstd: one way to reset a table, whatever it is a table of
claude 9204768
zstd: say that it is a port, where it was said to be a C library
claude 7259097
zstd: workers touch no more memory than they use
claude 23e4312
node:zlib: take the ZSTD_* constants from the crate
claude 553042b
zstd: test frames that no encoder makes, damaged frames, and abandone…
claude 8038637
zstd: fast with two segments goes about it as with one
claude 671862e
[autofix.ci] apply automated fixes
autofix-ci[bot] 4a978bc
zstd: dfast with two segments reads the input once per position
claude a393a00
zstd: fast with a dictionary's match state reads the input once per p…
claude f3eb4d3
zstd: dfast with a dictionary's match state reads the input once per …
claude 9104943
zstd: workers share one queue of jobs
claude a5239f0
zstd: a context keeps no books of the workspace that the C code would…
claude df51c0f
zstd: a Huffman table is made from the symbols that occur, and its tr…
claude 74929d0
zstd: less between the end of a match and the next look at the table,…
claude 7a59b4e
zstd: less between the end of a match and the next look at the tables…
claude 129ede9
zstd: how many bits a sequence takes is read from the bits themselves
claude ef448e6
zstd: fast copies up to 32 literals without looking at their number
claude 226e9d6
zstd: speed up the binary tree match finders and the optimal parser
claude aae1516
zstd: a table that gets shorter and longer again is not cleared as lo…
claude 75f0c60
zstd: the buffer of literals is cleared only as far as blocks are long
claude 08619eb
zstd: fast and dfast know the sizes of the tables of the default para…
claude 7bf8f5b
zstd: dfast only asks whether an index was too low once the bytes are…
claude f5e9ea1
zstd: likewise with two segments and with a dictionary's match state
claude ef0f733
zstd: room for sequences and literals is made for the blocks that com…
claude 5a0d69b
zstd: the table sizes of levels 1 and 3 for small inputs are known as…
claude 0890796
zstd: faster greedy, lazy and lazy2
claude 9f39636
zstd: fast only asks whether an index was too low once the bytes are …
claude 045f575
zstd: nor does dfast for the long match after a short one
claude fe2ce75
zstd: cheaper searches in the tree of a dictionary, and long runs of …
claude 7d8106d
Merge branch 'claude/zstd-rust-replacement-4qwmbq' into perf/tree
claude 9e7dc50
Merge branch 'claude/zstd-rust-replacement-4qwmbq' into perf/entropy
claude b834806
zstd: Table::reset tells whether the memory is new
claude 173f5c1
Merge branch 'claude/zstd-rust-replacement-4qwmbq' into perf/entropy
claude 16f007a
zstd: sequences are decoded by the batch, then executed
claude 3d05cac
zstd: a Huffman lookup yields up to 3 literals, and streams that end …
claude 68e6f8e
zstd: a streaming decoder touches no more of its buffers than it uses
claude 6196cf2
zstd: a skippable frame that goes on past the input has an extent too
claude da99d07
zstd: far matches are prefetched while a batch of sequences is decoded
claude edc24da
zstd: the decoder adapts to long matches, close matches, lengths with…
claude 1bcb9f7
zstd: four Huffman streams are decoded at a time until the last one i…
claude a8e5dae
zstd: the ends of streams are decoded from padded copies, and tables …
claude 0b8022c
zstd: matches in a dictionary are of the common kind, and a thread's …
claude f4b98cd
zstd: the weights of a Huffman table, a lone stream of literals and t…
claude bf490c6
zstd: a frame header is read into place, and the counts of a table wi…
claude e62cc5c
zstd: no shift by more than there are bits where a lookup is longer t…
claude 4bd2cf7
zstd: the loops of the decoder that come in several kinds are inlined…
claude eb486de
zstd: matches in what a streaming decoder's buffer holds from before …
claude 5f7d028
zstd: the tables of a few sequences are used where they are
claude 964de3e
zstd: what the tables of sequences tell about them is theirs to tell
claude 675a92a
zstd: the size of a decoder counts all that it allocates to decode bl…
claude 63cfe2c
zstd: merge the main line; what fast and dfast look their tables up w…
claude d99e43e
zstd: btlazy2 sets up its searches once per block
claude 2a7f4b4
Merge the faster fast and dfast match finders
claude bbfa74a
Merge branch 'claude/zstd-rust-replacement-4qwmbq' into perf/entropy
claude 742599d
Merge the faster binary tree match finders and optimal parser
claude 09f9383
zstd: the next sequence waits less for the states that it is decoded …
claude 5c967a6
zstd: a name for additional bits that is not that of another parameter
claude ff8bd0e
zstd: a hash rate above the window does not make for a table of 8 GiB
claude 34ccfab
zstd: a small output is decoded where there is room to copy in chunks…
claude 510435d
zstd: the loop over four Huffman streams knows how many bits it looks up
claude a91dd57
zstd: the older segment on its own goes by that name
claude dbf8fea
zstd: a block of a few literals is left as it is without making table…
claude 3b1ddb3
Merge branch 'claude/zstd-rust-replacement-4qwmbq' into perf/entropy
claude 460ed0b
zstd: zeros follow the literals of a block, and a thread's next decod…
claude 8ff6f3f
DecompressionStream: empty chunks in the middle of a zstd frame are n…
claude 3fe9b40
zstd: with workers, input that is not of the pledged size is an error
claude 745d76f
zstd: a streaming decoder starts each frame with empty buffers, and c…
claude 55d4fad
zstd: merge the main line
claude 7d3ca60
zstd: a small output is decoded elsewhere only if nothing before it i…
claude 27cd088
zstd: a compressed block that makes more than the largest block is an…
claude deafcc0
zstd: a long match does not end a run of common sequences
claude 362ddb1
Merge branch 'claude/zstd-rust-replacement-4qwmbq' into perf/entropy
claude 8041458
zstd: a streaming decoder refuses offsets beyond the window once its …
claude ddf9cc3
zstd: fewer copies and fewer batches for a small block
claude 212f9ba
zstd: merge the main line
claude 7461d8a
Merge the faster entropy stage and context setup
claude 9e0d395
zstd: workers reset their tables and make their contexts like anyone …
claude bc18f77
zstd: with matches far back, what few sequences a block has are known…
claude b5f24e0
zstd: without rare symbols, the states of a table find their symbols …
claude 916f38a
node:zlib: a zstd dictionary that cannot be loaded is an error
claude b8025bf
zstd: the total of the Huffman weights comes from how many there are …
claude dcbcaf1
zstd: long matches far back are still copied while the next sequences…
claude 977ef37
zstd: fast and dfast have one version for all parameters but the defa…
claude 2fa2eb4
zstd: dfast only asks ahead for what is in large tables
claude 2f442d1
zstd: test for the bugs that zstd has had
claude bee54ea
zstd: fast checks once for what it reads around the end of a match, a…
claude 8e241a5
zstd: so does dfast
claude eb2aea2
zstd: with a dictionary's match state, how many bytes are hashed is k…
claude fd951f4
zstd: nobody waits for workers to leave
claude e4de7df
zstd: test for the code of an error
claude 6625957
Merge the second round on the decoder
claude 8e12289
zstd: fast with a dictionary's match state checks once around the end…
claude 3c5a477
zstd: and with two segments
claude 06433fd
zstd: a thread's next decoder takes over no output either
claude 0ffd40a
zstd: fast and dfast index a dictionary by 10 bytes at a time, and wi…
claude 2717485
zstd: nothing to index is no error
claude 3c893fc
zstd: the lazy parsers weigh a repeat offset only where it gives a match
claude 573e4c0
zstd: fetch all of a row of 64 ahead of its search
claude 0db59a6
zstd: faster hash chains, most of all for short inputs
claude c2216a5
zstd: a match right after another one is not looked at for what is be…
claude 0e01a37
zstd: say what the versions of fast and dfast are for
claude 3d4a5e7
zstd: test for more of the bugs that zstd has had
claude 034f39d
zlib: the sizes of zstd contexts as they are now
claude e8013cc
zstd: merge the main line
claude 8f7a87e
verify-baseline: pass a per-test timeout to the emulated bun test run
claude 31103d4
Merge the second round on the fast and double fast match finders
claude 813eb78
zstd: workers that could not be started are started for the next frame
claude ec24e1d
zstd: indices start again where the C code replaces its workspace, an…
claude 6579103
zstd: a unit test for the size of the table of long distance matches
claude 7e0e64b
zstd: streams in tests report their errors
claude 0c8383f
zstd: fast and dfast with an attached dictionary branch on the tag al…
claude 7dc6d29
zstd: streams in tests report that they were closed, too
claude a9fef23
zstd: an output that is allocated for its frames has room to copy in …
claude aaa1236
zstd: a branch on additional bits is worth it up to one length in seven
claude 15a9f11
zstd: the copy of the start of a stream is made where it stays
claude fad8f8f
zstd: a tiny output is allocated without room after it
claude b721dc5
Merge branch 'main' into claude/zstd-rust-replacement-4qwmbq
claude dc8cf4b
node:zlib: zstd no longer hands back C strings of its own
claude ea26005
Tests no longer name zstd as a C dependency
claude 55c98c0
zstd: a growing output is zeroed at most 64 steps ahead
claude 708e9b4
zstd tests: level -2 instead of -1, which node:zlib takes for unset
claude 61bc26a
zstd: btlazy2 searches a short segment in the loop of the parser, wit…
claude c1cb938
zstd: btlazy2 compares candidates by their first 16 bytes, without wa…
claude 9863320
zstd: the comparison of 16 bytes is done on vectors with overflow che…
claude 8990d63
zstd: btlazy2 does not go down a tree that has nothing
claude 65d2eea
zstd: the errors are in error.rs, as Error and Result
claude 0b92c0c
zstd: workers give up a frame within a block or 4096 positions of a p…
claude 60b35fe
zstd: tests for four fixes to the optimal parser that nothing noticed…
claude 0a40fdc
zstd: rows of 64 entries stay a comparison of vectors in a release build
claude 8e0cb85
zstd: remove two parameters that follow from a third
claude 901a9d9
zstd: split three tests that took over 3 s each in a debug build
claude 1c78c76
zstd: fast and dfast ask whether there is a repeat offset only once i…
claude 1df8f8c
zstd: the length of a match is a comparison of vectors, the first one…
claude 7245ee8
zstd: the lookups between two reloads of four Huffman streams are wri…
claude e5c4c3c
zstd: few literals of long codes are decoded one to a lookup
claude efbf33d
zstd: the room for a frame is zeroed block by block
claude 7f195c7
zstd: the counts of a table are read with less to wait for
claude eacaef1
zstd: long copies are split where glibc's are slow
claude 6b2def7
zstd: what one call decodes is not zeroed before it is written
claude 5a031e0
ci: skip the bun_sema spelling test under Miri
claude e4d6720
zstd: decode long Huffman codes one symbol to a lookup
claude a77f135
zstd: a buffer whose content is of no use is not copied when it grows
claude a87cc41
zstd: less for the optimal parser to do where there is no match
claude 11efee8
zstd: btlazy2 hands shorter runs of positions to the tight insertion …
claude 7ddcdb5
zstd: the block splitter reports an allocation that fails
claude 0cfb03b
zstd: a block asks for room once its sequences are known
claude 40b288f
zstd: a unit test does not depend on where its buffers are
claude 4a881e2
zstd: long matches are counted 64 bytes at a time
claude 29f26b7
zstd: the optimal parsers go over what has no match in one loop
claude 72f95e5
zstd: long copies to the end of a large new buffer are made in parts
claude d574f50
zstd: short matches from less than 16 bytes back have a loop of their…
claude f629239
zstd: a stream that has failed stays failed, and is not mixed with wh…
claude a88148b
zstd: what starts a new job resets a stream, and a compressor that ha…
claude d106dc8
zstd: what `exec_batch` does before its loop is a function of its own
claude e48dc2d
zstd: matches in a dictionary are of the common kind wherever the out…
claude 39b1e9f
zstd: sequences are executed in batches also while literals would be …
claude 3403829
Merge branch 'main' into claude/zstd-rust-replacement-4qwmbq
claude 68fca8c
build: the C decoder of zstd stays for the `jsc` shell that reads a p…
claude f677884
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude c788848
zstd: lazy and lazy2 share their code
claude 9b6d330
zstd: fast and dfast are compiled for the sizes of tables only
claude 33a1c7b
zstd: literals that are stored or one byte are not written to a buffe…
claude a7fb6eb
zstd: a round trip that fails is not taken for a parameter that is ac…
claude 51386a6
build: --local-deps=zstd is taken again where zstd is built
claude dd89d78
zstd: the contexts of workers are in a pool, as in the C code
claude 79995be
test: one shape of build-scaling grows 2.58 times in memory
claude 41fad94
zstd: records that differ in a field no longer have every candidate c…
claude 605aa6e
zstd: a single Huffman stream of short codes is no longer left to the…
claude 07e8c4e
zstd: the second tags of the rows are given up within a block where t…
claude 9778f06
zstd: btlazy2 gives no hints in the first block and adds skipped posi…
claude b06f15d
zstd: the buffer of the literals is cleared as they come, not all at …
claude dd2ac32
zstd: which of two positions is the smaller one is told from 16 bytes…
claude 8f8c6f4
zstd: the optimal parser gives fewer hints while it adds the position…
claude d38578f
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude d0d121d
zstd: a match in the older segment of the window that ends within 16 …
claude 615100a
zstd: a unit test for where the current segment of the window starts
claude 8aec2cd
zstd: rows with a dictionary that is searched in place: a tighter loo…
claude 43bf90a
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude f8cc91e
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude 8d25bbb
The zstd side of the ICU hook is compiled for FreeBSD too
claude 21acd17
zstd: the contexts that are kept are not in place
claude 306afc8
zstd: the optimal parser passes what has no match faster with a dicti…
claude eb1ad32
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude 285d751
test: whether zstd's sources are fetched does not depend on the cache…
claude 666b766
zstd: the optimal parser's loop for what has no match goes on while a…
claude 3e02569
zstd: an enum for where the repeat offsets lead, in place of a second…
claude 3efec57
verify-baseline: the timeout of main for the emulated run
claude 55c9643
zstd: the same loop while a repeat offset leads into an older segment…
claude de7ed9c
zstd: positions that are added to the binary trees are compared 8 byt…
claude f9f4dec
zstd: a copy of get_all_matches for a dictionary in place and one for…
claude 04d5d27
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude 209068d
zstd: a position that is added to the binary trees does not wait for …
claude fc13be2
zstd: sequences that are taken one at a time are decoded in registers…
claude 1e78952
zstd: a worker stops while it indexes its prefix in rows, and a dicti…
claude 98c81e3
test: compile-trim-capabilities scans only the Mach-O sections that h…
claude 46048ba
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude 2510945
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude f3de4e9
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude ea47620
Merge remote-tracking branch 'origin/main' into claude/zstd-rust-repl…
claude File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.