Remove redundant BufReaders for decompressing tar archives#6317
Remove redundant BufReaders for decompressing tar archives#6317kskalski merged 3 commits intoanza-xyz:masterfrom
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6317 +/- ##
=========================================
- Coverage 82.8% 82.8% -0.1%
=========================================
Files 847 847
Lines 379488 379486 -2
=========================================
- Hits 314349 314309 -40
- Misses 65139 65177 +38 🚀 New features to boost your workflow:
|
Thanks for the link to where zstd creates its own BufReader. Can you also provide links for gzip and lz4 too, since this PR is changing them as well. |
|
Sure, I'm also switching bzip2 decoder as it has appropriate wrapper. Updated description. |
brooksprumo
left a comment
There was a problem hiding this comment.
Looks good to me. I think you'll need to rebase to pull in newer commits that fix the CI.
f9eacaf to
57a86cc
Compare
57a86cc to
71e2436
Compare
|
Let's also backport this to v2.3. We'll need to wait for #6430 to land, and then please add the "v2.3" label to kickoff the backport. |
|
Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis. |
* perf: remove redundant BufReaders for decompressing tar archives * Format * Switch bzip2 decoder to read (cherry picked from commit da1b789)
* perf: remove redundant BufReaders for decompressing tar archives * Format * Switch bzip2 decoder to read (cherry picked from commit da1b789)
…6317) * perf: remove redundant BufReaders for decompressing tar archives * Format * Switch bzip2 decoder to read
…6317) * perf: remove redundant BufReaders for decompressing tar archives * Format * Switch bzip2 decoder to read
Problem
ZSTD
Decoder::newcreates buf reader with tuned size already, other decoders handle buffering too:BzDecoderhttps://docs.rs/bzip2/latest/src/bzip2/read.rs.html#87-91GzDecoderhttps://docs.rs/flate2/latest/src/flate2/gz/read.rs.html#143-147lz4::Decoderhttps://docs.rs/lz4/1.28.1/src/lz4/decoder.rs.html#33-43 (it uses a vec buffer directly)zstd::Decoderhttps://github.com/gyscos/zstd-rs/blob/229054099aa73f7e861762f687d7e07cac1d9b3b/src/stream/read/mod.rs#L29In those cases wrapping reader with
BufReaderis counter-productive or unnecessary (bzip2 provides bothbufreadandreaddecoders)Summary of Changes
Remove redundant BufReaders for decompressing tar archives when underlying decompressor is already doing buffering.