Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compression level zero is very slow #156

Open
fintelia opened this issue Sep 21, 2024 · 1 comment
Open

Compression level zero is very slow #156

fintelia opened this issue Sep 21, 2024 · 1 comment
Assignees

Comments

@fintelia
Copy link

When set to not apply compression, writing compressed output should mostly just amount to memcpy'ing the input data to the output. However, actual performance seem to be on the order of 120 MB/s. Some quick profiling indicates that a substantial amount of time is spent updating hash chains and recording literals. Neither of these operations should be necessary to produce uncompressed output.

Benchmarking via flate2:

let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(0));
encoder.write_all(&bytes).unwrap();
encoder.flush_finish().unwrap();
@oyvindln
Copy link
Collaborator

I'm not done fixing it yet as it's still an order of magnitude slower than it should be, but I've started doing some work on it to split the stored compression mode to it's own function to skip the non-needed parts, and it's it seems to be something like twice as fast in the version I just published (0.8.3) compared to the previous versions.

It looks like the orig miniz didn't really care about making the stored mode performant and it's just a quick addon in the main compression function and thus does a lot of redundant stuff like you noted. I.e as ending up adding stuff to the lz dictionary even though that's pointless, and adding literals to the hash chain. Zlib just stores how many bytes that would be needed to be added which allows it to just defer it to the the non-stored compression function to add stuff from the backbuffer to the hash chain to be able to compress better if the compression mode is switched mid stream. (no idea in what situation one would change compression mode mid stream but it is a thing that is supported for whatever reason).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants