Skip to content

Conversation

terrelln
Copy link
Contributor

This flag allows users to skip the output buffer allocation + memcpy if they guarantee that the ZSTD_outBuffer* passed to ZSTD_decompressStream() is large enough to fit the entire frame, and is never modified by the user.

Also fixes a bug where empty frames weren't eligible for single-pass mode. It caused a small allocation (4 bytes), and is a little bit slower.

Adds tests for single-pass and stable output buffer modes.

Fuzz tests ZSTD_d_stableOutput to ensure it never crashes.

Fixes #2093

@Cyan4973
Copy link
Contributor

Looks good to me.
It's pretty well done, respecting project's coding style and responsibility layers, complete with a test set.

I'm just curious how many users would be interested by this new capability ?
Does it go beyond #2093 in your opinion ?

@terrelln
Copy link
Contributor Author

I'm just curious how many users would be interested by this new capability ?
Does it go beyond #2093 in your opinion ?

Yes, I expect this to be a reasonably common scenario, as far as streaming goes. We could use this in Managed Compression and folly::io::Codec. They can both take folly::IOBuf as an input, which is a chain of input buffers, and both generally write to a single output buffer.

Most of the time the IOBuf's aren't chained, but when they are we either have to copy the input into a single buffer (Managed Compression does this), or use streaming (folly does this). Both would be served by this API. Managed Compression would save an allocation, and might save copies if there are blocks that are entirely contained in one buffer. folly would save the allocation of the output buffer, and the extra memcpy.

@terrelln
Copy link
Contributor Author

Fixed a memory leak in zstreamtest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minimizing memory requirements for Decompression?

3 participants