Question on best practice for stream-based decompression

[dentiny]

This issue is to ask question about the best practice for stream-based decompression.

Context:
On compression side, I use zstd ZSTD_compressCCtx block-based compression.
On decompression side, I read content from cloud in chunks with zero-copy file handle: we could get the pointer for content in each chunk, rather than memcopy them into a large chunk then decompress.

I want to know if zstd currently supports stream-based decompression, which could better utilize the zero-copy feature?
The intention is to avoid copying the content of these chunks into a big block then decompress.

I tried to use buffer-less decompression API as the example code, but it doesn't show difference with copy content + decompress.

I also tried the stable output buffer option checked in this PR, still cannot see much difference, so I'm suspect I am using it in the wrong way.

My benchmark reads compressed content from cloud, and use block-based or stream-based decompress to get the final content. The un-compressed size is seperately 10MB, 50MB, and 500MB.
What confuses me is it seems "concatenating chunks together then decompress" and "decompress them in chunks for multiple" times doesn't show much difference, sometimes even slower.

I would be really grateful if zstd experts like you could give me some instructions on this issue. Thank you!

[Cyan4973]

I think this question was already asked in another place,
but for improved visibility let's provide an equivalent answer here.

In short : I recommend to not try decompression "in-place", and just rely on the generic (and safer) ZSTD_decompressStream().

ZSTD_decompressStream will proceed with copying and combining block parts whenever necessary, and will be able to read compressed blocks in-place whenever possible. In short, it already achieves the minimum optimization expected in such a scenario.
I wouldn't expect large benefits from these optimizations though, memcpy() is typically a very fast operation, so it's unlikely to be a limiting factor in total processing pipeline.

You may also use advanced (experimental) parameters ZSTD_c_stableInBuffer and ZSTD_c_stableOutBuffer if you believe they are applicable to your use case. They will limit the amount of memcpy() operations whenever possible, though note that it comes with strong requirements on memory layout and stability.
Here also, even though this will technically limit the amount of memcpy() operations, I wouldn't generally expect large gains because it is just saving a fraction of total work to do.

Finally, your question seems focused on the ingestion of compressed data by zstd, but in general, this stage is very fast, and speed and memory limitations are more likely to be present on the output side of decompression.

[yoniko]

@dentiny - I hope the answer helps you. Closing for now, please reopen if something's not clear.

[dentiny]

@yoniko yeah it's helpful, thanks for the answer!