Support decompression of compressed blocks of size ZSTD_BLOCKSIZE_MAX #3399
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The wording of the Zstandard specification
allows valid compressed blocks to be sized exactly
ZSTD_BLOCKSIZE_MAX
bytes (128 KB).This generally does not happen, since an uncompressed block would have same size but suffer no decompression speed cost.
Starting with this patch, the
libzstd
decoder will now accept decoding a compressed block of sizeZSTD_BLOCKSIZE_MAX
exactly.Note though that decoders from reference
libzstd
before <v1.5.4
consider this edge case as an error.As a consequence, avoid generating compressed blocks of size
ZSTD_BLOCKSIZE_MAX
for broader compatibility with the ecosystem of deployed zstd decoders.Note that the reference
libzstd
encoder never produces compressed blocks of sizeZSTD_BLOCKSIZE_MAX
, and will continue to do so for the foreseeable future, both to preserve compatibility with older decoders, and because it doesn't make sense to produce compressed blocks of such size (as opposed to uncompressed blocks).The impact of this modification to the ecosystem is expected to be none. This is mostly meant to respect the letter of the RFC8878 specification.
Thanks to @SmitaKumar for reporting this issue in #2577