-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ability to decode streams compressed with "zstd long=31" #390
Comments
FYI -- I monkey patched the const located in framedec.go and was able to stream in a zstd stream encoded with long=31 without any issue. My recommendation would be to increase MaxWindowSize to the actual maximum allowed by zstandard itself (when using the long flag, this is 2 << 31). When trying to decode a compressed file with zstd, it will generally complain when the file was encoded with a window size of 2 << 31 and request that you provide the --long=31 flag for decompression. This generally requires two gigs of memory, but I don't see how increasing the MaxWindowSize would cause any incompatibility with existing scripts that use this module.
|
Linking to pull request to change MaxWindowSize constant: #392 |
I made the maximum configurable and clarified the documentation. See #394 |
Thank you! That's great. Also, is 2 << 31 4GB for signed int? The warning zstd gives looks like:
I see what you mean, though. Either way, that appears to be the maximum that can be used for zstd, but I really like the idea of making the window configurable. Awesome project by the way -- really impressive work! |
@pushshift Thanks for the contribution and the kind words. I have merged the PR. |
I use very large window sizes when compressing data (up to the maximum of 2 << 31). I noticed that the package has a MaxWindowSize of 2 << 29.
How would I go about decoding streams compressed with long=31? Was support for larger window sizes purposely left out?
Thanks!
The text was updated successfully, but these errors were encountered: