Can we use compression on Parquet writing? #6061
-
Is there a |
Beta Was this translation helpful? Give feedback.
Answered by
tustvold
Apr 28, 2023
Replies: 1 comment
-
Parquet natively supports block level compression - see https://github.com/apache/parquet-format/blob/master/Compression.md Compressing the file as a whole would lose the ability to do meaningful IO pushdown, and would likely yield minimal size savings, so I'm not aware of anyone doing this |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
metesynnada
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Parquet natively supports block level compression - see https://github.com/apache/parquet-format/blob/master/Compression.md
Compressing the file as a whole would lose the ability to do meaningful IO pushdown, and would likely yield minimal size savings, so I'm not aware of anyone doing this