bump parquet converter marker version for parquet file with hash column #7154
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…h column Signed-off-by: yeya24 <[email protected]>
SungJin1212
approved these changes
Dec 11, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
This PR bumps the current parquet marker file version from 1 to 2 to differentiate parquet files with and without hash column. Hash column is important as it is required to enable projection pushdown.
By having different versions for parquet files with and without the hash column, querier can make the decision if we need to enable projection or not. Only if all blocks to query have the hash column, we can do projection.
In the converter, now as long as the parquet file marker has a valid version (1 or 2) for now, conversion for the block is skipped, to avoid unnecessary conversion. If users want to re-convert their existing parquet files they can delete the existing markers and convert again.
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]