-
Notifications
You must be signed in to change notification settings - Fork 1.5k
[PARQUET-2073] Fix estimate remaining row count in ColumnWriteStoreBase. #922
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
gszadovszky
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing this!
| usedMem == 0 ? | ||
| props.getMaxRowCountForPageSizeCheck() | ||
| : (long) rows / usedMem * remainingMem; | ||
| : rows * remainingMem / usedMem; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we worry about overflow by (rows * remainingMem)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've felt it as a low risk. We are talking about pages here so neither the row count nor the used memory should be too high that this would overflow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as gszadovszky +1
|
@shangxinli, I'm merging this. Let's see how it is working. |
…e. (apache#922) (cherry picked from commit 6e72dd4)
Make sure you have checked all steps below.
Jira
https://issues.apache.org/jira/browse/PARQUET-2073
Tests
Commits
Documentation