Skip to content

Conversation

@Ted-Jiang
Copy link
Member

@Ted-Jiang Ted-Jiang commented Aug 11, 2021

Make sure you have checked all steps below.

Jira

Tests

  • [] no need UT

Commits

  • My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain Javadoc that explain what it does

@Ted-Jiang Ted-Jiang closed this Aug 11, 2021
@Ted-Jiang Ted-Jiang reopened this Aug 11, 2021
Copy link
Contributor

@gszadovszky gszadovszky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

usedMem == 0 ?
props.getMaxRowCountForPageSizeCheck()
: (long) rows / usedMem * remainingMem;
: rows * remainingMem / usedMem;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we worry about overflow by (rows * remainingMem)?

Copy link
Contributor

@gszadovszky gszadovszky Aug 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've felt it as a low risk. We are talking about pages here so neither the row count nor the used memory should be too high that this would overflow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as gszadovszky +1

@gszadovszky
Copy link
Contributor

@shangxinli, I'm merging this. Let's see how it is working.

@gszadovszky gszadovszky merged commit 5154d04 into apache:master Aug 16, 2021
shangxinli pushed a commit to shangxinli/parquet-mr that referenced this pull request Sep 9, 2021
sunchao pushed a commit to sunchao/parquet-mr that referenced this pull request Mar 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants