HIVE-28268: Iceberg: Retrieve row count from iceberg SnapshotSummary in case of iceberg.hive.keep.stats=false #5215
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
asf-ci-hive
added
tests pending
tests unstable
tests failed
and removed
tests pending
tests unstable
labels
zhangbutao
force-pushed
the
iceberg_count_optimize
branch
from
0c14207 to
1a953e8
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests failed
tests pending
labels
zhangbutao
force-pushed
the
iceberg_count_optimize
branch
from
1a953e8 to
77d9a7e
Compare
zhangbutao
commented
Apr 28, 2024
| filterExpr: (a = 22) (type: boolean) | ||
| Snapshot ref: branch_test1 | ||
| Statistics: Num rows: 3 Data size: 291 Basic stats: COMPLETE Column stats: COMPLETE | ||
| Statistics: Num rows: 5 Data size: 485 Basic stats: COMPLETE Column stats: COMPLETE |
Contributor
Author
There was a problem hiding this comment.
Before this PR, we always get row count of branch/tag/timetravel by the current snapshot summary, which is not right.
zhangbutao
force-pushed
the
iceberg_count_optimize
branch
from
77d9a7e to
441db00
Compare
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
zhangbutao
force-pushed
the
iceberg_count_optimize
branch
from
441db00 to
9971db5
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests failed
tests pending
labels
zhangbutao
force-pushed
the
iceberg_count_optimize
branch
from
9971db5 to
0ffc9df
Compare
zhangbutao
force-pushed
the
iceberg_count_optimize
branch
from
1347fef to
7b2ffb2
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
deniskuzZ
reviewed
Oct 24, 2024
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
Outdated
Show resolved
Hide resolved
deniskuzZ
reviewed
Oct 24, 2024
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
Outdated
Show resolved
Hide resolved
deniskuzZ
reviewed
Oct 24, 2024
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java
Outdated
Show resolved
Hide resolved
deniskuzZ
reviewed
Oct 24, 2024
deniskuzZ
reviewed
Oct 24, 2024
ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java
Outdated
Show resolved
Hide resolved
…ceberg.hive.keep.stats=false
zhangbutao
force-pushed
the
iceberg_count_optimize
branch
from
7b2ffb2 to
deb46ff
Compare
deniskuzZ
approved these changes
Oct 25, 2024
Member
There was a problem hiding this comment.
LGTM +1, pending tests
thanks @zhangbutao for addressing the review comment quickly!
asf-ci-hive
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
At present, in case of
iceberg.hive.keep.stats=true&hive.compute.query.using.stats=true, HS2 will do a fetch task to get iceberg table'snumRowsproperty from HMS to optimizecountquery.If
iceberg.hive.keep.stats=false, HS2 will always launch tez task to compute table's row count when filing acountquery.However, as we know, iceberg table's metadata has some stats information, we can also just start a fetch task to retrieve the row count from iceberg's snapshot summary when
iceberg.hive.keep.stats=falseor no stats stored in hms. This can avoid launching tez task to compute the table's row count.BTW, timetravel or branch/tag has different stats from current snapshot, so we need to get the specified snapshotid based on the different iceberg version. Otherwise, we will get the wrong stats when querying the time travel/branch/tag.
Why are the changes needed?
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
Qtest