Skip to content

Avoid underestimating size of partitioned hive tables#17677

Merged
raunaqmorarka merged 2 commits intotrinodb:masterfrom
Dith3r:ke/not-analyzed
Jun 6, 2023
Merged

Avoid underestimating size of partitioned hive tables#17677
raunaqmorarka merged 2 commits intotrinodb:masterfrom
Dith3r:ke/not-analyzed

Conversation

@Dith3r
Copy link
Copy Markdown
Member

@Dith3r Dith3r commented May 29, 2023 •
edited by raunaqmorarka
Loading

Description

For not analyzed tables hive connector return rowCount equal to 0, which makes CBO to choose wrong execution plan, for example choosing broadcast join. This PR change 0 to NaN avoiding such situations.

Additional context and related issues

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive
* Improve query performance on partitioned hive tables in the absence of table statistics. ({issue}`17677`)

@cla-bot cla-bot bot added the cla-signed label May 29, 2023
@Dith3r Dith3r requested a review from raunaqmorarka May 29, 2023 11:12
@github-actions github-actions bot added hive Hive connector tests:hive labels May 29, 2023
@Dith3r Dith3r force-pushed the ke/not-analyzed branch 2 times, most recently from 8747df9 to 09324e0 Compare May 29, 2023 12:40
@Dith3r Dith3r requested a review from lukasz-stec May 29, 2023 13:15
@Dith3r Dith3r force-pushed the ke/not-analyzed branch 5 times, most recently from 2973348 to 1e0f15c Compare May 30, 2023 08:30
raunaqmorarka
raunaqmorarka reviewed May 30, 2023
View reviewed changes
Comment thread ...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/alluxio/AlluxioHiveMetastore.java Outdated
Comment thread plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java Outdated
Comment thread ...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/alluxio/AlluxioHiveMetastore.java Outdated
Comment thread plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastore.java Outdated
@Dith3r Dith3r force-pushed the ke/not-analyzed branch 2 times, most recently from 277a074 to 833d0c0 Compare May 30, 2023 11:40
@Dith3r Dith3r requested a review from raunaqmorarka May 30, 2023 11:59
@Dith3r Dith3r force-pushed the ke/not-analyzed branch 2 times, most recently from 9d42851 to 18677a5 Compare May 30, 2023 12:22
@Dith3r Dith3r marked this pull request as ready for review May 30, 2023 12:42
raunaqmorarka
raunaqmorarka reviewed May 30, 2023
View reviewed changes
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHive.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveFileMetastore.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveFileMetastore.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveInMemoryMetastore.java Outdated
@Dith3r Dith3r force-pushed the ke/not-analyzed branch 2 times, most recently from 3e4069b to b490f10 Compare May 31, 2023 08:07
@Dith3r Dith3r requested a review from raunaqmorarka May 31, 2023 08:13
@Dith3r Dith3r force-pushed the ke/not-analyzed branch from b490f10 to d3ec8f3 Compare May 31, 2023 08:33
raunaqmorarka
raunaqmorarka reviewed Jun 1, 2023
View reviewed changes
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHive.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveFileMetastore.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveFileMetastore.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveInMemoryMetastore.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveInMemoryMetastore.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveFileMetastore.java Outdated
@Dith3r Dith3r force-pushed the ke/not-analyzed branch 4 times, most recently from a073174 to 6a74e7c Compare June 1, 2023 10:40
@Dith3r Dith3r force-pushed the ke/not-analyzed branch 2 times, most recently from 2941ce6 to 76ebc5e Compare June 5, 2023 10:24
raunaqmorarka
raunaqmorarka reviewed Jun 5, 2023
View reviewed changes
Comment thread ...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/alluxio/AlluxioHiveMetastore.java Outdated
Comment thread ...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/alluxio/AlluxioHiveMetastore.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveFileMetastore.java Outdated
@Dith3r Dith3r force-pushed the ke/not-analyzed branch from 76ebc5e to 8d56f60 Compare June 6, 2023 07:54
@Dith3r Dith3r changed the title Partitioned table rowCount 0 treated as NaN Avoid underestimating size of partitioned hive tables Jun 6, 2023
raunaqmorarka
raunaqmorarka reviewed Jun 6, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@raunaqmorarka raunaqmorarka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm % minor comments

Comment thread plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/TestHive.java Outdated
Comment thread plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/TestHive.java Outdated
@Dith3r Dith3r force-pushed the ke/not-analyzed branch from 8d56f60 to e62319c Compare June 6, 2023 09:38
@raunaqmorarka raunaqmorarka merged commit 6028dec into trinodb:master Jun 6, 2023
@raunaqmorarka raunaqmorarka mentioned this pull request Jun 6, 2023
Closed
@github-actions github-actions bot added this to the 420 milestone Jun 6, 2023
@Dith3r Dith3r mentioned this pull request Jun 7, 2023
Merged
@colebow colebow mentioned this pull request Jun 21, 2023
Merged
@marvin-roesch marvin-roesch mentioned this pull request Aug 24, 2023
Closed
@Dith3r Dith3r deleted the ke/not-analyzed branch August 25, 2023 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raunaqmorarka raunaqmorarka raunaqmorarka approved these changes

@lukasz-stec lukasz-stec Awaiting requested review from lukasz-stec

Assignees

No one assigned

Labels

cla-signed hive Hive connector

Milestone

420

Development

Successfully merging this pull request may close these issues.

2 participants

@Dith3r @raunaqmorarka