Skip to content

Fix hive ANALYZE when row count stats are missing#18828

Merged
raunaqmorarka merged 1 commit intotrinodb:masterfrom
Dith3r:master
Aug 30, 2023
Merged

Fix hive ANALYZE when row count stats are missing#18828
raunaqmorarka merged 1 commit intotrinodb:masterfrom
Dith3r:master

Conversation

@Dith3r
Copy link
Copy Markdown
Member

@Dith3r Dith3r commented Aug 28, 2023 •
edited by raunaqmorarka
Loading

Description

Fixes #18798

Additional context and related issues

The problem is specific to hive metastore and newer versions of HMS have fixed this https://issues.apache.org/jira/browse/HIVE-19254

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive
* Fix failure in ANALYZE when row count stats are missing. ({issue}`18798`)

@cla-bot cla-bot bot added the cla-signed label Aug 28, 2023
@Dith3r Dith3r requested a review from raunaqmorarka August 28, 2023 08:33
@github-actions github-actions bot added tests:hive hive Hive connector labels Aug 28, 2023
raunaqmorarka
raunaqmorarka reviewed Aug 28, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@raunaqmorarka raunaqmorarka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a Hive product test which reproduces the problem before your fix ?
ANALYZE TABLE Table1 PARTITION(ds='2008-04-09', hr) COMPUTE STATISTICS NOSCAN; can be used in Hive to collect fileCount and onDiskDataSizeInBytes statistics without getting row count statistics

Comment thread plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveBasicStatistics.java Outdated
Comment thread plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHive.java Outdated
Comment thread plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastore.java Outdated
Comment thread plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java Outdated
Comment thread plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/TestHive.java Outdated
@Dith3r
Copy link
Copy Markdown
Member Author

Dith3r commented Aug 28, 2023

Using ANALYZE TABLE Table1 PARTITION(...) COMPUTE STATISTICS NOSCAN; and call to CALL hive.system.sync_partition_metadata does not trigger this issue.

@raunaqmorarka
Copy link
Copy Markdown
Member

raunaqmorarka commented Aug 28, 2023

Using ANALYZE TABLE Table1 PARTITION(...) COMPUTE STATISTICS NOSCAN; and call to CALL hive.system.sync_partition_metadata does not trigger this issue.

Okay, were you able to manually reproduce the problem and verify the fix ?

@Dith3r Dith3r changed the title Preserve thrift metastore fast stats Fix hive ANALYZE when row count stats are missing Aug 29, 2023
@Dith3r
Copy link
Copy Markdown
Member Author

Dith3r commented Aug 29, 2023

There is even fix for Hive for this issue (https://issues.apache.org/jira/browse/HIVE-19254). I can test it with some old version of hive.

@Dith3r Dith3r requested a review from raunaqmorarka August 29, 2023 13:09
@raunaqmorarka raunaqmorarka merged commit 9a5838a into trinodb:master Aug 30, 2023
@raunaqmorarka raunaqmorarka mentioned this pull request Aug 30, 2023
Closed
@github-actions github-actions bot added this to the 426 milestone Aug 30, 2023
@colebow colebow mentioned this pull request Aug 30, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raunaqmorarka raunaqmorarka raunaqmorarka approved these changes

Assignees

No one assigned

Labels

cla-signed hive Hive connector

Milestone

426

Development

Successfully merging this pull request may close these issues.

ANALYZE broken since 420 for Hive tables without (full) statistics

2 participants

@Dith3r @raunaqmorarka