Get stats from hive metastore only for the necessary columns #16203

lukasz-stec · 2023-02-21T13:13:58Z

Description

HiveTableHandle contains the list of projected columns. Since only projected columns will be available to the engine, HiveMetadata.getTableStatistics can return statistics limited to those columns.
This also requires CachingHiveMetastore to support caching statistics for a subset of table columns.

Benchmarks

I tested the change on the glue metastore and table with 100 columns (details below) with caching stats disabled (hive.metastore-stats-cache-ttl=0s) and a simple query (select count(c1) from test_col_stats_part) where planning dominates.

There is a difference of about 20% for planning time/elapsed (so visible)

baseline 2.251s on average and only required stats 1.853s

This would have a potentially higher impact on a metastore under heavy load.

The table used:

create table test_col_stats_part (c1, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11, c12, c13, c14, c15, c16, c17, c18, c19, c20, c21, c22, c23, c24, c25, c26, c27, c28, c29, c30, c31, c32, c33, c34, c35, c36, c37, c38, c39, c40, c41, c42, c43, c44, c45, c46, c47, c48, c49, c50, c51, c52, c53, c54, c55, c56, c57, c58, c59, c60, c61, c62, c63, c64, c65, c66, c67, c68, c69, c70, c71, c72, c73, c74, c75, c76, c77, c78, c79, c80, c81, c82, c83, c84, c85, c86, c87, c88, c89, c90, c91, c92, c93, c94, c95, c96, c97, c98, c99, c100, shipdate)
 WITH (                                              
    format = 'ORC',                                  
    partitioned_by = ARRAY['shipdate']               
 )
 as select 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, shipdate from hive.tpch_sf10_dec_orc.lineitem;

I also ran standard tpch/tpcds on scale factor 1000, orc, partitioned. The results show mostly no change (except for some variability)

sf1k-orc-part-240223.pdf

Additional context and related issues

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive
* Improve query planning performance for queries scanning tables with a larger number of columns. ({issue}`16203`)

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/cache/CachingHiveMetastore.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetastoreClosure.java

lukasz-stec

CA

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetastoreClosure.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetastoreClosure.java

.../trino-hive/src/main/java/io/trino/plugin/hive/metastore/SemiTransactionalHiveMetastore.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/cache/CachingHiveMetastore.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java

lukasz-stec

ca

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetastoreClosure.java

.../trino-hive/src/main/java/io/trino/plugin/hive/metastore/SemiTransactionalHiveMetastore.java

lukasz-stec · 2023-03-13T15:00:47Z

rebased on the latest master (auto rebase were causing compilation errors)

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/cache/CachingHiveMetastore.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveTableHandle.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java

Before applyProjection is called or when projection pushdown is disabled, we should assume that all table columns are to be projected for populating HiveTableHandle#projectedColumns

lukasz-stec

ca

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/cache/CachingHiveMetastore.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java

HiveTableHandle contains the list of projected columns. Since only projected columns will be available to the engine, HiveMetadata#getTableStatistics can return statistics limited to those columns. This also requires CachingHiveMetastore to support caching statistics for a subset of table columns.

findepi · 2023-03-23T13:52:05Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetastoreClosure.java

+            List<Column> requestedColumns = table.getDataColumns().stream()
+                    .filter(column -> requestedColumnNames.contains(column.getName()))
+                    .collect(toImmutableList());
+            table = Table.builder(table).setDataColumns(requestedColumns).build();


we're building fake table object, hoping the callee will trust the table, and not eg use knowledge about column list from some other place

that's an opaque design, something explicit would probably be better.
Why not pass Set<String> requestedColumns (or Optional thereof`) down the call chain?

Why not pass Set requestedColumns (or Optional thereof`) down the call chain?

That's an excellent idea. It also applies to the getPartitionStatistics, right?

potentially yes

cla-bot bot added the cla-signed label Feb 21, 2023

lukasz-stec added the performance label Feb 21, 2023

github-actions bot added the tests:hive label Feb 21, 2023

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch 5 times, most recently from 7a75b11 to 29b2138 Compare February 23, 2023 10:46

lukasz-stec changed the title ~~DRAFT Get stats from hive metastore only for the necessary columns~~ Get stats from hive metastore only for the necessary columns Feb 23, 2023

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch 3 times, most recently from 5d763ff to ee3f5fb Compare February 23, 2023 23:36

lukasz-stec marked this pull request as ready for review February 24, 2023 07:49

lukasz-stec requested review from Dith3r and raunaqmorarka February 24, 2023 07:50

raunaqmorarka requested a review from gaurav8297 February 24, 2023 08:00

Dith3r reviewed Feb 24, 2023

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/cache/CachingHiveMetastore.java Outdated Show resolved Hide resolved

Dith3r reviewed Feb 28, 2023

View reviewed changes

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch from ee3f5fb to 5cfb83d Compare February 28, 2023 11:52

lukasz-stec commented Feb 28, 2023

View reviewed changes

lukasz-stec requested a review from Dith3r February 28, 2023 11:52

Dith3r approved these changes Feb 28, 2023

View reviewed changes

github-actions bot added the hive Hive connector label Feb 28, 2023

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch from 5cfb83d to fede049 Compare March 13, 2023 10:49

raunaqmorarka reviewed Mar 13, 2023

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java Outdated Show resolved Hide resolved

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch from fede049 to d777bb7 Compare March 13, 2023 13:49

lukasz-stec commented Mar 13, 2023

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetastoreClosure.java Outdated Show resolved Hide resolved

.../trino-hive/src/main/java/io/trino/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved

Extract common empty projectedColumns constructor

f8d240e

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch from d777bb7 to d954002 Compare March 13, 2023 15:00

raunaqmorarka approved these changes Mar 13, 2023

View reviewed changes

Populate all columns in HiveTableHandle#projectedColumns by default

604c15d

Before applyProjection is called or when projection pushdown is disabled, we should assume that all table columns are to be projected for populating HiveTableHandle#projectedColumns

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch from d954002 to 7341c8f Compare March 14, 2023 08:00

lukasz-stec commented Mar 14, 2023

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/cache/CachingHiveMetastore.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java Outdated Show resolved Hide resolved

raunaqmorarka approved these changes Mar 14, 2023

View reviewed changes

lukasz-stec force-pushed the ls/067-hive-only-necessary-columns-stats branch from 7341c8f to 6280a6d Compare March 14, 2023 08:44

Dith3r approved these changes Mar 14, 2023

View reviewed changes

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

raunaqmorarka mentioned this pull request Mar 14, 2023

Flaky Query timed out after PT2S in Cassandra product tests #12818

Closed

raunaqmorarka merged commit d2130dd into trinodb:master Mar 14, 2023

raunaqmorarka deleted the ls/067-hive-only-necessary-columns-stats branch March 14, 2023 13:02

raunaqmorarka mentioned this pull request Mar 14, 2023

Release notes for 411 #16453

Closed

github-actions bot added this to the 411 milestone Mar 14, 2023

colebow mentioned this pull request Mar 14, 2023

Add Trino 411 release notes #16552

Merged

findepi mentioned this pull request Mar 21, 2023

Spurious CachingHiveMetastore failure when merging statistics #16653

Closed

findepi reviewed Mar 23, 2023

View reviewed changes

raunaqmorarka mentioned this pull request Dec 6, 2023

Fix negative caching for Hive column statistics #20034

Merged

losipiuk mentioned this pull request Mar 14, 2024

CachingHiveMetastore.getTableColumnStatistics not effective for some queries #21081

Open

Get stats from hive metastore only for the necessary columns #16203

Get stats from hive metastore only for the necessary columns #16203

Uh oh!

Conversation

lukasz-stec commented Feb 21, 2023 • edited by raunaqmorarka Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Benchmarks

Additional context and related issues

Release notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukasz-stec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukasz-stec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lukasz-stec commented Mar 13, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukasz-stec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

findepi Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukasz-stec Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

findepi Mar 24, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

lukasz-stec commented Feb 21, 2023 •

edited by raunaqmorarka

Loading

findepi Mar 23, 2023 •

edited

Loading