Skip to content

Cache metastore (table and partition) stats by default #15811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sopel39 merged 2 commits into from
Jan 31, 2023
Merged

Cache metastore (table and partition) stats by default #15811

sopel39 merged 2 commits into from
Jan 31, 2023

Conversation

@sopel39
Copy link
Member

@sopel39 sopel39 commented Jan 23, 2023 •
edited
Loading 

Split caching into metadata and stats cache. Stats
pulling puts significant pressure on metastore. However,
stats don't have to be always up to date in order
to get good query plance. Therefore, stats can
be cached by default.

RELEASE NOTES:

Hive/Delta
* Reduce query latency by caching partition and table statistics by default. {}

@cla-bot cla-bot bot added the cla-signed label Jan 23, 2023
@sopel39
Copy link
Member Author

sopel39 commented Jan 23, 2023

cc @electrum

@sopel39 sopel39 force-pushed the ks/stats_cache branch 2 times, most recently from b0a2b41 to da55b27 Compare January 24, 2023 10:49
findepi
findepi reviewed Jan 24, 2023
View reviewed changes
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Please make sure not to rebase wrt to current master when applying changes. No need for fixup commits.

@sopel39 sopel39 requested a review from findepi January 24, 2023 14:25
findepi
findepi reviewed Jan 24, 2023
View reviewed changes
@sopel39 sopel39 requested a review from findepi January 24, 2023 15:30
findepi
findepi approved these changes Jan 24, 2023
View reviewed changes
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@sopel39 sopel39 force-pushed the ks/stats_cache branch 2 times, most recently from cdacf39 to 9173c1e Compare January 25, 2023 08:28
This was referenced Jan 25, 2023
Merged
Merged
@sopel39
Copy link
Member Author

sopel39 commented Jan 26, 2023

On top of #15864

@findepi
Copy link
Member

findepi commented Jan 27, 2023

On top of #15864

Please ping me when the other one is merged. Thanks!

@sopel39
Fix formatting
6e3f96e
@sopel39
Cache metastore (table and partition) stats by default
689e248 
Split caching into metadata and stats cache. Stats
pulling puts significant pressure on metastore. However,
stats don't have to be always up to date in order
to get good query plance. Therefore, stats can
be cached by default.
@sopel39 sopel39 requested a review from findepi January 30, 2023 13:56
@sopel39
Copy link
Member Author

sopel39 commented Jan 30, 2023

@findepi rebased

findepi
findepi approved these changes Jan 31, 2023
View reviewed changes
plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHive.java
.hdfsEnvironment(hdfsEnvironment)
.build()))
.executor(executor)
.metadataCacheEnabled(true)
Copy link
Member

@findepi findepi Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why using a different setup than the production default here?
add a code comment

Copy link
Member Author

@sopel39 sopel39 Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure. Cache was enabled here "forever"

@sopel39 sopel39 merged commit a73ca06 into trinodb:master Jan 31, 2023
@sopel39 sopel39 deleted the ks/stats_cache branch January 31, 2023 12:12
@sopel39 sopel39 mentioned this pull request Jan 31, 2023
Closed
@github-actions github-actions bot added this to the 407 milestone Jan 31, 2023
@colebow colebow mentioned this pull request Jan 31, 2023
Merged
@atanasenko atanasenko mentioned this pull request Feb 27, 2023
Merged
sopel39 added a commit to starburstdata/trino that referenced this pull request Mar 19, 2023
@sopel39
Make hive.metastore-stats-cache-ttl >= to hive.metastore-cache-ttl
be8d1d3 
Some deployments might had hive.metastore-cache-ttl already set.
trinodb#15811 introduced new config
hive.metastore-stats-cache-ttl which could be lower (by default)
for such deployments. However, hive.metastore-cache-ttl
should take precedense over hive.metastore-stats-cache-ttl as
it's more generic (affects whole metastore cache).
@sopel39 sopel39 mentioned this pull request Mar 19, 2023
Merged
sopel39 added a commit that referenced this pull request Mar 19, 2023
@sopel39
Make hive.metastore-stats-cache-ttl >= to hive.metastore-cache-ttl
52ec6dc 
Some deployments might had hive.metastore-cache-ttl already set.
#15811 introduced new config
hive.metastore-stats-cache-ttl which could be lower (by default)
for such deployments. However, hive.metastore-cache-ttl
should take precedense over hive.metastore-stats-cache-ttl as
it's more generic (affects whole metastore cache).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@findepi findepi findepi approved these changes

@raunaqmorarka raunaqmorarka Awaiting requested review from raunaqmorarka

Assignees

No one assigned

Labels

cla-signed

Milestone

407

Development

Successfully merging this pull request may close these issues.

2 participants

@sopel39 @findepi