Cache file listings per transaction#11650

Merged

sopel39 merged 3 commits intotrinodb:masterfrom

starburstdata:ks/cache_listing

Apr 5, 2022

Member

sopel39 commented Mar 24, 2022

Related issues, pull requests, and links

# Hive
* Improve performance of queries that scan same table multiple times. ({issue}`issuenumber`)

cla-bot bot added the cla-signed label

sopel39 force-pushed the ks/cache_listing branch 2 times, most recently from c89ded2 to c49ef9b Compare

March 24, 2022 10:07

sopel39 requested review from findepi, findinpath, raunaqmorarka and skrzypo987

March 24, 2022 10:07

github-actions bot added the tests:hive label

findinpath reviewed

View reviewed changes

.../trino-hive/src/main/java/io/trino/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved

...no-hive/src/test/java/io/trino/plugin/hive/metastore/TestSemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/DirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/DirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveConfig.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

sopel39 commented

View reviewed changes

Member Author

sopel39 left a comment

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/DirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/DirectoryLister.java Outdated Show resolved Hide resolved

.../trino-hive/src/main/java/io/trino/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved

...no-hive/src/test/java/io/trino/plugin/hive/metastore/TestSemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

sopel39 force-pushed the ks/cache_listing branch from c49ef9b to f53edd7 Compare

March 24, 2022 14:00

sopel39 requested a review from findinpath

March 24, 2022 22:32

raunaqmorarka approved these changes

View reviewed changes

Member

raunaqmorarka left a comment

lgtm % minor comments

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeModule.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveConfig.java Outdated Show resolved Hide resolved

Member

raunaqmorarka commented Mar 25, 2022

This might have some implications for fault tolerant execution. E.g. if file listing is cached for a file which is deleted later on, now every retry of the query will fail due to a file being present in the listing cache but not on actual FS.

findepi requested a review from arhimondr

March 25, 2022 10:48

findinpath reviewed

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/FileSystemDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/DirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

sopel39 requested a review from findinpath

March 29, 2022 11:25

sopel39 force-pushed the ks/cache_listing branch from f53edd7 to 7e7fbcc Compare

March 29, 2022 11:25

findepi reviewed

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryListerModule.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystem.java Outdated Show resolved Hide resolved

.../trino-hive/src/main/java/io/trino/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

findinpath reviewed

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

sopel39 force-pushed the ks/cache_listing branch from 7e7fbcc to d235760 Compare

March 30, 2022 15:01

sopel39 commented

View reviewed changes

Member Author

sopel39 left a comment

ac

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

sopel39 requested review from findepi and findinpath

March 30, 2022 15:01

sopel39 force-pushed the ks/cache_listing branch from d235760 to f6c63c1 Compare

March 30, 2022 15:04

findepi reviewed

View reviewed changes

plugin/trino-hive/src/main/java/io/trino/plugin/hive/fs/CachingDirectoryLister.java Outdated Show resolved Hide resolved

sopel39 force-pushed the ks/cache_listing branch from f6c63c1 to 2bf8495 Compare

April 1, 2022 12:20

sopel39 requested a review from findepi

April 1, 2022 12:21

sopel39 force-pushed the ks/cache_listing branch from 2bf8495 to 1b20288 Compare

April 1, 2022 12:23


          Move CachingDirectoryLister to fs package

c9be173

sopel39 force-pushed the ks/cache_listing branch from 1b20288 to 4f8da02 Compare

April 1, 2022 12:37

findepi reviewed

View reviewed changes

plugin/trino-hive/src/test/java/io/trino/plugin/hive/HiveQueryRunner.java Outdated Show resolved Hide resolved

plugin/trino-hive/src/test/java/io/trino/plugin/hive/HiveQueryRunner.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

...trino-hive/src/main/java/io/trino/plugin/hive/fs/TransactionScopeCachingDirectoryLister.java Outdated Show resolved Hide resolved

findepi approved these changes

View reviewed changes

Member

findepi left a comment

% previous comments


          Fix formatting

0c5b67c

sopel39 force-pushed the ks/cache_listing branch 2 times, most recently from eb63898 to 17d02ce Compare

April 1, 2022 16:46


          Cache file listings per transaction

79cc1b5

Cache concurrent directory listing when same
table is being scanned during single query

sopel39 force-pushed the ks/cache_listing branch from 17d02ce to 79cc1b5 Compare

April 1, 2022 19:41

sopel39 merged commit 8aa13c4 into trinodb:master

sopel39 deleted the ks/cache_listing branch

April 5, 2022 13:22

sopel39 mentioned this pull request

Release notes for 376 #11692

Closed

github-actions bot added this to the 376 milestone

mosabua mentioned this pull request

Add Trino 376 release notes #11691

Merged

sopel39 mentioned this pull request

Make CachingDirectoryLister enabled by default on transaction level #10692

Closed

xiacongling mentioned this pull request

Coordinator out of memory due to too many FileStatuses been cached #14313

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels