Skip to content

[SPARK-36516][SQL] Support File Metadata Cache for ORC #33748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 26 commits into from
Closed

[SPARK-36516][SQL] Support File Metadata Cache for ORC #33748

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
45f4827
support orc file meta cache
LuciferYang Aug 16, 2021
a74c793
remove ForTailCacheReader
LuciferYang Aug 16, 2021
1a9bd3b
rename test case
LuciferYang Aug 16, 2021
30df269
remove private[sql] and add comments
LuciferYang Aug 16, 2021
42d2bfd
Reduce method encapsulation
LuciferYang Aug 16, 2021
0e6c52b
use PrivateMethodTester
LuciferYang Aug 16, 2021
95bae3c
add a configable maximumSize
LuciferYang Aug 16, 2021
ebb7e0b
rename config
LuciferYang Aug 17, 2021
c36a569
move test
LuciferYang Aug 17, 2021
b02de85
update benchmark result
LuciferYang Aug 17, 2021
406f91c
revert config name
LuciferYang Aug 18, 2021
3b15a82
add compile same type
LuciferYang Aug 18, 2021
2b99983
update mirco bench
LuciferYang Aug 18, 2021
82ddf4f
change the default value of ttlSinceLastAccess
LuciferYang Aug 18, 2021
1dd174e
change the default value of ttlSinceLastAccess
LuciferYang Aug 18, 2021
a339b1b
update conf doc to add Warning
LuciferYang Aug 18, 2021
7153d2a
use a list config
LuciferYang Aug 18, 2021
c3838e6
Add checkValue to spark.sql.fileMetaCache.enabledSourceList and test …
LuciferYang Aug 19, 2021
59d5bb9
change to use guava cache and update benchmark
LuciferYang Aug 19, 2021
4adeb62
rename test case
LuciferYang Aug 19, 2021
e5f9497
add SEC to ttl
LuciferYang Aug 19, 2021
ec8fa1c
Revert "change to use guava cache and update benchmark"
LuciferYang Aug 19, 2021
db90daf
Revert "Revert "change to use guava cache and update benchmark""
LuciferYang Aug 22, 2021
7327fdb
Merge branch 'upmaster' into SPARK-36516
LuciferYang Aug 22, 2021
2907b2c
Merge branch 'master' of github.com:apache/spark into SPARK-36516
LuciferYang Aug 23, 2021
a7eff43
Merge branch 'upmaster' into SPARK-36516
LuciferYang Sep 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -967,6 +967,32 @@ object SQLConf {
.booleanConf
.createWithDefault(false)

val FILE_META_CACHE_ENABLED_SOURCE_LIST = buildConf("spark.sql.fileMetaCache.enabledSourceList")
.doc("A comma-separated list of data source short names for which data source enabled file " +
"meta cache, now the file meta cache only support ORC, it is recommended to enabled this " +
"config when multiple queries are performed on the same dataset, default is false." +
"Warning: if the fileMetaCache is enabled, the existing data files should not be " +
"replaced with the same file name, otherwise there will be a risk of job failure or wrong " +
"data reading before the cache entry expires.")
.version("3.3.0")
.stringConf

@dongjoon-hyun dongjoon-hyun Aug 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add .checkValue? The valid value is only orc in this PR.
After merging this PR, you can extend it to parquet inside Parquet PR.

@LuciferYang LuciferYang Aug 19, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c3838e6 add .checkValue and test case

.createWithDefault("")

val FILE_META_CACHE_TTL_SINCE_LAST_ACCESS =
buildConf("spark.sql.fileMetaCache.ttlSinceLastAccess")
.version("3.3.0")
.doc("Time-to-live for file metadata cache entry after last access, the unit is seconds.")
.timeConf(TimeUnit.SECONDS)
.createWithDefault(600L)

val FILE_META_CACHE_MAXIMUM_SIZE =
buildConf("spark.sql.fileMetaCache.maximumSize")
.version("3.3.0")
.doc("Maximum number of file meta entries the file meta cache contains.")
.intConf
.checkValue(_ > 0, "The value of fileMetaCache maximumSize must be positive")
.createWithDefault(1000)

val HIVE_VERIFY_PARTITION_PATH = buildConf("spark.sql.hive.verifyPartitionPath")
.doc("When true, check all the partition paths under the table\'s root directory " +
"when reading data stored in HDFS. This configuration will be deprecated in the future " +
Expand Down Expand Up @@ -3600,6 +3626,12 @@ class SQLConf extends Serializable with Logging {

def parquetVectorizedReaderBatchSize: Int = getConf(PARQUET_VECTORIZED_READER_BATCH_SIZE)

def fileMetaCacheEnabled(ds: String): Boolean = {
val enabledList = getConf(FILE_META_CACHE_ENABLED_SOURCE_LIST).toLowerCase(Locale.ROOT)
.split(",").map(_.trim)
enabledList.contains(ds.toLowerCase(Locale.ROOT))
}

def columnBatchSize: Int = getConf(COLUMN_BATCH_SIZE)

def cacheVectorizedReaderEnabled: Boolean = getConf(CACHE_VECTORIZED_READER_ENABLED)
Expand Down
95 changes: 95 additions & 0 deletions sql/core/benchmarks/FileMetaCacheReadBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
================================================================================================
count(*) From 100 files
================================================================================================

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 10 columns with 100 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 218 224 4 24.1 41.5 1.0X
count(*): fileMetaCacheEnabled = true 152 155 3 34.6 28.9 1.4X
count(*) with Filter: fileMetaCacheEnabled = false 430 441 10 12.2 82.1 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 375 377 4 14.0 71.5 0.6X

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 50 columns with 100 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 216 218 1 24.2 41.3 1.0X
count(*): fileMetaCacheEnabled = true 166 167 2 31.5 31.7 1.3X
count(*) with Filter: fileMetaCacheEnabled = false 455 460 5 11.5 86.7 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 406 412 10 12.9 77.4 0.5X

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 100 columns with 100 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 289 292 3 18.2 55.1 1.0X
count(*): fileMetaCacheEnabled = true 221 231 7 23.7 42.2 1.3X
count(*) with Filter: fileMetaCacheEnabled = false 556 560 9 9.4 106.0 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 518 520 2 10.1 98.9 0.6X


================================================================================================
count(*) From 500 files
================================================================================================

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 10 columns with 500 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 774 777 4 6.8 147.5 1.0X
count(*): fileMetaCacheEnabled = true 532 547 12 9.8 101.6 1.5X
count(*) with Filter: fileMetaCacheEnabled = false 1344 1346 3 3.9 256.3 0.6X
count(*) with Filter: fileMetaCacheEnabled = true 1095 1104 12 4.8 208.9 0.7X

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 50 columns with 500 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 798 854 95 6.6 152.2 1.0X
count(*): fileMetaCacheEnabled = true 659 664 3 8.0 125.6 1.2X
count(*) with Filter: fileMetaCacheEnabled = false 1482 1485 4 3.5 282.7 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 1286 1320 49 4.1 245.2 0.6X

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 100 columns with 500 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 882 919 54 5.9 168.3 1.0X
count(*): fileMetaCacheEnabled = true 678 726 42 7.7 129.3 1.3X
count(*) with Filter: fileMetaCacheEnabled = false 1680 1706 36 3.1 320.5 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 1438 1440 4 3.6 274.2 0.6X


================================================================================================
count(*) From 1000 files
================================================================================================

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 10 columns with 1000 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 1475 1516 58 3.6 281.4 1.0X
count(*): fileMetaCacheEnabled = true 1280 1285 7 4.1 244.2 1.2X
count(*) with Filter: fileMetaCacheEnabled = false 2512 2519 9 2.1 479.2 0.6X
count(*) with Filter: fileMetaCacheEnabled = true 2304 2305 2 2.3 439.4 0.6X

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 50 columns with 1000 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 1514 1518 6 3.5 288.8 1.0X
count(*): fileMetaCacheEnabled = true 1376 1377 2 3.8 262.4 1.1X
count(*) with Filter: fileMetaCacheEnabled = false 2793 2797 5 1.9 532.8 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 2635 2646 15 2.0 502.7 0.6X

OpenJDK 64-Bit Server VM 11.0.12+7-LTS on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 100 columns with 1000 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 1621 1622 0 3.2 309.2 1.0X
count(*): fileMetaCacheEnabled = true 1413 1424 17 3.7 269.4 1.1X
count(*) with Filter: fileMetaCacheEnabled = false 3144 3150 9 1.7 599.6 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 2855 2873 25 1.8 544.6 0.6X

95 changes: 95 additions & 0 deletions sql/core/benchmarks/FileMetaCacheReadBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
================================================================================================
count(*) From 100 files
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 10 columns with 100 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 191 197 9 27.5 36.4 1.0X
count(*): fileMetaCacheEnabled = true 132 136 5 39.6 25.2 1.4X
count(*) with Filter: fileMetaCacheEnabled = false 376 382 7 13.9 71.7 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 328 341 14 16.0 62.5 0.6X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 50 columns with 100 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 189 193 6 27.8 36.0 1.0X
count(*): fileMetaCacheEnabled = true 146 150 6 36.0 27.8 1.3X
count(*) with Filter: fileMetaCacheEnabled = false 399 404 7 13.1 76.1 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 353 358 5 14.9 67.3 0.5X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 100 columns with 100 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 237 241 7 22.1 45.3 1.0X
count(*): fileMetaCacheEnabled = true 190 193 7 27.5 36.3 1.2X
count(*) with Filter: fileMetaCacheEnabled = false 469 473 7 11.2 89.5 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 426 433 8 12.3 81.2 0.6X


================================================================================================
count(*) From 500 files
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 10 columns with 500 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 643 653 9 8.1 122.7 1.0X
count(*): fileMetaCacheEnabled = true 438 446 7 12.0 83.6 1.5X
count(*) with Filter: fileMetaCacheEnabled = false 1163 1163 1 4.5 221.8 0.6X
count(*) with Filter: fileMetaCacheEnabled = true 954 959 5 5.5 182.0 0.7X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 50 columns with 500 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 662 673 10 7.9 126.3 1.0X
count(*): fileMetaCacheEnabled = true 461 487 29 11.4 88.0 1.4X
count(*) with Filter: fileMetaCacheEnabled = false 1277 1278 0 4.1 243.7 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 1050 1058 11 5.0 200.3 0.6X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 100 columns with 500 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 716 723 11 7.3 136.6 1.0X
count(*): fileMetaCacheEnabled = true 502 523 30 10.4 95.8 1.4X
count(*) with Filter: fileMetaCacheEnabled = false 1449 1454 7 3.6 276.4 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 1211 1218 11 4.3 230.9 0.6X


================================================================================================
count(*) From 1000 files
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 10 columns with 1000 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 1249 1254 7 4.2 238.2 1.0X
count(*): fileMetaCacheEnabled = true 1114 1116 3 4.7 212.4 1.1X
count(*) with Filter: fileMetaCacheEnabled = false 2169 2176 9 2.4 413.8 0.6X
count(*) with Filter: fileMetaCacheEnabled = true 1884 1903 26 2.8 359.4 0.7X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 50 columns with 1000 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 1286 1286 0 4.1 245.3 1.0X
count(*): fileMetaCacheEnabled = true 1110 1135 35 4.7 211.7 1.2X
count(*) with Filter: fileMetaCacheEnabled = false 2362 2372 14 2.2 450.6 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 2198 2199 1 2.4 419.2 0.6X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_152-b16 on Linux 4.14.0_1-0-0-42
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
count(*) from 100 columns with 1000 files: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
count(*): fileMetaCacheEnabled = false 1351 1362 16 3.9 257.7 1.0X
count(*): fileMetaCacheEnabled = true 1169 1208 56 4.5 222.9 1.2X
count(*) with Filter: fileMetaCacheEnabled = false 2706 2712 9 1.9 516.1 0.5X
count(*) with Filter: fileMetaCacheEnabled = true 2428 2452 34 2.2 463.0 0.6X

Loading