Skip to content

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #32473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
huaxingao wants to merge 30 commits into from
Closed

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #32473

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
ed16a1a
[SPARK-35345][SQL] Add BloomFilter Benchmark test for Parquet
huaxingao May 8, 2021
fa02810
enable ParquetInputFormat bloom filter to true
huaxingao May 8, 2021
1bb6675
fix lint scala
huaxingao May 8, 2021
2a3f5eb
update benchmark test result
huaxingao May 8, 2021
10d7a97
set parquet block size
huaxingao May 9, 2021
c8375d6
update benchmark result'
huaxingao May 9, 2021
34d0511
set parquet.block.size for withoutBF
huaxingao May 9, 2021
21cc2ac
update results
huaxingao May 9, 2021
47f70b7
adding measurements for 16, 64 and 128M
huaxingao May 12, 2021
05b20af
update test result
huaxingao May 12, 2021
df044b3
add benchmark test for IN set with 3000 predicates
huaxingao May 27, 2021
e582c26
change the num of predicates in IN set to 300 for now
huaxingao May 27, 2021
d4d39d3
comment out parquet IN set test for now
huaxingao May 28, 2021
c0e9d97
change num of predicate to 30
huaxingao May 28, 2021
5fc105f
[SPARK-35559][TEST] Speed up one test in AdaptiveQueryExecSuite
cloud-fan May 28, 2021
e5ee38c
[SPARK-35535][SQL] New data source V2 API: LocalScan
gengliangwang May 27, 2021
cf8e6d6
[SPARK-35194][SQL] Refactor nested column aliasing for readability
karenfeng May 28, 2021
0a2edad
[SPARK-35194][SQL][FOLLOWUP] Recover build error with Scala 2.13 on GA
sarutak May 28, 2021
b9e4faf
[SPARK-35098][PYTHON] Re-enable pandas-on-Spark test cases
xinrong-meng May 27, 2021
89eb722
change # of predicate
huaxingao Jun 9, 2021
2d089ff
Revert "change # of predicate"
huaxingao Jun 9, 2021
0220023
Revert "[SPARK-35098][PYTHON] Re-enable pandas-on-Spark test cases"
huaxingao Jun 9, 2021
62d8518
Revert "[SPARK-35194][SQL][FOLLOWUP] Recover build error with Scala 2…
huaxingao Jun 9, 2021
b7176e3
Revert "[SPARK-35194][SQL] Refactor nested column aliasing for readab…
huaxingao Jun 9, 2021
9a9f0ff
Revert "[SPARK-35535][SQL] New data source V2 API: LocalScan"
huaxingao Jun 9, 2021
324fedb
Revert "[SPARK-35559][TEST] Speed up one test in AdaptiveQueryExecSuite"
huaxingao Jun 9, 2021
d6e320c
reduce # of predicate
huaxingao Jun 9, 2021
d481ec1
update test results
huaxingao Jun 9, 2021
63edef3
address comment
huaxingao Jun 11, 2021
82e1e8e
fix error
huaxingao Jun 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 148 additions & 6 deletions sql/core/benchmarks/BloomFilterBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,165 @@
ORC Write
================================================================================================

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 19503 19621 166 5.1 195.0 1.0X
With bloom filter 22472 22710 335 4.4 224.7 0.9X
Without bloom filter 21931 21987 79 4.6 219.3 1.0X
With bloom filter 24876 25137 369 4.0 248.8 0.9X


================================================================================================
ORC Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 1981 2040 82 50.5 19.8 1.0X
With bloom filter 1428 1467 54 70.0 14.3 1.4X
Without bloom filter 1694 1753 82 59.0 16.9 1.0X
With bloom filter 1346 1355 13 74.3 13.5 1.3X


================================================================================================
Parquet Write
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 16252 16780 747 6.2 162.5 1.0X
With bloom filter 24145 24686 765 4.1 241.5 0.7X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 524288 1252 1399 208 79.9 12.5 1.0X
With bloom filter, blocksize: 524288 1036 1203 237 96.5 10.4 1.2X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 1048576 1346 1413 94 74.3 13.5 1.0X
With bloom filter, blocksize: 1048576 686 786 149 145.8 6.9 2.0X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 2097152 1121 1129 12 89.2 11.2 1.0X
With bloom filter, blocksize: 2097152 385 417 26 260.0 3.8 2.9X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 3145728 1548 1559 16 64.6 15.5 1.0X
With bloom filter, blocksize: 3145728 323 347 35 309.4 3.2 4.8X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 4194304 1337 1366 41 74.8 13.4 1.0X
With bloom filter, blocksize: 4194304 318 345 24 314.4 3.2 4.2X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 5242880 1311 1374 90 76.3 13.1 1.0X
With bloom filter, blocksize: 5242880 380 410 21 263.2 3.8 3.5X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 6291456 1292 1332 56 77.4 12.9 1.0X
With bloom filter, blocksize: 6291456 338 407 70 295.8 3.4 3.8X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 7340032 1353 1381 40 73.9 13.5 1.0X
With bloom filter, blocksize: 7340032 336 400 43 297.5 3.4 4.0X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 8388608 1456 1468 18 68.7 14.6 1.0X
With bloom filter, blocksize: 8388608 662 697 38 150.9 6.6 2.2X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 9437184 1538 1585 66 65.0 15.4 1.0X
With bloom filter, blocksize: 9437184 653 710 72 153.2 6.5 2.4X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 10485760 1518 1524 9 65.9 15.2 1.0X
With bloom filter, blocksize: 10485760 610 715 93 164.0 6.1 2.5X
Loading