Skip to content

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #32473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
huaxingao wants to merge 30 commits into from
Closed

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #32473

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
ed16a1a
[SPARK-35345][SQL] Add BloomFilter Benchmark test for Parquet
huaxingao May 8, 2021
fa02810
enable ParquetInputFormat bloom filter to true
huaxingao May 8, 2021
1bb6675
fix lint scala
huaxingao May 8, 2021
2a3f5eb
update benchmark test result
huaxingao May 8, 2021
10d7a97
set parquet block size
huaxingao May 9, 2021
c8375d6
update benchmark result'
huaxingao May 9, 2021
34d0511
set parquet.block.size for withoutBF
huaxingao May 9, 2021
21cc2ac
update results
huaxingao May 9, 2021
47f70b7
adding measurements for 16, 64 and 128M
huaxingao May 12, 2021
05b20af
update test result
huaxingao May 12, 2021
df044b3
add benchmark test for IN set with 3000 predicates
huaxingao May 27, 2021
e582c26
change the num of predicates in IN set to 300 for now
huaxingao May 27, 2021
d4d39d3
comment out parquet IN set test for now
huaxingao May 28, 2021
c0e9d97
change num of predicate to 30
huaxingao May 28, 2021
5fc105f
[SPARK-35559][TEST] Speed up one test in AdaptiveQueryExecSuite
cloud-fan May 28, 2021
e5ee38c
[SPARK-35535][SQL] New data source V2 API: LocalScan
gengliangwang May 27, 2021
cf8e6d6
[SPARK-35194][SQL] Refactor nested column aliasing for readability
karenfeng May 28, 2021
0a2edad
[SPARK-35194][SQL][FOLLOWUP] Recover build error with Scala 2.13 on GA
sarutak May 28, 2021
b9e4faf
[SPARK-35098][PYTHON] Re-enable pandas-on-Spark test cases
xinrong-meng May 27, 2021
89eb722
change # of predicate
huaxingao Jun 9, 2021
2d089ff
Revert "change # of predicate"
huaxingao Jun 9, 2021
0220023
Revert "[SPARK-35098][PYTHON] Re-enable pandas-on-Spark test cases"
huaxingao Jun 9, 2021
62d8518
Revert "[SPARK-35194][SQL][FOLLOWUP] Recover build error with Scala 2…
huaxingao Jun 9, 2021
b7176e3
Revert "[SPARK-35194][SQL] Refactor nested column aliasing for readab…
huaxingao Jun 9, 2021
9a9f0ff
Revert "[SPARK-35535][SQL] New data source V2 API: LocalScan"
huaxingao Jun 9, 2021
324fedb
Revert "[SPARK-35559][TEST] Speed up one test in AdaptiveQueryExecSuite"
huaxingao Jun 9, 2021
d6e320c
reduce # of predicate
huaxingao Jun 9, 2021
d481ec1
update test results
huaxingao Jun 9, 2021
63edef3
address comment
huaxingao Jun 11, 2021
82e1e8e
fix error
huaxingao Jun 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 150 additions & 8 deletions sql/core/benchmarks/BloomFilterBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,165 @@
ORC Write
================================================================================================

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 19503 19621 166 5.1 195.0 1.0X
With bloom filter 22472 22710 335 4.4 224.7 0.9X
Without bloom filter 18388 18445 81 5.4 183.9 1.0X
With bloom filter 21555 21617 87 4.6 215.6 0.9X


================================================================================================
ORC Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 1981 2040 82 50.5 19.8 1.0X
With bloom filter 1428 1467 54 70.0 14.3 1.4X
Without bloom filter 1841 1898 81 54.3 18.4 1.0X
With bloom filter 1558 1696 194 64.2 15.6 1.2X


================================================================================================
Parquet Write
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 18525 18692 236 5.4 185.2 1.0X
With bloom filter 25506 25558 73 3.9 255.1 0.7X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 524288 1214 1328 161 82.4 12.1 1.0X
With bloom filter, blocksize: 524288 1192 1358 235 83.9 11.9 1.0X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 1048576 1114 1146 45 89.8 11.1 1.0X
With bloom filter, blocksize: 1048576 634 705 123 157.8 6.3 1.8X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 2097152 1071 1099 40 93.4 10.7 1.0X
With bloom filter, blocksize: 2097152 468 484 10 213.6 4.7 2.3X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 3145728 1081 1103 32 92.5 10.8 1.0X
With bloom filter, blocksize: 3145728 332 369 44 301.6 3.3 3.3X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 4194304 1057 1070 20 94.7 10.6 1.0X
With bloom filter, blocksize: 4194304 295 342 31 339.3 2.9 3.6X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 5242880 1006 1049 60 99.4 10.1 1.0X
With bloom filter, blocksize: 5242880 237 278 22 422.5 2.4 4.3X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 6291456 988 1040 74 101.2 9.9 1.0X
With bloom filter, blocksize: 6291456 268 305 27 373.6 2.7 3.7X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 7340032 1004 1031 38 99.6 10.0 1.0X
With bloom filter, blocksize: 7340032 476 547 60 210.2 4.8 2.1X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 8388608 1452 1469 25 68.9 14.5 1.0X
With bloom filter, blocksize: 8388608 658 677 17 152.0 6.6 2.2X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 9437184 1456 1477 30 68.7 14.6 1.0X
With bloom filter, blocksize: 9437184 779 825 40 128.4 7.8 1.9X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 10485760 1472 1474 3 68.0 14.7 1.0X
With bloom filter, blocksize: 10485760 777 830 65 128.7 7.8 1.9X
Loading