Skip to content

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #34594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
huaxingao wants to merge 3 commits into from
Closed

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #34594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3688e50
[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
huaxingao Nov 14, 2021
dd9cbb4
update bench mark results
huaxingao Nov 17, 2021
bac368c
address comments
huaxingao Nov 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 104 additions & 8 deletions sql/core/benchmarks/BloomFilterBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,119 @@
ORC Write
================================================================================================

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 19503 19621 166 5.1 195.0 1.0X
With bloom filter 22472 22710 335 4.4 224.7 0.9X
Without bloom filter 22533 23149 871 4.4 225.3 1.0X
With bloom filter 25897 26118 313 3.9 259.0 0.9X


================================================================================================
ORC Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 1981 2040 82 50.5 19.8 1.0X
With bloom filter 1428 1467 54 70.0 14.3 1.4X
Without bloom filter 2010 2093 118 49.8 20.1 1.0X
With bloom filter 1325 1388 90 75.5 13.2 1.5X


================================================================================================
Parquet Write
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 20190 20281 129 5.0 201.9 1.0X
With bloom filter 25112 25419 433 4.0 251.1 0.8X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 2097152 1198 1348 213 83.5 12.0 1.0X
With bloom filter, blocksize: 2097152 387 485 80 258.1 3.9 3.1X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 4194304 1087 1140 75 92.0 10.9 1.0X
With bloom filter, blocksize: 4194304 304 364 46 328.7 3.0 3.6X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 6291456 998 1016 25 100.2 10.0 1.0X
With bloom filter, blocksize: 6291456 363 411 45 275.7 3.6 2.8X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 8388608 1289 1356 95 77.6 12.9 1.0X
With bloom filter, blocksize: 8388608 632 668 40 158.1 6.3 2.0X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 12582912 1357 1402 62 73.7 13.6 1.0X
With bloom filter, blocksize: 12582912 1009 1041 45 99.1 10.1 1.3X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 16777216 1359 1394 49 73.6 13.6 1.0X
With bloom filter, blocksize: 16777216 1217 1249 45 82.2 12.2 1.1X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 33554432 1362 1362 1 73.4 13.6 1.0X
With bloom filter, blocksize: 33554432 1345 1395 71 74.3 13.5 1.0X


112 changes: 104 additions & 8 deletions sql/core/benchmarks/BloomFilterBenchmark-jdk17-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,119 @@
ORC Write
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.8.0-1042-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 23519 23980 652 4.3 235.2 1.0X
With bloom filter 26703 26898 275 3.7 267.0 0.9X
Without bloom filter 18752 18847 134 5.3 187.5 1.0X
With bloom filter 20964 21131 236 4.8 209.6 0.9X


================================================================================================
ORC Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.8.0-1042-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 1503 1514 16 66.6 15.0 1.0X
With bloom filter 1142 1159 23 87.5 11.4 1.3X
Without bloom filter 1445 1474 41 69.2 14.4 1.0X
With bloom filter 1085 1093 11 92.2 10.8 1.3X


================================================================================================
Parquet Write
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Write 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter 20162 20282 170 5.0 201.6 1.0X
With bloom filter 24370 24524 218 4.1 243.7 0.8X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 2097152 770 798 28 130.0 7.7 1.0X
With bloom filter, blocksize: 2097152 251 276 16 398.5 2.5 3.1X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 4194304 716 733 17 139.7 7.2 1.0X
With bloom filter, blocksize: 4194304 199 209 7 501.8 2.0 3.6X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 6291456 735 761 24 136.1 7.3 1.0X
With bloom filter, blocksize: 6291456 225 241 15 444.5 2.2 3.3X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 8388608 741 755 19 135.0 7.4 1.0X
With bloom filter, blocksize: 8388608 335 340 5 298.9 3.3 2.2X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 12582912 760 788 44 131.6 7.6 1.0X
With bloom filter, blocksize: 12582912 581 589 7 172.0 5.8 1.3X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 16777216 732 739 8 136.6 7.3 1.0X
With bloom filter, blocksize: 16777216 642 698 96 155.7 6.4 1.1X


================================================================================================
Parquet Read
================================================================================================

OpenJDK 64-Bit Server VM 17.0.1+12-LTS on Linux 5.11.0-1020-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Read a row from 100M rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------
Without bloom filter, blocksize: 33554432 1047 1049 3 95.5 10.5 1.0X
With bloom filter, blocksize: 33554432 1252 1265 19 79.9 12.5 0.8X


Loading