Skip to content

[SPARK-32346][SQL] Support filters pushdown in Avro datasource #29145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 23 commits into from
Closed

[SPARK-32346][SQL] Support filters pushdown in Avro datasource #29145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c010a4e
Support row skipping in Avro datasource
MaxGekk Jul 17, 2020
ab17bd0
Apply NoopFilters so far
MaxGekk Jul 17, 2020
665865d
Pushdown real filters
MaxGekk Jul 17, 2020
33d9383
Add a benchmark
MaxGekk Jul 17, 2020
ad21524
Fix the benchmark
MaxGekk Jul 17, 2020
0ea0ef7
Make timestamp read more expensive
MaxGekk Jul 17, 2020
cf083e2
Update AvroReadBenchmark-results.txt
MaxGekk Jul 17, 2020
5c4c177
Update AvroReadBenchmark-jdk11-results.txt
MaxGekk Jul 17, 2020
eb0c983
Tests for Avro deserializer
MaxGekk Jul 18, 2020
72022bb
Set JIRA ID in the benchmark
MaxGekk Jul 18, 2020
072eab0
CSVFilters -> OrderedFilters
MaxGekk Jul 18, 2020
7eb3f50
Move config checking from OrderedFilters
MaxGekk Jul 19, 2020
dc2f2a7
Add the SQL config spark.sql.avro.filterPushdown.enabled
MaxGekk Jul 19, 2020
fdfd2ca
Check the flag spark.sql.avro.filterPushdown.enabled
MaxGekk Jul 19, 2020
92efd4b
Add JIRA id to AvroCatalystDataConversionSuite
MaxGekk Jul 19, 2020
668e497
Add a test to AvroSuite to check the SQL config
MaxGekk Jul 19, 2020
fc48d13
Add an assert
MaxGekk Jul 20, 2020
74f8f1b
Add a comment to AvroReadBenchmark
MaxGekk Jul 20, 2020
19e3321
Add comment for nested records
MaxGekk Jul 21, 2020
438e634
Put common code to RowReader
MaxGekk Jul 21, 2020
0756a2d
!isEmpty -> nonEmpty
MaxGekk Jul 22, 2020
a20c04c
Remove an unused import
MaxGekk Jul 22, 2020
f6b2a9a
Add an assert to skipRow()
MaxGekk Jul 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 40 additions & 32 deletions external/avro/benchmarks/AvroReadBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,121 +2,129 @@
SQL Single Numeric Column Scan
================================================================================================

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum 2689 2694 7 5.8 170.9 1.0X
Sum 2872 2936 90 5.5 182.6 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum 2741 2759 26 5.7 174.2 1.0X
Sum 2810 2838 40 5.6 178.6 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum 2736 2748 17 5.7 173.9 1.0X
Sum 2901 2922 30 5.4 184.4 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum 3305 3317 17 4.8 210.2 1.0X
Sum 3387 3391 5 4.6 215.4 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum 2904 2952 68 5.4 184.6 1.0X
Sum 2890 2960 99 5.4 183.7 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum 3090 3093 4 5.1 196.5 1.0X
Sum 3067 3088 30 5.1 195.0 1.0X


================================================================================================
Int and String Scan
================================================================================================

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of columns 5351 5365 20 2.0 510.3 1.0X
Sum of columns 4736 4818 116 2.2 451.7 1.0X


================================================================================================
Partitioned Table Scan
================================================================================================

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Data column 3278 3288 14 4.8 208.4 1.0X
Partition column 3149 3193 62 5.0 200.2 1.0X
Both columns 3198 3204 7 4.9 203.4 1.0X
Data column 3383 3400 23 4.6 215.1 1.0X
Partition column 2949 2959 14 5.3 187.5 1.1X
Both columns 3522 3545 33 4.5 223.9 1.0X


================================================================================================
Repeated String Scan
================================================================================================

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length 3435 3438 5 3.1 327.6 1.0X
Sum of string length 3332 3355 32 3.1 317.7 1.0X


================================================================================================
String with Nulls Scan
================================================================================================

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length 5634 5650 23 1.9 537.3 1.0X
Sum of string length 5588 5652 90 1.9 532.9 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length 4725 4752 39 2.2 450.6 1.0X
Sum of string length 3858 3865 9 2.7 368.0 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length 3550 3566 23 3.0 338.6 1.0X
Sum of string length 2562 2571 12 4.1 244.3 1.0X


================================================================================================
Single Column Scan From Wide Columns
================================================================================================

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Single Column Scan from 100 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column 5271 5279 11 0.2 5027.0 1.0X
Sum of single column 5241 5243 3 0.2 4998.0 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Single Column Scan from 200 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column 10393 10516 174 0.1 9911.3 1.0X
Sum of single column 10178 10185 10 0.1 9706.5 1.0X

OpenJDK 64-Bit Server VM 11.0.5+10-post-Ubuntu-0ubuntu1.118.04 on Linux 4.15.0-1044-aws
OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Single Column Scan from 300 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column 15330 15343 19 0.1 14619.6 1.0X
Sum of single column 15201 15232 44 0.1 14496.4 1.0X


OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Filters pushdown: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
w/o filters 9614 9669 54 0.1 9614.1 1.0X
pushdown disabled 10077 10141 66 0.1 10077.2 1.0X
w/ filters 4681 4713 29 0.2 4681.5 2.1X

Loading