Skip to content

[SPARK-31398][SQL] Fix perf regression of loading dates before 1582 year by non-vectorized ORC reader #28169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 6 commits into from
Closed

[SPARK-31398][SQL] Fix perf regression of loading dates before 1582 year by non-vectorized ORC reader #28169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
168d64d
Use rebaseJulianToGregorianDays in read
MaxGekk Apr 9, 2020
1bf0920
Remove import
MaxGekk Apr 9, 2020
6740329
Remove import
MaxGekk Apr 9, 2020
1438194
Re-gen DateTimeRebaseBenchmark results on JDK 8
MaxGekk Apr 9, 2020
9b5ca6b
Re-gen DateTimeRebaseBenchmark results on JDK 11
MaxGekk Apr 9, 2020
69bcf23
Merge branch 'orc-optimize-dates' of github.com:MaxGekk/spark into or…
MaxGekk Apr 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 44 additions & 44 deletions sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,49 +6,49 @@ OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 19037 19037 0 5.3 190.4 1.0X
before 1582, noop 11543 11543 0 8.7 115.4 1.6X
after 1582, rebase off 31999 31999 0 3.1 320.0 0.6X
after 1582, rebase on 31657 31657 0 3.2 316.6 0.6X
before 1582, rebase off 23777 23777 0 4.2 237.8 0.8X
before 1582, rebase on 24767 24767 0 4.0 247.7 0.8X
after 1582, noop 19946 19946 0 5.0 199.5 1.0X
before 1582, noop 10910 10910 0 9.2 109.1 1.8X
after 1582, rebase off 32065 32065 0 3.1 320.7 0.6X
after 1582, rebase on 31424 31424 0 3.2 314.2 0.6X
before 1582, rebase off 21764 21764 0 4.6 217.6 0.9X
before 1582, rebase on 22700 22700 0 4.4 227.0 0.9X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 12901 13119 211 7.8 129.0 1.0X
after 1582, vec off, rebase on 12968 13078 153 7.7 129.7 1.0X
after 1582, vec on, rebase off 3633 3673 37 27.5 36.3 3.6X
after 1582, vec on, rebase on 4988 5035 42 20.0 49.9 2.6X
before 1582, vec off, rebase off 12767 12856 77 7.8 127.7 1.0X
before 1582, vec off, rebase on 14012 14051 41 7.1 140.1 0.9X
before 1582, vec on, rebase off 3626 3641 15 27.6 36.3 3.6X
before 1582, vec on, rebase on 5388 5458 114 18.6 53.9 2.4X
after 1582, vec off, rebase off 13054 13273 193 7.7 130.5 1.0X
after 1582, vec off, rebase on 13337 13418 117 7.5 133.4 1.0X
after 1582, vec on, rebase off 3666 3740 116 27.3 36.7 3.6X
after 1582, vec on, rebase on 5097 5160 70 19.6 51.0 2.6X
before 1582, vec off, rebase off 12908 12984 76 7.7 129.1 1.0X
before 1582, vec off, rebase on 14514 14575 53 6.9 145.1 0.9X
before 1582, vec on, rebase off 3718 3790 63 26.9 37.2 3.5X
before 1582, vec on, rebase on 5472 5527 48 18.3 54.7 2.4X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 3075 3075 0 32.5 30.7 1.0X
before 1582, noop 3058 3058 0 32.7 30.6 1.0X
after 1582, rebase off 17236 17236 0 5.8 172.4 0.2X
after 1582, rebase on 19237 19237 0 5.2 192.4 0.2X
before 1582, rebase off 17317 17317 0 5.8 173.2 0.2X
before 1582, rebase on 19982 19982 0 5.0 199.8 0.2X
after 1582, noop 2718 2718 0 36.8 27.2 1.0X
before 1582, noop 2774 2774 0 36.0 27.7 1.0X
after 1582, rebase off 16515 16515 0 6.1 165.2 0.2X
after 1582, rebase on 18619 18619 0 5.4 186.2 0.1X
before 1582, rebase off 16382 16382 0 6.1 163.8 0.2X
before 1582, rebase on 19580 19580 0 5.1 195.8 0.1X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 14865 14957 142 6.7 148.6 1.0X
after 1582, vec off, rebase on 17154 17190 45 5.8 171.5 0.9X
after 1582, vec on, rebase off 4878 4912 56 20.5 48.8 3.0X
after 1582, vec on, rebase on 8604 8653 42 11.6 86.0 1.7X
before 1582, vec off, rebase off 14891 14929 33 6.7 148.9 1.0X
before 1582, vec off, rebase on 17708 17758 68 5.6 177.1 0.8X
before 1582, vec on, rebase off 4928 4946 17 20.3 49.3 3.0X
before 1582, vec on, rebase on 9147 9152 7 10.9 91.5 1.6X
after 1582, vec off, rebase off 15082 15112 43 6.6 150.8 1.0X
after 1582, vec off, rebase on 18035 18105 107 5.5 180.3 0.8X
after 1582, vec on, rebase off 4948 4970 19 20.2 49.5 3.0X
after 1582, vec on, rebase on 8761 8780 25 11.4 87.6 1.7X
before 1582, vec off, rebase off 15285 15389 103 6.5 152.8 1.0X
before 1582, vec off, rebase on 18277 18368 80 5.5 182.8 0.8X
before 1582, vec on, rebase off 4938 4954 15 20.3 49.4 3.1X
before 1582, vec on, rebase on 9220 9248 33 10.8 92.2 1.6X


================================================================================================
Expand All @@ -59,36 +59,36 @@ OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 18943 18943 0 5.3 189.4 1.0X
before 1582, noop 11151 11151 0 9.0 111.5 1.7X
after 1582 26738 26738 0 3.7 267.4 0.7X
before 1582 19636 19636 0 5.1 196.4 1.0X
after 1582, noop 19820 19820 0 5.0 198.2 1.0X
before 1582, noop 10825 10825 0 9.2 108.2 1.8X
after 1582 27705 27705 0 3.6 277.1 0.7X
before 1582 19390 19390 0 5.2 193.9 1.0X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 36176 36378 206 2.8 361.8 1.0X
after 1582, vec on 3802 3829 29 26.3 38.0 9.5X
before 1582, vec off 35815 35882 74 2.8 358.1 1.0X
before 1582, vec on 4163 4174 10 24.0 41.6 8.7X
after 1582, vec off 10401 10426 33 9.6 104.0 1.0X
after 1582, vec on 3788 3813 29 26.4 37.9 2.7X
before 1582, vec off 10832 10843 11 9.2 108.3 1.0X
before 1582, vec on 4182 4219 46 23.9 41.8 2.5X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 3033 3033 0 33.0 30.3 1.0X
before 1582, noop 3004 3004 0 33.3 30.0 1.0X
after 1582 53907 53907 0 1.9 539.1 0.1X
before 1582 57241 57241 0 1.7 572.4 0.1X
after 1582, noop 2787 2787 0 35.9 27.9 1.0X
before 1582, noop 2777 2777 0 36.0 27.8 1.0X
after 1582 52456 52456 0 1.9 524.6 0.1X
before 1582 54983 54983 0 1.8 549.8 0.1X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 38266 38306 66 2.6 382.7 1.0X
after 1582, vec on 30233 30339 162 3.3 302.3 1.3X
before 1582, vec off 43584 43753 259 2.3 435.8 0.9X
before 1582, vec on 35681 35832 140 2.8 356.8 1.1X
after 1582, vec off 38336 38425 88 2.6 383.4 1.0X
after 1582, vec on 30089 30198 96 3.3 300.9 1.3X
before 1582, vec off 42739 42801 87 2.3 427.4 0.9X
before 1582, vec on 31530 31591 61 3.2 315.3 1.2X


Loading