Skip to content

[SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource #28076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 8 commits into from
Closed

[SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource #28076

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c71829d
Add benchmarks for ORC
MaxGekk Mar 30, 2020
5007c14
Re-gen DateTimeBenchmark results on JDK 8
MaxGekk Mar 31, 2020
33694d2
Re-gen DateTimeBenchmark results on JDK 11
MaxGekk Mar 31, 2020
13df03e
Merge remote-tracking branch 'origin/master' into rebase-benchmark-orc
MaxGekk Mar 31, 2020
e70808c
Re-gen DateTimeBenchmark results on JDK 11
MaxGekk Mar 31, 2020
bdb7bc7
Regen benchmark results on Linux JDK 8
MaxGekk Mar 31, 2020
0a6f5ce
Merge remote-tracking branch 'remotes/origin/master' into rebase-benc…
MaxGekk Apr 1, 2020
91d1133
Fix merge issue
MaxGekk Apr 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 69 additions & 28 deletions sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,48 +6,89 @@ OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 9304 9304 0 10.7 93.0 1.0X
before 1582, noop 9187 9187 0 10.9 91.9 1.0X
after 1582, rebase off 22054 22054 0 4.5 220.5 0.4X
after 1582, rebase on 20361 20361 0 4.9 203.6 0.5X
before 1582, rebase off 20286 20286 0 4.9 202.9 0.5X
before 1582, rebase on 22230 22230 0 4.5 222.3 0.4X
after 1582, noop 9299 9299 0 10.8 93.0 1.0X
before 1582, noop 9220 9220 0 10.8 92.2 1.0X
after 1582, rebase off 20390 20390 0 4.9 203.9 0.5X
after 1582, rebase on 20378 20378 0 4.9 203.8 0.5X
before 1582, rebase off 20069 20069 0 5.0 200.7 0.5X
before 1582, rebase on 20637 20637 0 4.8 206.4 0.5X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 12773 12866 129 7.8 127.7 1.0X
after 1582, vec off, rebase on 13063 13086 39 7.7 130.6 1.0X
after 1582, vec on, rebase off 3678 3719 61 27.2 36.8 3.5X
after 1582, vec on, rebase on 5078 5121 52 19.7 50.8 2.5X
before 1582, vec off, rebase off 12942 12972 42 7.7 129.4 1.0X
before 1582, vec off, rebase on 13866 13904 58 7.2 138.7 0.9X
before 1582, vec on, rebase off 3678 3711 43 27.2 36.8 3.5X
before 1582, vec on, rebase on 5621 5657 44 17.8 56.2 2.3X
after 1582, vec off, rebase off 12927 13017 78 7.7 129.3 1.0X
after 1582, vec off, rebase on 13127 13176 50 7.6 131.3 1.0X
after 1582, vec on, rebase off 3725 3779 91 26.8 37.3 3.5X
after 1582, vec on, rebase on 5134 5221 99 19.5 51.3 2.5X
before 1582, vec off, rebase off 13049 13061 16 7.7 130.5 1.0X
before 1582, vec off, rebase on 13877 13916 51 7.2 138.8 0.9X
before 1582, vec on, rebase off 3702 3736 56 27.0 37.0 3.5X
before 1582, vec on, rebase on 5567 5637 78 18.0 55.7 2.3X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 2983 2983 0 33.5 29.8 1.0X
before 1582, noop 2979 2979 0 33.6 29.8 1.0X
after 1582, rebase off 17452 17452 0 5.7 174.5 0.2X
after 1582, rebase on 70193 70193 0 1.4 701.9 0.0X
before 1582, rebase off 17784 17784 0 5.6 177.8 0.2X
before 1582, rebase on 83498 83498 0 1.2 835.0 0.0X
after 1582, noop 2988 2988 0 33.5 29.9 1.0X
before 1582, noop 3000 3000 0 33.3 30.0 1.0X
after 1582, rebase off 16163 16163 0 6.2 161.6 0.2X
after 1582, rebase on 68399 68399 0 1.5 684.0 0.0X
before 1582, rebase off 16921 16921 0 5.9 169.2 0.2X
before 1582, rebase on 74425 74425 0 1.3 744.3 0.0X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 15114 15151 32 6.6 151.1 1.0X
after 1582, vec off, rebase on 45804 45912 126 2.2 458.0 0.3X
after 1582, vec on, rebase off 4900 4947 56 20.4 49.0 3.1X
after 1582, vec on, rebase on 34599 34650 45 2.9 346.0 0.4X
before 1582, vec off, rebase off 15093 15174 70 6.6 150.9 1.0X
before 1582, vec off, rebase on 47367 47472 121 2.1 473.7 0.3X
before 1582, vec on, rebase off 4884 4952 80 20.5 48.8 3.1X
before 1582, vec on, rebase on 35831 35883 59 2.8 358.3 0.4X
after 1582, vec off, rebase off 15147 15258 97 6.6 151.5 1.0X
after 1582, vec off, rebase on 45035 45101 60 2.2 450.3 0.3X
after 1582, vec on, rebase off 4934 5012 100 20.3 49.3 3.1X
after 1582, vec on, rebase on 34263 34360 88 2.9 342.6 0.4X
before 1582, vec off, rebase off 15177 15220 37 6.6 151.8 1.0X
before 1582, vec off, rebase on 46754 46761 12 2.1 467.5 0.3X
before 1582, vec on, rebase off 4892 4956 61 20.4 48.9 3.1X
before 1582, vec on, rebase on 35989 36014 22 2.8 359.9 0.4X


================================================================================================
Rebasing dates/timestamps in ORC datasource
================================================================================================

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 9295 9295 0 10.8 93.0 1.0X
before 1582, noop 9352 9352 0 10.7 93.5 1.0X
after 1582 17112 17112 0 5.8 171.1 0.5X
before 1582 17979 17979 0 5.6 179.8 0.5X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 20874 20905 38 4.8 208.7 1.0X
after 1582, vec on 3813 3844 28 26.2 38.1 5.5X
before 1582, vec off 25912 25949 38 3.9 259.1 0.8X
before 1582, vec on 4322 4343 19 23.1 43.2 4.8X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 3003 3003 0 33.3 30.0 1.0X
before 1582, noop 3012 3012 0 33.2 30.1 1.0X
after 1582 41031 41031 0 2.4 410.3 0.1X
before 1582 44436 44436 0 2.3 444.4 0.1X

OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 28477 28582 92 3.5 284.8 1.0X
after 1582, vec on 20754 20924 237 4.8 207.5 1.4X
before 1582, vec off 32858 32921 58 3.0 328.6 0.9X
before 1582, vec on 25734 25769 30 3.9 257.3 1.1X


Loading