Skip to content

[SPARK-46528][BUILD] Upgrade zstd-jni to 1.5.5-11 #44515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 2 commits into from
Closed

[SPARK-46528][BUILD] Upgrade zstd-jni to 1.5.5-11 #44515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
370fd51
[SPARK-46528][BUILD] Upgrade zstd-jni to 1.5.5-11
dongjoon-hyun Dec 27, 2023
5bbf13d
Update benchmark results
dongjoon-hyun Dec 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 28 additions & 28 deletions core/benchmarks/ZStandardBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,48 @@
Benchmark ZStandardCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
Compression 10000 times at level 1 without buffer pool 672 681 10 0.0 67171.0 1.0X
Compression 10000 times at level 2 without buffer pool 715 718 4 0.0 71458.8 0.9X
Compression 10000 times at level 3 without buffer pool 831 835 4 0.0 83139.1 0.8X
Compression 10000 times at level 1 with buffer pool 609 611 2 0.0 60881.5 1.1X
Compression 10000 times at level 2 with buffer pool 648 649 1 0.0 64791.0 1.0X
Compression 10000 times at level 3 with buffer pool 744 751 6 0.0 74392.4 0.9X
Compression 10000 times at level 1 without buffer pool 674 920 293 0.0 67406.4 1.0X
Compression 10000 times at level 2 without buffer pool 882 884 3 0.0 88195.1 0.8X
Compression 10000 times at level 3 without buffer pool 973 978 4 0.0 97301.3 0.7X
Compression 10000 times at level 1 with buffer pool 955 955 1 0.0 95452.0 0.7X
Compression 10000 times at level 2 with buffer pool 994 996 2 0.0 99432.1 0.7X
Compression 10000 times at level 3 with buffer pool 1093 1101 12 0.0 109300.9 0.6X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Decompression 10000 times from level 1 without buffer pool 842 849 12 0.0 84240.0 1.0X
Decompression 10000 times from level 2 without buffer pool 842 846 6 0.0 84185.2 1.0X
Decompression 10000 times from level 3 without buffer pool 843 844 1 0.0 84285.4 1.0X
Decompression 10000 times from level 1 with buffer pool 770 771 1 0.0 77024.9 1.1X
Decompression 10000 times from level 2 with buffer pool 771 771 0 0.0 77120.4 1.1X
Decompression 10000 times from level 3 with buffer pool 770 771 0 0.0 77031.9 1.1X
Decompression 10000 times from level 1 without buffer pool 826 829 3 0.0 82591.4 1.0X
Decompression 10000 times from level 2 without buffer pool 825 826 1 0.0 82533.4 1.0X
Decompression 10000 times from level 3 without buffer pool 827 830 5 0.0 82715.3 1.0X
Decompression 10000 times from level 1 with buffer pool 763 764 1 0.0 76271.6 1.1X
Decompression 10000 times from level 2 with buffer pool 763 777 23 0.0 76321.2 1.1X
Decompression 10000 times from level 3 with buffer pool 763 765 2 0.0 76286.1 1.1X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 48 50 3 0.0 376597.0 1.0X
Parallel Compression with 1 workers 41 42 3 0.0 318927.3 1.2X
Parallel Compression with 2 workers 38 40 2 0.0 297410.2 1.3X
Parallel Compression with 4 workers 37 39 1 0.0 287605.8 1.3X
Parallel Compression with 8 workers 39 40 1 0.0 301948.1 1.2X
Parallel Compression with 16 workers 41 43 1 0.0 317095.6 1.2X
Parallel Compression with 0 workers 49 50 1 0.0 384188.1 1.0X
Parallel Compression with 1 workers 42 44 4 0.0 328139.4 1.2X
Parallel Compression with 2 workers 40 42 1 0.0 309013.2 1.2X
Parallel Compression with 4 workers 40 41 1 0.0 309732.2 1.2X
Parallel Compression with 8 workers 41 43 2 0.0 319730.2 1.2X
Parallel Compression with 16 workers 43 45 1 0.0 337944.2 1.1X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 174 175 1 0.0 1360596.3 1.0X
Parallel Compression with 1 workers 189 228 24 0.0 1477060.7 0.9X
Parallel Compression with 2 workers 109 118 15 0.0 851455.9 1.6X
Parallel Compression with 4 workers 114 118 3 0.0 891964.9 1.5X
Parallel Compression with 8 workers 115 122 4 0.0 899748.7 1.5X
Parallel Compression with 16 workers 119 123 2 0.0 931210.7 1.5X
Parallel Compression with 0 workers 160 161 1 0.0 1250203.7 1.0X
Parallel Compression with 1 workers 196 197 2 0.0 1529028.2 0.8X
Parallel Compression with 2 workers 114 121 10 0.0 892592.4 1.4X
Parallel Compression with 4 workers 111 113 1 0.0 865617.7 1.4X
Parallel Compression with 8 workers 112 117 2 0.0 878723.8 1.4X
Parallel Compression with 16 workers 114 117 2 0.0 889199.7 1.4X


56 changes: 28 additions & 28 deletions core/benchmarks/ZStandardBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,48 @@
Benchmark ZStandardCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
Compression 10000 times at level 1 without buffer pool 666 669 3 0.0 66598.6 1.0X
Compression 10000 times at level 2 without buffer pool 711 711 1 0.0 71077.5 0.9X
Compression 10000 times at level 3 without buffer pool 816 816 0 0.0 81575.8 0.8X
Compression 10000 times at level 1 with buffer pool 591 592 1 0.0 59095.6 1.1X
Compression 10000 times at level 2 with buffer pool 630 632 1 0.0 62995.1 1.1X
Compression 10000 times at level 3 with buffer pool 742 742 0 0.0 74180.7 0.9X
Compression 10000 times at level 1 without buffer pool 666 669 3 0.0 66613.9 1.0X
Compression 10000 times at level 2 without buffer pool 708 709 1 0.0 70817.8 0.9X
Compression 10000 times at level 3 without buffer pool 818 819 1 0.0 81828.4 0.8X
Compression 10000 times at level 1 with buffer pool 591 596 9 0.0 59119.9 1.1X
Compression 10000 times at level 2 with buffer pool 629 630 1 0.0 62943.8 1.1X
Compression 10000 times at level 3 with buffer pool 746 747 2 0.0 74593.2 0.9X

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Decompression 10000 times from level 1 without buffer pool 600 602 1 0.0 60024.1 1.0X
Decompression 10000 times from level 2 without buffer pool 600 603 2 0.0 59973.0 1.0X
Decompression 10000 times from level 3 without buffer pool 601 602 1 0.0 60075.9 1.0X
Decompression 10000 times from level 1 with buffer pool 553 553 0 0.0 55316.4 1.1X
Decompression 10000 times from level 2 with buffer pool 553 554 1 0.0 55271.5 1.1X
Decompression 10000 times from level 3 with buffer pool 553 553 0 0.0 55261.4 1.1X
Decompression 10000 times from level 1 without buffer pool 600 603 2 0.0 60027.9 1.0X
Decompression 10000 times from level 2 without buffer pool 603 604 1 0.0 60270.8 1.0X
Decompression 10000 times from level 3 without buffer pool 602 604 2 0.0 60224.9 1.0X
Decompression 10000 times from level 1 with buffer pool 548 548 0 0.0 54774.5 1.1X
Decompression 10000 times from level 2 with buffer pool 548 548 1 0.0 54763.3 1.1X
Decompression 10000 times from level 3 with buffer pool 548 548 0 0.0 54751.8 1.1X

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 49 50 1 0.0 380070.1 1.0X
Parallel Compression with 1 workers 41 42 4 0.0 319807.1 1.2X
Parallel Compression with 2 workers 38 41 2 0.0 297706.4 1.3X
Parallel Compression with 4 workers 38 40 2 0.0 296505.8 1.3X
Parallel Compression with 8 workers 39 41 1 0.0 305793.6 1.2X
Parallel Compression with 16 workers 43 44 1 0.0 332233.1 1.1X
Parallel Compression with 0 workers 49 50 1 0.0 382234.6 1.0X
Parallel Compression with 1 workers 42 43 1 0.0 327419.1 1.2X
Parallel Compression with 2 workers 38 41 2 0.0 299638.6 1.3X
Parallel Compression with 4 workers 38 40 2 0.0 294671.3 1.3X
Parallel Compression with 8 workers 40 42 1 0.0 308641.0 1.2X
Parallel Compression with 16 workers 43 45 2 0.0 335800.5 1.1X

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure
OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 175 175 0 0.0 1363800.8 1.0X
Parallel Compression with 1 workers 186 187 3 0.0 1455096.4 0.9X
Parallel Compression with 2 workers 110 115 6 0.0 863272.6 1.6X
Parallel Compression with 4 workers 104 108 2 0.0 810721.1 1.7X
Parallel Compression with 8 workers 110 112 2 0.0 859303.5 1.6X
Parallel Compression with 16 workers 109 112 2 0.0 847863.6 1.6X
Parallel Compression with 0 workers 159 161 1 0.0 1242039.9 1.0X
Parallel Compression with 1 workers 201 203 4 0.0 1568116.4 0.8X
Parallel Compression with 2 workers 115 122 12 0.0 900801.9 1.4X
Parallel Compression with 4 workers 111 114 3 0.0 868535.8 1.4X
Parallel Compression with 8 workers 113 117 2 0.0 886200.6 1.4X
Parallel Compression with 16 workers 113 117 2 0.0 880790.0 1.4X


2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,4 +267,4 @@ xz/1.9//xz-1.9.jar
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
zookeeper-jute/3.9.1//zookeeper-jute-3.9.1.jar
zookeeper/3.9.1//zookeeper-3.9.1.jar
zstd-jni/1.5.5-10//zstd-jni-1.5.5-10.jar
zstd-jni/1.5.5-11//zstd-jni-1.5.5-11.jar
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -800,7 +800,7 @@
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>1.5.5-10</version>
<version>1.5.5-11</version>
</dependency>
<dependency>
<groupId>com.clearspring.analytics</groupId>
Expand Down