Skip to content

[SPARK-52924][CORE] Support ZSTD_strategy for compression #51630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 3 commits into from
Closed

[SPARK-52924][CORE] Support ZSTD_strategy for compression #51630

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4e68a6d
zstd-strategy
yaooqinn Jul 23, 2025
b0dc08c
[SPARK-52924]
yaooqinn Jul 23, 2025
516973a
[SPARK-52924]
yaooqinn Jul 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 64 additions & 31 deletions core/benchmarks/ZStandardBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,81 @@
Benchmark ZStandardCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1018-azure
OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
Compression 10000 times at level 1 without buffer pool 646 668 20 0.0 64639.2 1.0X
Compression 10000 times at level 2 without buffer pool 715 716 2 0.0 71496.2 0.9X
Compression 10000 times at level 3 without buffer pool 810 818 7 0.0 81013.5 0.8X
Compression 10000 times at level 1 with buffer pool 603 604 0 0.0 60335.0 1.1X
Compression 10000 times at level 2 with buffer pool 638 641 3 0.0 63817.7 1.0X
Compression 10000 times at level 3 with buffer pool 739 740 1 0.0 73912.1 0.9X

OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1018-azure
Compression 10000 times at level 1 without buffer pool 645 664 13 0.0 64521.5 1.0X
Compression 10000 times at level 2 without buffer pool 700 706 6 0.0 70013.1 0.9X
Compression 10000 times at level 3 without buffer pool 811 813 3 0.0 81082.2 0.8X
Compression 10000 times at level 1 with buffer pool 597 598 1 0.0 59661.6 1.1X
Compression 10000 times at level 2 with buffer pool 630 632 2 0.0 63037.2 1.0X
Compression 10000 times at level 3 with buffer pool 735 742 6 0.0 73535.4 0.9X

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Decompression 10000 times from level 1 without buffer pool 830 833 4 0.0 83030.7 1.0X
Decompression 10000 times from level 2 without buffer pool 832 833 1 0.0 83236.0 1.0X
Decompression 10000 times from level 3 without buffer pool 832 833 1 0.0 83183.1 1.0X
Decompression 10000 times from level 1 with buffer pool 758 759 1 0.0 75813.5 1.1X
Decompression 10000 times from level 2 with buffer pool 758 758 1 0.0 75767.1 1.1X
Decompression 10000 times from level 3 with buffer pool 757 758 1 0.0 75652.4 1.1X

OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1018-azure
Decompression 10000 times from level 1 without buffer pool 833 835 3 0.0 83296.4 1.0X
Decompression 10000 times from level 2 without buffer pool 831 832 2 0.0 83090.3 1.0X
Decompression 10000 times from level 3 without buffer pool 837 845 7 0.0 83701.2 1.0X
Decompression 10000 times from level 1 with buffer pool 762 764 1 0.0 76246.2 1.1X
Decompression 10000 times from level 2 with buffer pool 760 763 4 0.0 76020.8 1.1X
Decompression 10000 times from level 3 with buffer pool 761 763 1 0.0 76128.2 1.1X

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 66 66 0 0.0 512667.9 1.0X
Parallel Compression with 1 workers 56 58 2 0.0 435183.0 1.2X
Parallel Compression with 2 workers 46 47 1 0.0 356034.8 1.4X
Parallel Compression with 4 workers 41 43 1 0.0 318331.7 1.6X
Parallel Compression with 8 workers 44 46 1 0.0 342564.5 1.5X
Parallel Compression with 16 workers 48 51 2 0.0 371266.4 1.4X

OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1018-azure
Parallel Compression with 0 workers 66 68 1 0.0 513077.9 1.0X
Parallel Compression with 1 workers 54 58 3 0.0 424042.7 1.2X
Parallel Compression with 2 workers 44 47 1 0.0 347613.7 1.5X
Parallel Compression with 4 workers 41 43 1 0.0 318932.6 1.6X
Parallel Compression with 8 workers 43 45 1 0.0 337266.1 1.5X
Parallel Compression with 16 workers 47 50 1 0.0 363883.0 1.4X

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers 236 237 1 0.0 1847245.2 1.0X
Parallel Compression with 1 workers 251 252 2 0.0 1961753.5 0.9X
Parallel Compression with 2 workers 141 148 4 0.0 1100274.2 1.7X
Parallel Compression with 4 workers 129 133 3 0.0 1009465.5 1.8X
Parallel Compression with 8 workers 135 139 3 0.0 1054496.8 1.8X
Parallel Compression with 16 workers 135 139 6 0.0 1051577.1 1.8X
Parallel Compression with 0 workers 236 241 5 0.0 1841027.0 1.0X
Parallel Compression with 1 workers 259 265 3 0.0 2025243.3 0.9X
Parallel Compression with 2 workers 140 151 9 0.0 1095661.5 1.7X
Parallel Compression with 4 workers 132 140 6 0.0 1027414.9 1.8X
Parallel Compression with 8 workers 138 143 3 0.0 1079814.7 1.7X
Parallel Compression with 16 workers 135 139 4 0.0 1053311.9 1.7X

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
Compression at level 1: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression by strategy -1 54 57 2 0.0 422355.5 1.0X
Compression by strategy 1 53 55 1 0.0 411561.0 1.0X
Compression by strategy 3 87 89 1 0.0 676134.2 0.6X
Compression by strategy 5 231 233 1 0.0 1805263.1 0.2X
Compression by strategy 7 252 253 1 0.0 1969415.3 0.2X
Compression by strategy 9 252 253 1 0.0 1965181.2 0.2X

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression by strategy -1 65 67 1 0.0 511151.0 1.0X
Compression by strategy 1 49 51 1 0.0 380610.7 1.3X
Compression by strategy 3 85 87 1 0.0 665865.9 0.8X
Compression by strategy 5 236 240 3 0.0 1841743.1 0.3X
Compression by strategy 7 272 274 1 0.0 2128783.4 0.2X
Compression by strategy 9 271 274 2 0.0 2119704.9 0.2X

OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
AMD EPYC 7763 64-Core Processor
Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression by strategy -1 233 234 1 0.0 1821157.0 1.0X
Compression by strategy 1 48 50 1 0.0 377701.6 4.8X
Compression by strategy 3 86 87 1 0.0 670743.4 2.7X
Compression by strategy 5 236 241 5 0.0 1844175.2 1.0X
Compression by strategy 7 274 277 2 0.0 2141909.1 0.9X
Compression by strategy 9 275 277 2 0.0 2148129.3 0.8X


Loading