Skip to content

[SPARK-40985][BUILD] Upgrade RoaringBitmap to 0.9.35 #38465

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 3 commits into from
Closed

[SPARK-40985][BUILD] Upgrade RoaringBitmap to 0.9.35 #38465

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e1e696d
upgrade
LuciferYang Nov 1, 2022
5f2cb63
update bench result
LuciferYang Nov 1, 2022
f9087df
re-update bench result
LuciferYang Nov 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions core/benchmarks/MapStatusesConvertBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
OpenJDK 64-Bit Server VM 11.0.16.1+1 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 1269 1276 8 0.0 1268666001.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2672 2695 39 0.0 2671542753.0 0.5X
Num Maps: 50000 Fetch partitions:1500 4034 4069 50 0.0 4033696987.0 0.3X
Num Maps: 50000 Fetch partitions:500 1227 1262 47 0.0 1226744907.0 1.0X
Copy link
Contributor

@itholic itholic Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, seems like the Stdev slow-dowm almost 6x, it is acceptable performance degradation?

Copy link
Contributor Author

@LuciferYang LuciferYang Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stdev refers to the standard deviation estimated based on the sample. The standard deviation reflects the dispersion of the value relative to the mean.

From Per Row(ns), although the cpu frequency of the tester is slower than the base, the performance has not changed significantly

Num Maps: 50000 Fetch partitions:1000 2620 2637 15 0.0 2620288061.0 0.5X
Num Maps: 50000 Fetch partitions:1500 3975 3990 17 0.0 3974979610.0 0.3X


10 changes: 5 additions & 5 deletions core/benchmarks/MapStatusesConvertBenchmark-jdk17-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
OpenJDK 64-Bit Server VM 17.0.4.1+1 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 1228 1238 17 0.0 1228191051.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2380 2393 16 0.0 2379601524.0 0.5X
Num Maps: 50000 Fetch partitions:1500 3803 3857 55 0.0 3802550172.0 0.3X
Num Maps: 50000 Fetch partitions:500 1159 1184 38 0.0 1159155979.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2329 2387 57 0.0 2328833805.0 0.5X
Num Maps: 50000 Fetch partitions:1500 3608 3712 92 0.0 3607631972.0 0.3X


10 changes: 5 additions & 5 deletions core/benchmarks/MapStatusesConvertBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 1179 1187 13 0.0 1178581948.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2237 2257 21 0.0 2236562602.0 0.5X
Num Maps: 50000 Fetch partitions:1500 3448 3680 367 0.0 3447793753.0 0.3X
Num Maps: 50000 Fetch partitions:500 1099 1127 27 0.0 1099192398.0 1.0X
Num Maps: 50000 Fetch partitions:1000 1981 1999 16 0.0 1981390271.0 0.6X
Num Maps: 50000 Fetch partitions:1500 2973 3011 39 0.0 2973029597.0 0.4X


4 changes: 2 additions & 2 deletions dev/deps/spark-deps-hadoop-2-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
HikariCP/2.5.1//HikariCP-2.5.1.jar
JLargeArrays/1.5//JLargeArrays-1.5.jar
JTransforms/3.1//JTransforms-3.1.jar
RoaringBitmap/0.9.32//RoaringBitmap-0.9.32.jar
RoaringBitmap/0.9.35//RoaringBitmap-0.9.35.jar
ST4/4.0.4//ST4-4.0.4.jar
activation/1.1.1//activation-1.1.1.jar
aircompressor/0.21//aircompressor-0.21.jar
Expand Down Expand Up @@ -247,7 +247,7 @@ scala-library/2.12.17//scala-library-2.12.17.jar
scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
scala-reflect/2.12.17//scala-reflect-2.12.17.jar
scala-xml_2.12/2.1.0//scala-xml_2.12-2.1.0.jar
shims/0.9.32//shims-0.9.32.jar
shims/0.9.35//shims-0.9.35.jar
slf4j-api/2.0.3//slf4j-api-2.0.3.jar
snakeyaml/1.31//snakeyaml-1.31.jar
snappy-java/1.1.8.4//snappy-java-1.1.8.4.jar
Expand Down
4 changes: 2 additions & 2 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
HikariCP/2.5.1//HikariCP-2.5.1.jar
JLargeArrays/1.5//JLargeArrays-1.5.jar
JTransforms/3.1//JTransforms-3.1.jar
RoaringBitmap/0.9.32//RoaringBitmap-0.9.32.jar
RoaringBitmap/0.9.35//RoaringBitmap-0.9.35.jar
ST4/4.0.4//ST4-4.0.4.jar
activation/1.1.1//activation-1.1.1.jar
aircompressor/0.21//aircompressor-0.21.jar
Expand Down Expand Up @@ -234,7 +234,7 @@ scala-library/2.12.17//scala-library-2.12.17.jar
scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
scala-reflect/2.12.17//scala-reflect-2.12.17.jar
scala-xml_2.12/2.1.0//scala-xml_2.12-2.1.0.jar
shims/0.9.32//shims-0.9.32.jar
shims/0.9.35//shims-0.9.35.jar
slf4j-api/2.0.3//slf4j-api-2.0.3.jar
snakeyaml/1.31//snakeyaml-1.31.jar
snappy-java/1.1.8.4//snappy-java-1.1.8.4.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -830,7 +830,7 @@
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<version>0.9.32</version>
<version>0.9.35</version>
</dependency>

<!-- Netty Begin -->
Expand Down