[SPARK-53966][CORE] Add utility functions to detect JVM GCs

[wankunde]

What changes were proposed in this pull request?

This PR aims to add utility functions to detect JVM GCs.

Why are the changes needed?

To provide a general capability to optimize based on the GC types.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual review.

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

According to https://dev.java/learn/jvm/tool/garbage-collection/zgc-deepdive/

ZGC has three different sizes for regions; small, medium, and large.

Seems the waste is not as bad as G1, but this change still benefits ZGC memory utilization.

Have you used Shenandoah GC? According to #34846 (review), I think we should apply the same change to Shenandoah GC.

The same behavior affects the Shenandoah GC as well since it reuses the G1 region-based heap structure.

[dongjoon-hyun]

Hi, @wankunde . Thank you for making a PR. Could you spin off this contribution? This sounds like a good idea to add a general utility function to detect GCs. We can proceed this one first.

As @pan3793 mentioned, if you can generalize more to detect other GCs, it would be a perfect contribution.

[wankunde]

Thanks for your review. For this PR, I will only add a function to detect GCs for G1, ZGC and ShenandoahGC

[wankunde]

According to https://dev.java/learn/jvm/tool/garbage-collection/zgc-deepdive/

ZGC has three different sizes for regions; small, medium, and large.

Seems the waste is not as bad as G1, but this change still benefits ZGC memory utilization.

Have you used Shenandoah GC? According to #34846 (review), I think we should apply the same change to Shenandoah GC.

The same behavior affects the Shenandoah GC as well since it reuses the G1 region-based heap structure.

Hi, @pan3793, @dongjoon-hyun run the previous test with -Xmx1000m and different GCs, I think this optimization is also helpful for ZGC and ShenandoahGC.

Test code: https://gist.githubusercontent.com/wankunde/b7cf073b31f3a4bf5b525ab4a6b1f700/raw/85db977f5ef8609db27ced49d10ace9a248433c1/TestBlockSize.java

Test result:

	Max allocated size without optimization	Max allocated size with optimization
G1GC	817889280	1019211984
ZGC	671088640	1006629120
ShenandoahGC	855638016	964686240

Detail test result: https://gist.github.com/wankunde/6f3b6ed8abaa247307469ab5a4fae2bb

[wankunde]

@pan3793

ZGC has three different sizes for regions; small, medium, and large.

Large Regions
Large regions are reserved for humongous objects and are tightly fitted in 2 MB increments to the object's size.

For ZGC, if we allocate a 4MB long array, approximately 2MB / (4MB + 2MB) ~ 33% of memory is wasted;
if we allocate a 16MB long array, approximately 2MB / (16MB + 2MB) ~ 11% of memory are wasted.

Test ZGC with -Xmx1000m without this optimization:

[0.580s][info][gc] GC(11) Major Collection (Allocation Stall) 988M(99%)->996M(100%) 0.030s
Total allocated size: 889192448
[0.581s][info][gc] GC(12) Minor Collection (Allocation Stall)
[0.587s][info][gc] Relocation Stall (Attach Listener) 0.053ms
[0.587s][info][gc] GC(12) Minor Collection (Allocation Stall) 996M(100%)->1000M(100%) 0.005s
[0.587s][info][gc] GC(13) Major Collection (Allocation Stall)
[0.593s][info][gc] GC(14) Minor Collection (Allocation Stall)
[0.597s][info][gc] GC(14) Minor Collection (Allocation Stall) 996M(100%)->996M(100%) 0.004s
[0.606s][info][gc] GC(13) Major Collection (Allocation Stall) 1000M(100%)->992M(99%) 0.019s
[0.606s][info][gc] Allocation Stall (main) 24.366ms
[0.606s][info][gc] Out Of Memory (main)
Terminating due to java.lang.OutOfMemoryError: Java heap space

[dongjoon-hyun]

+1, LGTM. Thank you for revision, @wankunde .

I revised the PR title and description according to your latest code change.

Also, thank you, @pan3793 , too.

[dongjoon-hyun]

I manually verified. Merged to master for Apache Spark 4.1.0-preview3.

$ bin/spark-shell --conf "spark.driver.extraJavaOptions=-XX:+UseShenandoahGC"
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.1.0-SNAPSHOT
      /_/

Using Scala version 2.13.17 (OpenJDK 64-Bit Server VM, Java 21.0.8)
Type in expressions to have them evaluated.
Type :help for more information.
25/10/23 09:11:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1761235900588).
Spark session available as 'spark'.

scala> org.apache.spark.util.Utils.isG1GC
val res0: Boolean = false

scala> org.apache.spark.util.Utils.isZGC
val res1: Boolean = false

scala> org.apache.spark.util.Utils.isShenandoahGC
val res2: Boolean = true