Skip to content

[SPARK-53966][CORE] Add utility functions to detect JVM GCs #52678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wankunde wants to merge 3 commits into from
Closed

[SPARK-53966][CORE] Add utility functions to detect JVM GCs #52678

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d60f948
[SPARK-53966][CORE] Reduce default page size by LONG_ARRAY_OFFSET if …
Oct 21, 2025
02868ac
Support Shenandoah check
Oct 22, 2025
ebf2e92
Support Shenandoah check
Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/memory/MemoryManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ private[spark] abstract class MemoryManager(
* by looking at the number of cores available to the process, and the total amount of memory,
* and then divide it by a factor of safety.
*
* SPARK-37593 If we are using G1GC, it's better to take the LONG_ARRAY_OFFSET
* SPARK-37593 If we are using G1GC or ZGC, it's better to take the LONG_ARRAY_OFFSET
* into consideration so that the requested memory size is power of 2
* and can be divided by G1 heap region size to reduce memory waste within one G1 region.
*/
Expand All @@ -260,7 +260,7 @@ private[spark] abstract class MemoryManager(
}
val size = ByteArrayMethods.nextPowerOf2(maxTungstenMemory / cores / safetyFactor)
val chosenPageSize = math.min(maxPageSize, math.max(minPageSize, size))
if (Utils.isG1GC && tungstenMemoryMode == MemoryMode.ON_HEAP) {
if ((Utils.isG1GC || Utils.isZGC) && tungstenMemoryMode == MemoryMode.ON_HEAP) {
chosenPageSize - Platform.LONG_ARRAY_OFFSET
} else {
chosenPageSize
Expand Down
15 changes: 11 additions & 4 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3138,7 +3138,14 @@ private[spark] object Utils
/**
* Return whether we are using G1GC or not
*/
lazy val isG1GC: Boolean = {
lazy val isG1GC: Boolean = checkUseGC("UseG1GC")

/**
* Return whether we are using ZGC or not
*/
lazy val isZGC: Boolean = checkUseGC("UseZGC")

def checkUseGC(useGCObjectStr: String): Boolean = {
Copy link
Member

@dongjoon-hyun dongjoon-hyun Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @wankunde . Thank you for making a PR. Could you spin off this contribution? This sounds like a good idea to add a general utility function to detect GCs. We can proceed this one first.

As @pan3793 mentioned, if you can generalize more to detect other GCs, it would be a perfect contribution.

Copy link
Contributor Author

@wankunde wankunde Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review. For this PR, I will only add a function to detect GCs for G1, ZGC and ShenandoahGC

Try {
val clazz = Utils.classForName("com.sun.management.HotSpotDiagnosticMXBean")
.asInstanceOf[Class[_ <: PlatformManagedObject]]
Expand All @@ -3147,9 +3154,9 @@ private[spark] object Utils
val vmOptionMethod = clazz.getMethod("getVMOption", classOf[String])
val valueMethod = vmOptionClazz.getMethod("getValue")

val useG1GCObject = vmOptionMethod.invoke(hotSpotDiagnosticMXBean, "UseG1GC")
val useG1GC = valueMethod.invoke(useG1GCObject).asInstanceOf[String]
"true".equals(useG1GC)
val useGCObject = vmOptionMethod.invoke(hotSpotDiagnosticMXBean, useGCObjectStr)
val useGC = valueMethod.invoke(useGCObject).asInstanceOf[String]
"true".equals(useGC)
}.getOrElse(false)
}

Expand Down