Skip to content

test: enabling Spark tests with offHeap requirement #1177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kazuyukitanimura merged 7 commits into from
Dec 18, 2024
Merged

test: enabling Spark tests with offHeap requirement #1177

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
46f215d
fix: enabling Spark tests with offHeap requirement
kazuyukitanimura Dec 17, 2024
dd27c5f
test: enabling Spark tests with offHeap requirement
kazuyukitanimura Dec 18, 2024
8b3d6fa
test: enabling Spark tests with offHeap requirement
kazuyukitanimura Dec 18, 2024
f1e53aa
test: enabling Spark tests with offHeap requirement
kazuyukitanimura Dec 18, 2024
e019f75
test: enabling Spark tests with offHeap requirement
kazuyukitanimura Dec 18, 2024
67295cf
test: enabling Spark tests with offHeap requirement
kazuyukitanimura Dec 18, 2024
40c5371
test: enabling Spark tests with offHeap requirement
kazuyukitanimura Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ import org.apache.spark.sql.types.{DoubleType, FloatType}

import org.apache.comet.CometConf._
import org.apache.comet.CometExplainInfo.getActualPlan
import org.apache.comet.CometSparkSessionExtensions.{createMessage, getCometBroadcastNotEnabledReason, getCometShuffleNotEnabledReason, isANSIEnabled, isCometBroadCastForceEnabled, isCometEnabled, isCometExecEnabled, isCometJVMShuffleMode, isCometNativeShuffleMode, isCometScan, isCometScanEnabled, isCometShuffleEnabled, isSpark34Plus, isSpark40Plus, shouldApplySparkToColumnar, withInfo, withInfos}
import org.apache.comet.CometSparkSessionExtensions.{createMessage, getCometBroadcastNotEnabledReason, getCometShuffleNotEnabledReason, isANSIEnabled, isCometBroadCastForceEnabled, isCometEnabled, isCometExecEnabled, isCometJVMShuffleMode, isCometNativeShuffleMode, isCometScan, isCometScanEnabled, isCometShuffleEnabled, isOffHeapEnabled, isSpark34Plus, isSpark40Plus, isTesting, shouldApplySparkToColumnar, withInfo, withInfos}
import org.apache.comet.parquet.{CometParquetScan, SupportsComet}
import org.apache.comet.rules.RewriteJoin
import org.apache.comet.serde.OperatorOuterClass.Operator
Expand Down Expand Up @@ -921,8 +921,8 @@ class CometSparkSessionExtensions
override def apply(plan: SparkPlan): SparkPlan = {

// Comet required off-heap memory to be enabled
if ("true" != conf.getConfString("spark.memory.offHeap.enabled", "false")) {
logInfo("Comet extension disabled because spark.memory.offHeap.enabled=false")
if (!isOffHeapEnabled(conf) && !isTesting) {
logWarning("Comet extension disabled because spark.memory.offHeap.enabled=false")

@parthchandra parthchandra Dec 17, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add this to extended explain info?

@kazuyukitanimura kazuyukitanimura Dec 18, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

return plan
}

Expand Down Expand Up @@ -1174,8 +1174,7 @@ object CometSparkSessionExtensions extends Logging {
}

private[comet] def isOffHeapEnabled(conf: SQLConf): Boolean =
conf.contains("spark.memory.offHeap.enabled") &&
conf.getConfString("spark.memory.offHeap.enabled").toBoolean
conf.getConfString("spark.memory.offHeap.enabled", "false").toBoolean

// Copied from org.apache.spark.util.Utils which is private to Spark.
private[comet] def isTesting: Boolean = {
Expand Down