Skip to content

[SPARK-37394][CORE] Skip registering with external shuffle server if a customized shuffle manager is configured #34672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yangwwei wants to merge 4 commits into from
Closed

[SPARK-37394][CORE] Skip registering with external shuffle server if a customized shuffle manager is configured #34672

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1ce523d
[SPARK-37394] Skip registering to ESS if a customized shuffle manager…
yangwwei Nov 19, 2021
ba7a800
Revert "[SPARK-37394] Skip registering to ESS if a customized shuffle…
yangwwei Nov 29, 2021
e01dc86
add a method in ShuffleManager trait
yangwwei Nov 30, 2021
0f08a5b
text update
yangwwei Nov 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions core/src/main/scala/org/apache/spark/storage/BlockManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -560,13 +560,22 @@ private[spark] class BlockManager(
}

private def registerWithExternalShuffleServer(): Unit = {
val shuffleManagerName = shuffleManager.getClass.getName
if (shuffleManagerName != classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName) {
// there is no need to register with the default external shuffle server
// when a customized shuffle manager is configured
logInfo(s"Customized shuffle manager ${shuffleManagerName} configured, " +
s"skip registering with the default external shuffle service")
return
}

logInfo("Registering executor with local external shuffle service.")
val shuffleManagerMeta =
if (Utils.isPushBasedShuffleEnabled(conf, isDriver = isDriver, checkSerializer = false)) {
s"${shuffleManager.getClass.getName}:" +
s"${shuffleManagerName}:" +
s"${diskBlockManager.getMergeDirectoryAndAttemptIDJsonString()}}}"
} else {
shuffleManager.getClass.getName
shuffleManagerName
}
val shuffleConfig = new ExecutorShuffleInfo(
diskBlockManager.localDirsString,
Expand Down