Skip to content

[SPARK-41376][CORE] Correct the Netty preferDirectBufs check logic on executor start #38901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pan3793 wants to merge 4 commits into from
Closed

[SPARK-41376][CORE] Correct the Netty preferDirectBufs check logic on executor start #38901

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9def9fc
[SPARK-41376][CORE] Executor netty direct memory check should respect…
pan3793 Dec 4, 2022
91515df
address
pan3793 Dec 5, 2022
5fc5080
address comment
pan3793 Dec 7, 2022
e1198f4
nit
pan3793 Dec 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,19 @@ private[spark] class CoarseGrainedExecutorBackend(

logInfo("Connecting to driver: " + driverUrl)
try {
if (PlatformDependent.directBufferPreferred() &&
// The following logic originally comes from the constructor of
Copy link
Member

@Ngone51 Ngone51 Dec 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add this logic into a NettyUitls function?

Copy link
Member Author

@pan3793 pan3793 Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid not, NettyUtils located at network-common, it can not access SparkConf which in core

Copy link
Contributor

@LuciferYang LuciferYang Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about NettyUtils just accept boolean inputs, not config

Copy link
Member Author

@pan3793 pan3793 Dec 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, @Ngone51 would you please take a look again?

// org.apache.spark.network.client.TransportClientFactory
val sharedByteBufAllocators =
env.conf.getBoolean("spark.network.sharedByteBufAllocators.enabled", true)
val preferDirectBufsForSharedByteBufAllocators =
env.conf.getBoolean("spark.network.io.preferDirectBufs", true)
val preferDirectBufs =
env.conf.getBoolean("spark.shuffle.io.preferDirectBufs", true)
Copy link
Member

@Ngone51 Ngone51 Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you see where is this conf accessed? I didn't find it used anywhere.

Copy link
Member Author

@pan3793 pan3793 Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a combination of ("spark", module, "io.preferDirectBufs"), for shuffle client, the module is "shuffle", see org.apache.spark.network.util.TransportConf

val shuffleClientPreferDirectBufs = PlatformDependent.directBufferPreferred() && {
if (sharedByteBufAllocators) preferDirectBufsForSharedByteBufAllocators
else preferDirectBufs
}
if (shuffleClientPreferDirectBufs &&
PlatformDependent.maxDirectMemory() < env.conf.get(MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM)) {
throw new SparkException(s"Netty direct memory should at least be bigger than " +
s"'${MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM.key}', but got " +
Expand Down