Skip to content

[SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. #24671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
beliefer wants to merge 12 commits into from
Closed

[SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. #24671

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ef6c505
improve docs about overhead.
beliefer May 22, 2019
d018b35
expansion of the docs to clarify memoryOverhead.
beliefer May 22, 2019
554caa2
Fix Scala style.
beliefer May 22, 2019
2fdc94f
Fix Scala style.
beliefer May 22, 2019
2c89b42
Fix Scala style.
beliefer May 22, 2019
3f79e89
Fix Scala style.
beliefer May 22, 2019
c4ab8a9
revert comment in core and add pyspark extended comment in docs.
beliefer May 27, 2019
f96f936
delete a blank.
beliefer May 27, 2019
1f31fc6
correct some typo.
beliefer May 28, 2019
f23c1b7
Adjust docs.
beliefer May 29, 2019
5dbbf4a
revert This option ...
beliefer May 30, 2019
7980652
Add may need to.
beliefer May 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/internal/config/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ package object config {
.createWithDefaultString("1g")

private[spark] val DRIVER_MEMORY_OVERHEAD = ConfigBuilder("spark.driver.memoryOverhead")
.doc("The amount of off-heap memory to be allocated per driver in cluster mode, " +
.doc("Amount of memory to be allocated outside the driver process in cluster mode, " +
"in MiB unless otherwise specified.")
.bytesConf(ByteUnit.MiB)
.createOptional
Expand Down Expand Up @@ -185,7 +185,7 @@ package object config {
.createWithDefaultString("1g")

private[spark] val EXECUTOR_MEMORY_OVERHEAD = ConfigBuilder("spark.executor.memoryOverhead")
.doc("The amount of off-heap memory to be allocated per executor in cluster mode, " +
.doc("Amount of memory to be allocated outside per executor process in cluster mode, " +
"in MiB unless otherwise specified.")
.bytesConf(ByteUnit.MiB)
.createOptional
Expand Down
10 changes: 5 additions & 5 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ of the most common options to set are:
<td><code>spark.driver.memoryOverhead</code></td>
<td>driverMemory * 0.10, with minimum of 384 </td>
<td>
The amount of off-heap memory to be allocated per driver in cluster mode, in MiB unless
Amount of memory to be allocated outside the driver process in cluster mode, in MiB unless
otherwise specified. This is memory that accounts for things like VM overheads, interned strings,
other native overheads, etc. This tends to grow with the container size (typically 6-10%).
This option is currently supported on YARN, Mesos and Kubernetes.
Expand Down Expand Up @@ -215,10 +215,10 @@ of the most common options to set are:
<td><code>spark.executor.memoryOverhead</code></td>
<td>executorMemory * 0.10, with minimum of 384 </td>
<td>
The amount of off-heap memory to be allocated per executor, in MiB unless otherwise specified.
This is memory that accounts for things like VM overheads, interned strings, other native
overheads, etc. This tends to grow with the executor size (typically 6-10%).
This option is currently supported on YARN and Kubernetes.
Amount of memory to be allocated outside per executor process in cluster mode, in MiB
unless otherwise specified.This is memory that accounts for things like VM overheads,
interned strings, other native overheads, etc. This tends to grow with the executor
size (typically 6-10%).This option is currently supported on YARN and Kubernetes.
</td>
</tr>
<tr>
Expand Down