Skip to content

[SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. #24671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
beliefer wants to merge 12 commits into from
Closed

[SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. #24671

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ef6c505
improve docs about overhead.
beliefer May 22, 2019
d018b35
expansion of the docs to clarify memoryOverhead.
beliefer May 22, 2019
554caa2
Fix Scala style.
beliefer May 22, 2019
2fdc94f
Fix Scala style.
beliefer May 22, 2019
2c89b42
Fix Scala style.
beliefer May 22, 2019
3f79e89
Fix Scala style.
beliefer May 22, 2019
c4ab8a9
revert comment in core and add pyspark extended comment in docs.
beliefer May 27, 2019
f96f936
delete a blank.
beliefer May 27, 2019
1f31fc6
correct some typo.
beliefer May 28, 2019
f23c1b7
Adjust docs.
beliefer May 29, 2019
5dbbf4a
revert This option ...
beliefer May 30, 2019
7980652
Add may need to.
beliefer May 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/internal/config/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ package object config {
.createWithDefaultString("1g")

private[spark] val DRIVER_MEMORY_OVERHEAD = ConfigBuilder("spark.driver.memoryOverhead")
.doc("The amount of off-heap memory to be allocated per driver in cluster mode, " +
.doc("The amount of non-heap memory to be allocated per driver in cluster mode, " +
"in MiB unless otherwise specified.")
.bytesConf(ByteUnit.MiB)
.createOptional
Expand Down Expand Up @@ -185,7 +185,7 @@ package object config {
.createWithDefaultString("1g")

private[spark] val EXECUTOR_MEMORY_OVERHEAD = ConfigBuilder("spark.executor.memoryOverhead")
.doc("The amount of off-heap memory to be allocated per executor in cluster mode, " +
.doc("The amount of non-heap memory to be allocated per executor in cluster mode, " +
"in MiB unless otherwise specified.")
.bytesConf(ByteUnit.MiB)
.createOptional
Expand Down
26 changes: 21 additions & 5 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,10 +181,16 @@ of the most common options to set are:
<td><code>spark.driver.memoryOverhead</code></td>
<td>driverMemory * 0.10, with minimum of 384 </td>
<td>
The amount of off-heap memory to be allocated per driver in cluster mode, in MiB unless
otherwise specified. This is memory that accounts for things like VM overheads, interned strings,
Amount of non-heap memory to be allocated per driver process in cluster mode, in MiB unless
otherwise specified. This is memory that accounts for things like VM overheads, interned strings,
other native overheads, etc. This tends to grow with the container size (typically 6-10%).
This option is currently supported on YARN, Mesos and Kubernetes.
<em>Note:</em> Non-heap memory includes off-heap memory
(when <code>spark.memory.offHeap.enabled=true</code>) and memory used by other driver processes
(e.g. python process that goes with a PySpark driver) and memory used by other non-driver
processes running in the same container. The maximum memory size of container to running
driver is determined by the sum of <code>spark.driver.memoryOverhead</code>
and <code>spark.driver.memory</code>.
</td>
</tr>
<tr>
Expand Down Expand Up @@ -215,10 +221,17 @@ of the most common options to set are:
<td><code>spark.executor.memoryOverhead</code></td>
<td>executorMemory * 0.10, with minimum of 384 </td>
<td>
The amount of off-heap memory to be allocated per executor, in MiB unless otherwise specified.
This is memory that accounts for things like VM overheads, interned strings, other native
overheads, etc. This tends to grow with the executor size (typically 6-10%).
Amount of non-heap memory to be allocated per executor process in cluster mode, in MiB unless
otherwise specified. This is memory that accounts for things like VM overheads, interned strings,
other native overheads, etc. This tends to grow with the executor size (typically 6-10%).
This option is currently supported on YARN and Kubernetes.
<br/>
<em>Note:</em> Non-heap memory includes off-heap memory
(when <code>spark.memory.offHeap.enabled=true</code>) and memory used by other executor processes
(e.g. python process that goes with a PySpark executor) and memory used by other non-executor
processes running in the same container. The maximum memory size of container to running executor
is determined by the sum of <code>spark.executor.memoryOverhead</code> and
<code>spark.executor.memory</code>.
</td>
</tr>
<tr>
Expand Down Expand Up @@ -1233,6 +1246,9 @@ Apart from these, the following properties are also available, and may be useful
<td>
If true, Spark will attempt to use off-heap memory for certain operations. If off-heap memory
use is enabled, then <code>spark.memory.offHeap.size</code> must be positive.
<em>Note:</em> If off-heap memory is enabled, may need to raise the non-heap memory size
(e.g. increase <code>spark.driver.memoryOverhead</code> or
<code>spark.executor.memoryOverhead</code>).
</td>
</tr>
<tr>
Expand Down