Skip to content

Commit 062c336

Browse files
jinxingcloud-fan
authored andcommitted
[SPARK-21343] Refine the document for spark.reducer.maxReqSizeShuffleToMem.
## What changes were proposed in this pull request? In current code, reducer can break the old shuffle service when `spark.reducer.maxReqSizeShuffleToMem` is enabled. Let's refine document. Author: jinxing <[email protected]> Closes #18566 from jinxing64/SPARK-21343.
1 parent 9131bdb commit 062c336

File tree

2 files changed

+14
-2
lines changed

2 files changed

+14
-2
lines changed

core/src/main/scala/org/apache/spark/internal/config/package.scala

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -323,9 +323,11 @@ package object config {
323323

324324
private[spark] val REDUCER_MAX_REQ_SIZE_SHUFFLE_TO_MEM =
325325
ConfigBuilder("spark.reducer.maxReqSizeShuffleToMem")
326-
.internal()
327326
.doc("The blocks of a shuffle request will be fetched to disk when size of the request is " +
328-
"above this threshold. This is to avoid a giant request takes too much memory.")
327+
"above this threshold. This is to avoid a giant request takes too much memory. We can " +
328+
"enable this config by setting a specific value(e.g. 200m). Note that this config can " +
329+
"be enabled only when the shuffle shuffle service is newer than Spark-2.2 or the shuffle" +
330+
" service is disabled.")
329331
.bytesConf(ByteUnit.BYTE)
330332
.createWithDefault(Long.MaxValue)
331333

docs/configuration.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -528,6 +528,16 @@ Apart from these, the following properties are also available, and may be useful
528528
By allowing it to limit the number of fetch requests, this scenario can be mitigated.
529529
</td>
530530
</tr>
531+
<tr>
532+
<td><code>spark.reducer.maxReqSizeShuffleToMem</code></td>
533+
<td>Long.MaxValue</td>
534+
<td>
535+
The blocks of a shuffle request will be fetched to disk when size of the request is above
536+
this threshold. This is to avoid a giant request takes too much memory. We can enable this
537+
config by setting a specific value(e.g. 200m). Note that this config can be enabled only when
538+
the shuffle shuffle service is newer than Spark-2.2 or the shuffle service is disabled.
539+
</td>
540+
</tr>
531541
<tr>
532542
<td><code>spark.shuffle.compress</code></td>
533543
<td>true</td>

0 commit comments

Comments
 (0)