Skip to content

Commit 8a8b373

Browse files
committed
Address Patrick's comments
1 parent 3b1f84e commit 8a8b373

File tree

1 file changed

+2
-20
lines changed

1 file changed

+2
-20
lines changed

docs/configuration.md

Lines changed: 2 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -866,33 +866,15 @@ Apart from these, the following properties are also available, and may be useful
866866
<td>1</td>
867867
<td>
868868
(Netty only) Connections between hosts are reused in order to reduce connection buildup for
869-
large clusters. For small clusters with many hard disks, this may result in insufficient
869+
large clusters. For clusters with many hard disks and few hosts, this may result in insufficient
870870
concurrency to saturate all disks, and so users may consider increasing this value.
871871
</td>
872872
</tr>
873-
<tr>
874-
<td><code>spark.shuffle.io.serverThreads</code></td>
875-
<td>min(num-cores, 8)</td>
876-
<td>
877-
(Netty only) Size of fixed pool of threads to make requests. As each thread gets 16 MB of
878-
off-heap buffer space initially, the default is limited to 8 cores to avoid excessive off-heap
879-
allocation. It has been experimentally determined that this default should be sufficient to
880-
saturate a 10Gb/s network, but in some cases this may have to be adjusted.
881-
</td>
882-
</tr>
883-
<tr>
884-
<td><code>spark.shuffle.io.clientThreads</code></td>
885-
<td>min(num-cores, 8)</td>
886-
<td>
887-
(Netty only) Size of fixed pool of threads to make requests. This has the same properties as
888-
<code>spark.shuffle.io.serverThreads</code>, above.
889-
</td>
890-
</tr>
891873
<tr>
892874
<td><code>spark.shuffle.io.maxRetries</code></td>
893875
<td>3</td>
894876
<td>
895-
(Netty only) Fetches that fail due to IO-related exceptions are automatically retried if this is
877+
(Netty only) Fetches that fail due to IO-related exceptions are automatically retried if this is
896878
set to a non-zero value. This retry logic helps stabilize large shuffles in the face of long GC
897879
pauses or transient network connectivity issues.
898880
</td>

0 commit comments

Comments
 (0)