[SPARK-21530] Update description of spark.shuffle.maxChunksBeingTransferred. #18735
Conversation
|
Test build #79949 has finished for PR 18735 at commit
|
|
LGTM |
docs/configuration.md
Outdated
| <td>Long.MAX_VALUE</td> | ||
| <td> | ||
| The max number of chunks allowed to being transferred at the same time on shuffle service. | ||
| Note that new coming connections will be closed when the max number is hit. Client should |
There was a problem hiding this comment.
change to say: allowed to be transferred
new coming should be new incoming.
Perhaps we could clarify a bit because Spark has a built in retry mechanism. Could we rephrase to something like: The client will retry according to the shuffle retry configs (see spark.shuffle.io.maxRetries and spark.shuffle.io.retryWait), if those limits are reached the task will fail with fetch failure.
|
|
||
| /** | ||
| * The max number of chunks allowed to being transferred at the same time on shuffle service. | ||
| * Note that new coming connections will be closed when the max number is hit. Client should |
docs/configuration.md
Outdated
| The max number of chunks allowed to be transferred at the same time on shuffle service. | ||
| Note that new incoming connections will be closed when the max number is hit. The client will | ||
| retry according to the shuffle retry configs (see spark.shuffle.io.maxRetries and | ||
| spark.shuffle.io.retryWait), if those limits are reached the task will fail with fetch failure. |
There was a problem hiding this comment.
thanks for updating, sorry I forgot to ask you to put spark.shuffle.io.maxRetries and spark.shuffle.io.retryWait inside so they show up as references like in other places. See the description of
spark.shuffle.compress
as an example on how ti references other configs.
|
thanks for updating, minor formatting thing I forgot to mention before, otherwise looks good. |
|
Test build #79965 has finished for PR 18735 at commit
|
|
Test build #79966 has finished for PR 18735 at commit
|
|
@tgravescs |
|
Test build #79983 has finished for PR 18735 at commit
|
|
thanks, merging to master! |
|
thanks for fixing @jinxing64 |
4 participants
What changes were proposed in this pull request?
Update the description of
spark.shuffle.maxChunksBeingTransferredto include that the new coming connections will be closed when the max is hit and client should have retry mechanism.