[SPARK-25641] Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100

[redsanket]

What changes were proposed in this pull request?

We want to change the default percentage to 100 for spark.shuffle.server.chunkFetchHandlerThreadsPercent. The reason being
currently this is set to 0. Which means currently if server.ioThreads > 0, the default number of threads would be 2 * #cores instead of server.io.Threads. We want the default to server.io.Threads in case this is not set at all. Also here a default of 0 would also mean 2 * #cores

How was this patch tested?

Manual
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Please review http://spark.apache.org/contributing.html before opening a pull request.

[redsanket]

@tgravescs @abellina plz take a look thanks

[tgravescs]

ok to test

[abellina]

+1 pending tests

[SparkQA]

Test build #96943 has finished for PR 22628 at commit 0258d19.

• This patch fails Java style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #96951 has finished for PR 22628 at commit a95c608.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

+1

[srowen]

I'm not clear from the description what the issue is. I get that the number of threads is undesirable but can you illustrate with a clearer example?

[tgravescs]

the default of 0 gets you the netty default which is always 2 * the # of cores. It ignores if you set the io.serverThreads to set the # of shuffle threads. With a default of 100, it actually applies the setting for io.serverThreads or if you don't have it set then uses the 2 * # of cores. So having a default of 100 is better here.

io.serverThreads = 10 and spark.shuffle.server.chunkFetchHandlerThreadsPercent=100 then the # of threads here for chunked fetches is 10
io.serverThreads=10 and spark.shuffle.server.chunkFetchHandlerThreadsPercent=0 then the # of threads here for chunked fetches is 2*#cores

Its better to have a default of 100 to keep the current behavior.

[redsanket]

Thanks @tgravescs for explaining the issue

[tgravescs]

pulled into master, thanks @redsanket

[zsxwing]

Is it supposed to the flaky ChunkFetchIntegrationSuite? http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.network.ChunkFetchIntegrationSuite&test_name=fetchFileChunk

[tgravescs]

not sure what you are asking, if you are saying its flaky then file a jira and please cc me and Sanket.

The link above doesn't load for me, please give me permissions if its needed.