[SPARK-25181][CORE] Limit Thread Pool size in BlockManager Master and Slave endpoints

[mukulmurthy]

What changes were proposed in this pull request?

Limit Thread Pool size in BlockManager Master and Slave endpoints.

Currently, BlockManagerMasterEndpoint and BlockManagerSlaveEndpoint both have thread pools with nearly unbounded (Integer.MAX_VALUE) numbers of threads. In certain cases, this can lead to driver OOM errors. This change limits the thread pools to 100 threads; this should not break any existing behavior because any tasks beyond that number will get queued.

How was this patch tested?

Manual testing

Please review http://spark.apache.org/contributing.html before opening a pull request.

[SparkQA]

Test build #95057 has finished for PR 22176 at commit c2223c3.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[mukulmurthy]

retest this please

[mukulmurthy]

Jenkins, retest this please

[SparkQA]

Test build #95066 has finished for PR 22176 at commit c2223c3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mukulmurthy]

@zsxwing , @JoshRosen , @marmbrus for review and merge.

[zsxwing]

LGTM. Merging to master.

[markhamstra]

@zsxwing we really should have considered whether this should be a configuration variable instead of a fixed number of threads in any environment.

[zsxwing]

@markhamstra That's a good point. However, since this is just following our current codes if you check the usages of newDaemonCachedThreadPool, and the changes here should be safe considering how we use them, I don't want to block this PR for this reason. We can open a new ticket to add configurations if that's necessary.

[markhamstra]

Yes, this is better than what we had, but maybe it can be better still.