[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming

[liupc]

What changes were proposed in this pull request?

Currently, when enabled streaming dynamic allocation for streaming applications, the maxNumExecutorFailures in ApplicationMaster is still computed with spark.dynamicAllocation.maxExecutors.

Actually, we should consider spark.streaming.dynamicAllocation.maxExecutors instead.

Related codes:

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala

Line 101 in f87153a

private val maxNumExecutorFailures = {

How was this patch tested?

NA

Please review http://spark.apache.org/contributing.html before opening a pull request.

[srowen]

For other reviewers -- this is the fix itself.

[SparkQA]

Test build #4565 has finished for PR 23845 at commit d5a5107.

• This patch fails to build.
• This patch does not merge cleanly.
• This patch adds no public classes.

[srowen]

@liupc could you have a look at the build failure?

[SparkQA]

Test build #4575 has finished for PR 23845 at commit 155f284.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[liupc]

retest this please!

[liupc]

@srowen UT seems failed with some unrelated tests, can you help to retest this please?

[SparkQA]

Test build #4583 has finished for PR 23845 at commit 155f284.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[liupc]

retest this please

[liupc]

retest this please

[liupc]

cc @HyukjinKwon @srowen

[SparkQA]

Test build #4598 has started for PR 23845 at commit f825118.

[SparkQA]

Test build #4603 has finished for PR 23845 at commit f825118.

• This patch fails Java style tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4608 has finished for PR 23845 at commit f825118.

• This patch fails Java style tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[srowen]

Hm, that's weird. The failure is not directly related, and I thought we fixed this in #23887 Not sure what's going on. dev/lint-java and dev/sbt-checkstyle pass locally for master too, and this isn't failing other PR builds. Maybe some odd cached artifacts on a Jenkins build server? no idea.

[SparkQA]

Test build #4609 has started for PR 23845 at commit f825118.

[SparkQA]

Test build #4612 has finished for PR 23845 at commit f825118.

• This patch fails Java style tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[vanzin]

I'm really not familiar with streaming dynamic allocation, but I assume you can't enable both that and regular dynamic allocation together?

If for some reason that is allowed, then the fix should take the max of both values.

To fix the tests, probably needs a merge with master. Maybe github is confused.

[vanzin]

Could you add these to a new Streaming.scala instead? We shouldn't be adding more stuff to package.scala.

[liupc]

@vanzin

If for some reason that is allowed, then the fix should take the max of both values.

This is not allowed to enable both of them. This check is done in the scheduler/ExecutorAllocationManager

spark/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala

Line 211 in 7beb464

throw new IllegalArgumentException(

I agree to put these configs to Streaming.scala, it's more clear. I will update.

[vanzin]

ok to test

[SparkQA]

Test build #103459 has finished for PR 23845 at commit 2f46380.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4625 has finished for PR 23845 at commit 2f46380.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

@liupc can you take a look at ...

[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/core/src/main/scala/org/apache/spark/util/Utils.scala:2495: not found: value STREAMING_DYN_ALLOCATION_ENABLED
[error]     val streamingDynamicAllocationEnabled = conf.get(STREAMING_DYN_ALLOCATION_ENABLED)
[error]                

[SparkQA]

Test build #103570 has finished for PR 23845 at commit a46d4d3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

merged to master