[SPARK-31235][FOLLOWUP][TESTS][test-hadoop3.2] Fix test "specify a more specific type for the ap… #28456

wang-zhun · 2020-05-06T02:33:10Z

What changes were proposed in this pull request?

Update the input parameters for instantiating RMAppManager and ClientRMService

Why are the changes needed?

For hadoop3.2, if RMAppManager is not created correctly, the following exception will occur:

java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:135)
	at org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:55)
	at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.<init>(RMAppManager.java:117)

How was this patch tested?

UTs

…plication"

wang-zhun · 2020-05-06T02:35:44Z

test this please

viirya · 2020-05-06T02:35:49Z

ok to test

SparkQA · 2020-05-06T03:10:35Z

Test build #122330 has finished for PR 28456 at commit 1e415a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-06T03:36:02Z

Test build #122331 has finished for PR 28456 at commit 42f090c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-06T04:37:36Z

Test build #122337 has finished for PR 28456 at commit fd08a7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-06T12:24:37Z

Test build #122361 has finished for PR 28456 at commit 135ac1f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wang-zhun · 2020-05-06T12:41:28Z

@viirya @tgravescs How to specify "Hadoop profile hadoop3.2" to perform the test

tgravescs · 2020-05-06T13:22:26Z

test this please

SparkQA · 2020-05-06T13:53:10Z

Test build #122365 has finished for PR 28456 at commit 135ac1f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987

LGTM

tgravescs · 2020-05-07T15:19:49Z

test this please

SparkQA · 2020-05-07T15:43:15Z

Test build #122408 has finished for PR 28456 at commit 135ac1f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-07T19:04:08Z

Retest this please.

SparkQA · 2020-05-07T19:30:39Z

Test build #122412 has finished for PR 28456 at commit 135ac1f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-08T19:17:51Z

Hi, @tgravescs . Could you review this PR?

tgravescs

changes look fine to me, I was running the tests multiple times since this seemed to be intermittent failure

tgravescs · 2020-05-08T20:40:43Z

merging this to master

dongjoon-hyun · 2020-05-08T21:26:42Z

Thank you so much, @tgravescs !

dongjoon-hyun · 2020-05-14T21:15:15Z

Ur, in Maven environment, this seems to cause more UT failures in the other suite.

It seems that this have a side effect.

Test Result (11 failures / +7)
org.apache.spark.deploy.yarn.YarnClusterSuite.run Spark in yarn-client mode
org.apache.spark.deploy.yarn.YarnClusterSuite.run Spark in yarn-cluster mode with different configurations, ensuring redaction
org.apache.spark.deploy.yarn.YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630)
org.apache.spark.deploy.yarn.YarnClusterSuite.run Spark in yarn-cluster mode with additional jar
org.apache.spark.deploy.yarn.YarnClusterSuite.run Python application in yarn-cluster mode
org.apache.spark.deploy.yarn.YarnClusterSuite.run Python application in yarn-cluster mode using spark.yarn.appMasterEnv to override local envvar
org.apache.spark.deploy.yarn.YarnClusterSuite.user class path first in cluster mode
org.apache.spark.deploy.yarn.YarnClusterSuite.executor env overwrite AM env in cluster mode
org.apache.spark.deploy.yarn.YarnShuffleAuthSuite.external shuffle service
org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite.external shuffle service

dongjoon-hyun · 2020-05-14T21:22:26Z

Although this is the first one hitting them, this may not be the root cause.

wang-zhun · 2020-05-15T01:52:21Z

Thank you @dongjoon-hyun , it may be caused by me, I will investigate this reason

dongjoon-hyun · 2020-05-15T02:12:48Z

Thank you so much, @wang-zhun .

dongjoon-hyun · 2020-05-15T17:32:55Z

Unfortunately, this test case follow-up broadens the failure surface to the other test suites, YarnClusterSuite, YarnShuffleAuthSuite, and YarnShuffleIntegrationSuite.

Definitely, this seems that some uncleaned YarnClient issue, but it's not clear to me. I'll revert this first to narrow down the failure this specific test case only.

After that, if the original test case, specify a more specific type for ... , is not fixed in a short term, we may consider another approaches (ignoring this in Hadoop 3.2 or reverting the original commit).

Sorry guys.

dongjoon-hyun · 2020-05-15T17:33:59Z

For the follow-up PR, please use [test-maven][test-hadoop3.2] in the PR title, @wang-zhun .

tgravescs · 2020-05-15T17:51:27Z

that's fine, if its causing problems lets revert the test. We can do some other more basic tests just to make sure value passed as expected and perhaps add something in the Cluster Suite

dongjoon-hyun · 2020-05-16T04:26:28Z

Hi, All. I made a follow-up PR. Until now, it looks promising.

[SPARK-31235][TESTS][FOLLOWUP][test-maven][test-hadoop3.2] Fix Hadoop-3.2 profile UT failure #28550

dongjoon-hyun · 2020-05-16T14:58:13Z

Inevitably, the test case will be disabled temporarily on Hadoop 3.2 profile only. We will keep this test coverage in Hadoop 2.7.

[SPARK-31235][TESTS][FOLLOWUP] Disable test case specify a more specific type in Hadoop-3.2 #28552

…ific type` in Hadoop-3.2 ### What changes were proposed in this pull request? This PR aims to recover Hadoop-3.2 profile jobs on `master` branch by disabling a UT added by SPARK-31235 in Hadoop 3.2 temporarily. The target UT is not a flaky test. It always fail on Hadoop-3.2 profile currently although it works in Hadoop 2.7 profile. So, in this PR, we keep the test coverage in Hadoop 2.7 and ignore the test in Hadoop 3.2 temporarily to unblock the other PRs. ### Why are the changes needed? SPARK-31235 added a test case which is breaking Hadoop 3.2 and there are two follow-up to fix it. Although two follow-ups can fix the UT in Hadoop 3.2 environment. The side-effect on Hadoop classes cause some random UT failures in the other suites. - #28456 - #28550 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins with SBT/Maven. Closes #28552 from dongjoon-hyun/SPARK-31235-2. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

wang-zhun · 2020-05-18T01:47:29Z

Thank you @dongjoon-hyun

[SPARK-31235][YARN] Fix test "specify a more specific type for the ap…

1e415a9

…plication"

probot-autolabeler bot added the YARN label May 6, 2020

wang-zhun mentioned this pull request May 6, 2020

[SPARK-31235][YARN] Separates different categories of applications #28009

Closed

fix

42f090c

fix

fd08a7a

viirya mentioned this pull request May 6, 2020

[SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor #26624

Closed

fix hadoop-3.2 exception

135ac1f

tgravescs changed the title ~~[SPARK-31235][YARN] Fix test "specify a more specific type for the ap…~~ [SPARK-31235][YARN][test-hadoop3.2] Fix test "specify a more specific type for the ap… May 6, 2020

jiangxb1987 approved these changes May 6, 2020

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-31235][YARN][test-hadoop3.2] Fix test "specify a more specific type for the ap…~~ [SPARK-31235][FOLLOWUP][TESTS][test-hadoop3.2] Fix test "specify a more specific type for the ap… May 7, 2020

dongjoon-hyun added the TESTS label May 7, 2020

tgravescs approved these changes May 8, 2020

View reviewed changes

asfgit closed this in c1801fd May 8, 2020

dongjoon-hyun mentioned this pull request May 16, 2020

[SPARK-31235][TESTS][FOLLOWUP] Disable test case specify a more specific type in Hadoop-3.2 #28552

Closed

[SPARK-31235][FOLLOWUP][TESTS][test-hadoop3.2] Fix test "specify a more specific type for the ap… #28456

[SPARK-31235][FOLLOWUP][TESTS][test-hadoop3.2] Fix test "specify a more specific type for the ap… #28456

Uh oh!

Conversation

wang-zhun commented May 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Uh oh!

wang-zhun commented May 6, 2020

Uh oh!

viirya commented May 6, 2020

Uh oh!

SparkQA commented May 6, 2020

Uh oh!

SparkQA commented May 6, 2020

Uh oh!

SparkQA commented May 6, 2020

Uh oh!

SparkQA commented May 6, 2020

Uh oh!

wang-zhun commented May 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgravescs commented May 6, 2020

Uh oh!

SparkQA commented May 6, 2020

Uh oh!

jiangxb1987 left a comment

Choose a reason for hiding this comment

Uh oh!

tgravescs commented May 7, 2020

Uh oh!

SparkQA commented May 7, 2020

Uh oh!

dongjoon-hyun commented May 7, 2020

Uh oh!

SparkQA commented May 7, 2020

Uh oh!

dongjoon-hyun commented May 8, 2020

Uh oh!

tgravescs left a comment

Choose a reason for hiding this comment

Uh oh!

tgravescs commented May 8, 2020

Uh oh!

dongjoon-hyun commented May 8, 2020

Uh oh!

dongjoon-hyun commented May 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 14, 2020

Uh oh!

wang-zhun commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

tgravescs commented May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 16, 2020

Uh oh!

dongjoon-hyun commented May 16, 2020

Uh oh!

wang-zhun commented May 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wang-zhun commented May 6, 2020 •

edited

Loading

wang-zhun commented May 6, 2020 •

edited

Loading

dongjoon-hyun commented May 14, 2020 •

edited

Loading

tgravescs commented May 15, 2020 •

edited

Loading