[SPARK-31831][SQL][TESTS] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl #29039

HeartSaVioR · 2020-07-08T12:40:37Z

What changes were proposed in this pull request?

This patch changes the HiveSessionImplSuite to mock instances "before" initializing HiveSessionImpl, to avoid possible classloader issue.

Why are the changes needed?

The failures of HiveSessionImplSuite always come from classloader issue. While I don't have clear idea what is happening, there's no part possibly dealing with classloader, except initializing HiveSessionImpl. We can move the mock initializations earlier than initialing HiveSessionImpl so that it can avoid possible classloader issue.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Verified with multiple triggers of Jenkins builds

…than initializing HiveSessionImpl

HeartSaVioR · 2020-07-08T12:47:00Z

Rationalization: the failures always come from classloader issue;

sbt.ForkMain$ForkError: org.mockito.exceptions.base.MockitoException: 
ClassCastException occurred while creating the mockito mock :
  class to mock : 'org.apache.hive.service.cli.session.SessionManager', loaded by classloader : 'sun.misc.Launcher$AppClassLoader@2d6d8735'
  created class : 'org.mockito.codegen.SessionManager$MockitoMock$2028320452', loaded by classloader : 'net.bytebuddy.dynamic.loading.MultipleParentClassLoader@4058f9df'
  proxy instance class : 'org.mockito.codegen.SessionManager$MockitoMock$2028320452', loaded by classloader : 'net.bytebuddy.dynamic.loading.MultipleParentClassLoader@4058f9df'
  instance creation by : ObjenesisInstantiator

You might experience classloading issues, please ask the mockito mailing-list.

	at org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.beforeAll(HiveSessionImplSuite.scala:44)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:59)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:317)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:510)
	at sbt.ForkMain$Run$2.call(ForkMain.java:296)
	at sbt.ForkMain$Run$2.call(ForkMain.java:286)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: sbt.ForkMain$ForkError: java.lang.ClassCastException: org.mockito.codegen.SessionManager$MockitoMock$2028320452 cannot be cast to org.mockito.internal.creation.bytebuddy.MockAccess
	at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:48)
	at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
	at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
	at org.mockito.internal.MockitoCore.mock(MockitoCore.java:63)
	at org.mockito.Mockito.mock(Mockito.java:1908)
	at org.mockito.Mockito.mock(Mockito.java:1817)
	... 13 more

while I don't have clear idea what is happening, there's no part possibly dealing with classloader, except initializing HiveSessionImpl (only suspect, no clear evidence yet). This patch is to verify the idea that mocking instances before initializing HiveSessionImpl would avoid the classloader issue.

It's less flaky then CliSuite and only checked once per build, so to verify multiple times I just create 9 duplicated suites to test 10 times per build.

HeartSaVioR · 2020-07-08T12:48:34Z

retest this, please

HeartSaVioR · 2020-07-08T12:54:39Z

retest this, please

HeartSaVioR · 2020-07-08T13:09:15Z

retest this, please

SparkQA · 2020-07-08T13:28:57Z

Test build #125365 has finished for PR 29039 at commit 3d0d7e8.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-08T13:53:46Z

Test build #125363 has finished for PR 29039 at commit 3d0d7e8.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-08T14:09:07Z

retest this, please

SparkQA · 2020-07-08T14:16:50Z

Test build #125371 has started for PR 29039 at commit 9c48467.

HeartSaVioR · 2020-07-08T14:24:10Z

retest this, please

SparkQA · 2020-07-08T14:31:52Z

Test build #125375 has started for PR 29039 at commit 9c48467.

SparkQA · 2020-07-08T14:41:15Z

Test build #125361 has finished for PR 29039 at commit 3d0d7e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-08T18:52:15Z

Test build #125368 has finished for PR 29039 at commit 9c48467.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-08T22:59:40Z

Summary of 4 builds:

125361 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125361/testReport/)

HiveSessionImplSuite(s) passed

125368 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125368/testReport/)

HiveSessionImplSuite(s) passed

125371 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125371/testReport/)

HiveSessionImplSuite(s) passed

125375 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125375/testReport/)

HiveSessionImplSuite(s) passed

I'll trigger more builds to confirm, and once the new builds also pass HiveSessionImplSuite(s), turn the PR to be ready to review.

HeartSaVioR · 2020-07-08T23:00:18Z

retest this, please

HeartSaVioR · 2020-07-08T23:03:16Z

retest this, please

HeartSaVioR · 2020-07-08T23:21:43Z

retest this, please

HeartSaVioR · 2020-07-08T23:23:08Z

retest this, please

HeartSaVioR · 2020-07-08T23:26:54Z

retest this, please

HeartSaVioR · 2020-07-08T23:46:58Z

retest this, please

HeartSaVioR · 2020-07-08T23:53:32Z

retest this, please

SparkQA · 2020-07-08T23:54:43Z

Test build #125405 has finished for PR 29039 at commit 9c48467.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T00:44:05Z

Test build #125410 has finished for PR 29039 at commit 9c48467.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T00:52:15Z

Test build #125416 has finished for PR 29039 at commit 9c48467.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T01:09:58Z

Test build #125412 has finished for PR 29039 at commit 9c48467.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T01:33:27Z

Test build #125420 has finished for PR 29039 at commit 9c48467.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-09T12:38:02Z

Another summary for next set of builds

125405 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125405/testReport/)

All suites passed

125410 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125410/testReport/)

Lots of tests in CliSuite failed - #29036 is to fix flakiness, so please cross-check this and #29036 together

125416 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125416/testReport/)

All suites passed

125412 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125412/testReport/)

Lots of tests in CliSuite failed

125420 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125420/testReport/)

Lots of tests in CliSuite failed

This reverts commit 9c48467.

HeartSaVioR · 2020-07-09T13:13:41Z

cc. @dongjoon-hyun @alismess-db

srowen

Likewise looks good if it passes and fixes the issue.

SparkQA · 2020-07-09T14:17:10Z

Test build #125473 has finished for PR 29039 at commit 9b22750.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T14:36:07Z

Test build #125475 has finished for PR 29039 at commit 8e083c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-09T21:30:00Z

Thank you for pinging me, @HeartSaVioR .

dongjoon-hyun

+1, LGTM. Thank you, @HeartSaVioR and @srowen .
Merged to master because the affected version of SPARK-31831 is only 3.1.0.

HeartSaVioR · 2020-07-09T22:45:23Z

Thanks for reviewing and merging! We can port back anytime when we find the flakiness in other branches, so it should be OK to start with only master branch.

WIP HiveSessionImplSuite flakiness fix via mocking instances earlier …

3d0d7e8

…than initializing HiveSessionImpl

probot-autolabeler bot added the SQL label Jul 8, 2020

HeartSaVioR changed the title ~~[WIP] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl~~ [WIP][SPARK-XXXXX] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl Jul 8, 2020

HeartSaVioR marked this pull request as draft July 8, 2020 12:40

temporarily disable building doc

9c48467

probot-autolabeler bot added the BUILD label Jul 8, 2020

HeartSaVioR mentioned this pull request Jul 9, 2020

[SPARK-32242][SQL] CliSuite flakiness fix via differentiating cli driver bootup timeout and query execution timeout #29036

Closed

Revert "temporarily disable building doc"

9b22750

This reverts commit 9c48467.

HeartSaVioR changed the title ~~[WIP][SPARK-XXXXX] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl~~ [SPARK-31831] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl Jul 9, 2020

HeartSaVioR marked this pull request as ready for review July 9, 2020 13:10

HeartSaVioR changed the title ~~[SPARK-31831] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl~~ [SPARK-31831][SQL] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl Jul 9, 2020

Remove suites on debug

8e083c2

srowen approved these changes Jul 9, 2020

View reviewed changes

dongjoon-hyun approved these changes Jul 9, 2020

View reviewed changes

dongjoon-hyun closed this in ac6406e Jul 9, 2020

dongjoon-hyun changed the title ~~[SPARK-31831][SQL] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl~~ [SPARK-31831][SQL][TESTS] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl Jul 9, 2020

dongjoon-hyun added the TESTS label Jul 9, 2020

HeartSaVioR deleted the hive-session-impl-suite-flakiness-fix branch July 9, 2020 22:45

[SPARK-31831][SQL][TESTS] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl #29039

[SPARK-31831][SQL][TESTS] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl #29039

Uh oh!

Conversation

HeartSaVioR commented Jul 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

HeartSaVioR commented Jul 9, 2020

Uh oh!

HeartSaVioR commented Jul 9, 2020

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

dongjoon-hyun commented Jul 9, 2020

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HeartSaVioR commented Jul 9, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading