[SPARK-32242][SQL] CliSuite flakiness fix via differentiating cli driver bootup timeout and query execution timeout #29036

HeartSaVioR · 2020-07-08T08:38:07Z

What changes were proposed in this pull request?

This patch tries to mitigate the flakiness of CliSuite, via below changes:

differentiate cli driver boot-up timeout (2 mins) and query execution timeout (parameter)

Cli driver boot-up is determined by master and app ID message. Given spark-sql doesn't print the message if -e option is specified, the patch simply add 2 mins on timeout for the case to cover the boot-up timeout.

don't fail the test even spark-sql doesn't gracefully shut down in 1 min.
extend timeout for path command test in CliSuite

Why are the changes needed?

It took around 40 seconds for boot-up message (master: ... Application Id: ...) to be printed in stderr, while the overall timeout is 1 minute in many tests. This case the actual timeout for query execution is just 20 seconds, which may not be enough.

Some of the tests also failed with org.scalatest.exceptions.TestFailedException: spark-sql did not exit gracefully, which I don't feel the test has to be failed.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Verified with multiple triggers of Jenkins builds

HeartSaVioR · 2020-07-08T08:46:19Z

Rationalization: I've been looking into the recent failures of CliSuite, and realized it took around 40 seconds for bootup message (master: ... Application Id: ...) to be printed in stderr, while the overall timeout is 1 minute in many tests.

This PR is a POC to validate whether it helps if we can differentiate cli driver bootup timeout (with enough time to avoid failing on slow env) and query execution timeout. If this PR succeeds to pass the build (at least for CliSuite) 5 times sequentially, it becomes pretty much better than current.

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125152/testReport/junit/org.apache.spark.sql.hive.thriftserver/CliSuite/history

You'll be surprised if you go backward and see the occurrence of failures, and agree that 5 sequential builds passing can be the end condition.

HeartSaVioR · 2020-07-08T08:46:29Z

retest this, please

HeartSaVioR · 2020-07-08T08:59:53Z

retest this, please

SparkQA · 2020-07-08T09:09:17Z

Test build #125339 has finished for PR 29036 at commit ab1acf6.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-08T09:11:36Z

I’ll temporarily disable gendoc here soon to unblock experiments.

SparkQA · 2020-07-08T09:50:19Z

Test build #125338 has finished for PR 29036 at commit ab1acf6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-08T10:08:54Z

Test build #125341 has finished for PR 29036 at commit ab1acf6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…and query execution timeout

HeartSaVioR · 2020-07-08T11:03:36Z

retest this, please

HeartSaVioR · 2020-07-08T11:26:52Z

retest this, please

SparkQA · 2020-07-08T11:32:16Z

Test build #125350 has started for PR 29036 at commit 3f166b8.

HeartSaVioR · 2020-07-08T11:34:01Z

retest this, please

SparkQA · 2020-07-08T11:39:37Z

Test build #125351 has started for PR 29036 at commit 3f166b8.

HeartSaVioR · 2020-07-08T11:42:53Z

retest this, please

SparkQA · 2020-07-08T11:47:21Z

Test build #125353 has started for PR 29036 at commit 3f166b8.

SparkQA · 2020-07-08T15:49:42Z

Test build #125347 has finished for PR 29036 at commit 3f166b8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-08T18:28:21Z

Test build #125348 has finished for PR 29036 at commit 3f166b8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-08T22:47:25Z

Summary of the 5 builds:

125347 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125347/testReport/)

All tests in CliSuite passed.

125348 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125348/testReport/)

A bunch of tests in CliSuite failed, though looks like the build was super slow, bunch of other suites also failed as well.

125350 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125350/testReport/)

Only one test in CliSuite failed, and it wasn't about timeout - org.scalatest.exceptions.TestFailedException: spark-sql did not exit gracefully.. I wonder we really have to fail the test for the case.

125351 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125351/testReport/)

All tests in CliSuite passed.

125353 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125353/testReport/)

Except the failures of org.scalatest.exceptions.TestFailedException: spark-sql did not exit gracefully., only one test failed (path command), and boot timeout actually helped to run the query. Just because query execution timeout was short.

Looks like this fix helps mitigating the flakiness. Let me update the patch to not fail the query for spark-sql did not exit gracefully, and give more time on path command.

…own on spark-sql

HeartSaVioR · 2020-07-08T22:56:13Z

retest this, please

HeartSaVioR · 2020-07-08T23:03:24Z

retest this, please

HeartSaVioR · 2020-07-08T23:11:40Z

retest this, please

HeartSaVioR · 2020-07-08T23:21:35Z

retest this, please

SparkQA · 2020-07-08T23:43:50Z

Test build #125404 has finished for PR 29036 at commit 0f3384a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-08T23:46:24Z

retest this, please

SparkQA · 2020-07-08T23:48:25Z

Test build #125406 has finished for PR 29036 at commit 0f3384a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T00:01:31Z

Test build #125407 has finished for PR 29036 at commit 0f3384a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T00:10:20Z

Test build #125409 has finished for PR 29036 at commit 0f3384a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T00:51:34Z

Test build #125417 has finished for PR 29036 at commit 0f3384a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2020-07-09T10:45:13Z

Another summary for next set of builds

125404 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125404/testReport/)

All suites passed

125406 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125406/testReport/)

All suites passed

125407 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125407/testReport/)

All suites passed

125409 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125409/testReport/)

Only one suite failed - org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)

#29039 is to fix flakiness, so please cross-check this and #29039 together

125417 (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125417/testReport/)

All suites passed

This reverts commit 3f166b8.

HeartSaVioR · 2020-07-09T13:00:39Z

cc. @cloud-fan @HyukjinKwon

srowen

If it passes, looks good.

SparkQA · 2020-07-09T14:04:34Z

Test build #125471 has finished for PR 29036 at commit 1802d60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-09T15:06:13Z

@yaooqinn can you take a look?

yaooqinn · 2020-07-10T02:49:33Z

Thanks for ping me @cloud-fan,

"It took around 40 seconds for boot-up" a local mode backend with UI disabled sounds a bit weird to me.

Spark SQL CLI command line: ../../bin/spark-sql --master local \
--driver-java-options -Dderby.system.durability=test \
--driver-class-path /home/jenkins/workspace/SparkPullRequestBuilder@3/sql/hive-thriftserver/src/test/noclasspath \
--conf spark.ui.enabled=false \
--hiveconf javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/home/jenkins/workspace/SparkPullRequestBuilder@3/target/tmp/spark-64c2cb57-c790-4fa1-a682-92f54805fbe3;create=true \
--hiveconf hive.exec.scratchdir=/home/jenkins/workspace/SparkPullRequestBuilder@3/target/tmp/spark-46cc494d-0aa3-455f-aeb8-7232c7634a1a \
--hiveconf conf1=conftest \
--hiveconf conf2=1
Exception: java.util.concurrent.TimeoutException: Futures timed out after [1 minute]

But I am also +1 with this approach.

HeartSaVioR · 2020-07-10T04:11:38Z

Thanks for the feedbacks! Merging to master. As same as #29039, we can port back anytime when we find the flakiness in other branches, so it should be OK to start with only master branch.

probot-autolabeler bot added the SQL label Jul 8, 2020

HeartSaVioR marked this pull request as draft July 8, 2020 09:14

HeartSaVioR added 2 commits July 8, 2020 19:30

WIP CliSuite flakiness fix - differentiate cli driver bootup timeout …

bf3fc01

…and query execution timeout

temporarily disable building doc

3f166b8

HeartSaVioR force-pushed the clisuite-flakiness-fix branch from ab1acf6 to 3f166b8 Compare July 8, 2020 10:41

probot-autolabeler bot added the BUILD label Jul 8, 2020

extend query timeout, avoid failing tests on fail to gracefully shutd…

0f3384a

…own on spark-sql

HeartSaVioR mentioned this pull request Jul 9, 2020

[SPARK-31831][SQL][TESTS] HiveSessionImplSuite flakiness fix via mocking instances earlier than initializing HiveSessionImpl #29039

Closed

HeartSaVioR changed the title ~~[WIP][SPARK-XXXXX][SQL] CliSuite flakiness fix via differentiating cli driver bootup timeout and query execution timeout~~ [SPARK-32242][SQL] CliSuite flakiness fix via differentiating cli driver bootup timeout and query execution timeout Jul 9, 2020

HeartSaVioR marked this pull request as ready for review July 9, 2020 12:57

Revert "temporarily disable building doc"

1802d60

This reverts commit 3f166b8.

srowen approved these changes Jul 9, 2020

View reviewed changes

HyukjinKwon approved these changes Jul 10, 2020

View reviewed changes

HeartSaVioR closed this in e6e43cb Jul 10, 2020

HeartSaVioR deleted the clisuite-flakiness-fix branch July 10, 2020 04:13

[SPARK-32242][SQL] CliSuite flakiness fix via differentiating cli driver bootup timeout and query execution timeout #29036

[SPARK-32242][SQL] CliSuite flakiness fix via differentiating cli driver bootup timeout and query execution timeout #29036

Uh oh!

Conversation

HeartSaVioR commented Jul 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HeartSaVioR commented Jul 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

HeartSaVioR commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

HeartSaVioR commented Jul 9, 2020

Uh oh!

HeartSaVioR commented Jul 9, 2020

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

cloud-fan commented Jul 9, 2020

Uh oh!

yaooqinn commented Jul 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HeartSaVioR commented Jul 8, 2020 •

edited

Loading

HeartSaVioR commented Jul 8, 2020 •

edited

Loading

yaooqinn commented Jul 10, 2020 •

edited

Loading