[SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via Thrift Server #25567

wangyum · 2019-08-23T09:27:41Z

What changes were proposed in this pull request?

This PR build a test framework that directly re-run all the tests in SQLQueryTestSuite via Thrift Server. But it's a little different from SQLQueryTestSuite:

Can not support UDF testing.
Can not support DESC command and SHOW command because SQLQueryTestSuite formatted the output.

When building this framework, found two bug:
SPARK-28624: make_date is inconsistent when reading from table
SPARK-28611: Histogram's height is different

found two features that ThriftServer can not support:
SPARK-28636: ThriftServer can not support decimal type with negative scale
SPARK-28637: ThriftServer can not support interval type

Also, found two inconsistent behavior:
SPARK-28620: Double type returned for float type in Beeline/JDBC
SPARK-28619: The golden result file is different when tested by bin/spark-sql

Why are the changes needed?

Improve the overall test coverage for Thrift Server.

Does this PR introduce any user-facing change?

No

How was this patch tested?

N/A

…in other jars.

wangyum · 2019-08-23T09:29:50Z

Please note that this PR need to test with test-hadoop3.2 and test-maven.

SparkQA · 2019-08-23T12:36:00Z

Test build #109639 has finished for PR 25567 at commit beafc57.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-23T12:51:01Z

retest this please

wangyum · 2019-08-23T13:00:57Z

sbt with hadoop-2.7 test results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109639/testReport/org.apache.spark.sql/SQLQueryTestSuite
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109639/testReport/org.apache.spark.sql.hive.thriftserver/ThriftServerQueryTestSuite

SparkQA · 2019-08-23T18:11:03Z

Test build #109641 has finished for PR 25567 at commit beafc57.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-23T20:59:16Z

retest this please

dongjoon-hyun · 2019-08-24T00:26:44Z

So, only test-hadoop3.2 and test-mave testing is remaining in this PR? For now, it looks working.

SparkQA · 2019-08-24T01:36:38Z

Test build #109657 has finished for PR 25567 at commit beafc57.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-24T01:58:52Z

Yes. It works. But pgSQL/text.sql failed:

- pgSQL/text.sql *** FAILED ***
  "-1		
  -2		
  -3		
  -4		
  -5		
  0		
  1	a	j
  2	ah	oj
  3	aho	hoj
  4	ahoj	ahoj
  5	ahoj	ahoj" did not contain "Exception" Exception did not match for query #19
  select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i, expected: -1		
  -2		
  -3		
  -4		
  -5		
  0		
  1	a	j
  2	ah	oj
  3	aho	hoj
  4	ahoj	ahoj
  5	ahoj	ahoj, but got: java.sql.SQLException
  org.apache.spark.sql.catalyst.parser.ParseException: 
  no viable alternative at input 'left'(line 1, pos 10)
  
  == SQL ==
  select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i
  ----------^^^ (ThriftServerQueryTestSuite.scala:178)

org.apache.spark.sql.catalyst.parser.ParseException: no viable alternative at input 'left'(line 1, pos 10) == SQL == select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i ----------^^^

SparkQA · 2019-08-24T12:51:33Z

Test build #109665 has finished for PR 25567 at commit 4aa5ee2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-24T12:57:07Z

retest this please

SparkQA · 2019-08-24T18:48:29Z

Test build #109666 has finished for PR 25567 at commit 4aa5ee2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-24T20:11:45Z

retest this please

SparkQA · 2019-08-25T02:01:00Z

Test build #109672 has finished for PR 25567 at commit 4aa5ee2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-08-25T05:02:10Z

Oh! @wangyum sorry I didn't know you opened a PR here.

SparkQA · 2019-08-25T19:37:39Z

Test build #109702 has finished for PR 25567 at commit 2254809.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-08-26T00:33:06Z

retest this please

SparkQA · 2019-08-26T05:23:46Z

Test build #109710 has finished for PR 25567 at commit 2254809.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-26T05:38:19Z

19/08/25 22:03:34 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 13470 with 5...500 worker threads
- subquery/exists-subquery/exists-basic.sql *** FAILED ***
  java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:13470: java.net.ConnectException: Connection refused (Connection refused)

wangyum · 2019-08-26T05:38:29Z

retest this please

SparkQA · 2019-08-26T06:19:22Z

Test build #109715 has finished for PR 25567 at commit 2254809.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-26T06:38:04Z

CoarseGrainedSchedulerBackendSuite:
- serialized task larger than max RPC message size
- compute max number of concurrent tasks can be launched
- compute max number of concurrent tasks can be launched when spark.task.cpus > 1
- compute max number of concurrent tasks can be launched when some executors are busy
- custom log url for Spark UI is applied
- extra resources from executor *** FAILED ***
  The code passed to eventually never returned normally. Attempted 322 times over 5.0078623669999995 seconds. Last failure message: ArrayBuffer("1", "3") did not equal Array("0", "1", "3"). (CoarseGrainedSchedulerBackendSuite.scala:249)

wangyum · 2019-08-26T06:38:13Z

retest this please

SparkQA · 2019-08-26T07:05:01Z

Test build #109718 has finished for PR 25567 at commit 2254809.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-26T07:09:34Z

retest this please

SparkQA · 2019-08-26T13:12:13Z

Test build #109722 has finished for PR 25567 at commit 2254809.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-08-26T13:38:47Z

Merged to master.

dongjoon-hyun · 2019-08-27T04:26:41Z

Hi, @wangyum and @HyukjinKwon .
Sorry, but #25587 failed two times consecutively due to ThriftServerQueryTestSuite. Since ThriftServerQueryTestSuite is irrelevant to that PR, it seems to be due to its flakiness. Could you take a look at that?

dongjoon-hyun · 2019-08-27T05:31:11Z

Also, the followings are the recent failures in master branch across various profiles.

In the PR builder, there will be more failures.

wangyum · 2019-08-27T05:32:40Z

Could we ignore this test on jenkins?

wangyum · 2019-08-27T05:49:03Z

I plan to ignore this test: #25592 We can run it manually.

dongjoon-hyun · 2019-08-27T05:56:29Z

Since I already reverted the first trial of your this PR, I'm honestly reluctant to revert this by myself again.

Let me ping PMCs since we need to fix this in anyway urgently.

Hi, @gatorsmile , @cloud-fan , @HyukjinKwon , @srowen . Could you make a decision to this?
This is committed 16 hours ago and already caused at least 7 failures in various profiles (including PR builders)

@wangyum is suggesting to add @Ignore tag. That would be one possible solution.
In that case, one thing I'm not sure is that who will fix this later.

HyukjinKwon · 2019-08-27T06:39:54Z

I think we can ignore them for now. Looks @wangyum going to fix the flakiness.

HyukjinKwon · 2019-08-27T06:40:31Z

I am going to merge #25592. If we prefer to revert them, we can revert both - I don't mind.

juliuszsompolski · 2019-08-27T14:12:40Z

...erver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala

+    (1 to 3).foldLeft(Try(startThriftServer(listeningPort, 0))) { case (started, attempt) =>
+      started.orElse {
+        listeningPort += 1
+        Try(startThriftServer(listeningPort, attempt))


@wangyum
I think this is the cause of flakiness.

This goes down to https://github.com/apache/spark/blob/master/sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java#L178, which starts the service in an asynchronous background thread. It can race and try to connect in the test before the server actually gets initialized.
Moving it to beforeAll() and adding a few seconds sleep should help.
Or we could transplant the whole way from HiveThriftServer2Test that starts uses the start-thriftserver.sh script and monitors it's log until its started. Would it be possible to make HiveThriftServer2Test a trait that can be mixed in here?

Thank you so much for the investigation, @juliuszsompolski !

Move it to beforeAll() also unstable: #25567 (comment)

Anyway, I will try to move it to beforeAll() again.

MaxGekk · 2019-11-15T16:48:43Z

...erver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala

+ *   2. Support DESC command.
+ *   3. Support SHOW command.
+ */
+class ThriftServerQueryTestSuite extends SQLQueryTestSuite {


I am trying to troubleshoot an jenkins build failure on this this, but the regular command which I use to run SQLQueryTestSuite

SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/test-only *ThriftServerQueryTestSuite -- -z interval.sql"

Could someone help me to run ThriftServerQueryTestSuite?

@wangyum, can you write some guides in the documentation about how we run each tests?

wangyum added 9 commits August 7, 2019 14:21

Directly re-run all the tests in SQLQueryTestSuite via Thrift Server

699b4db

avoid test with ConfigSets

908cdc3

Merge remote-tracking branch 'upstream/master' into SPARK-28527

66e9b25

Add ThriftServerQueryTestSuite to RunInTheirOwnDedicatedJvm

3f21189

Move startThriftServer from beforeEach to beforeAll

5802888

Merge remote-tracking branch 'upstream/master' into SPARK-28527

02c6911

Maven can not get the correct baseResourcePath

2823f34

Fix pgSQL/text.sql

4845088

Workaround Maven can't get correct resource directory when resources …

beafc57

…in other jars.

wangyum changed the title ~~[SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via Thrift Server~~ [SPARK-28527][SQL][TEST][test-maven] Re-run all the tests in SQLQueryTestSuite via Thrift Server Aug 23, 2019

dongjoon-hyun added SQL TESTS labels Aug 23, 2019

Fix no viable alternative:

4aa5ee2

org.apache.spark.sql.catalyst.parser.ParseException: no viable alternative at input 'left'(line 1, pos 10) == SQL == select i, left('ahoj', i), right('ahoj', i) from range(-5, 6) t(i) order by i ----------^^^

wangyum changed the title ~~[SPARK-28527][SQL][TEST][test-maven] Re-run all the tests in SQLQueryTestSuite via Thrift Server~~ [SPARK-28527][SQL][TEST][test-maven][test-hadoop3.2] Re-run all the tests in SQLQueryTestSuite via Thrift Server Aug 24, 2019

HyukjinKwon mentioned this pull request Aug 25, 2019

[SPARK-28527][SQL][TEST][test-maven] Re-run all the tests in SQLQueryTestSuite via Thrift Server #25574

Closed

HyukjinKwon changed the title ~~[SPARK-28527][SQL][TEST][test-maven][test-hadoop3.2] Re-run all the tests in SQLQueryTestSuite via Thrift Server~~ [SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via Thrift Server Aug 26, 2019

HyukjinKwon closed this in 6e12b58 Aug 26, 2019

wangyum deleted the SPARK-28527 branch August 26, 2019 14:14

dongjoon-hyun mentioned this pull request Aug 27, 2019

[SPARK-28877][PYSPARK][test-hadoop3.2][test-java11] Make jaxb-runtime compile-time dependency #25587

Closed

HyukjinKwon mentioned this pull request Aug 27, 2019

[SPARK-28527][SQL][TEST][FOLLOW-UP] Ignores Thrift server ThriftServerQueryTestSuite #25592

Closed

juliuszsompolski reviewed Aug 27, 2019

View reviewed changes

wangyum mentioned this pull request Oct 19, 2019

[SPARK-28527][SQL][TEST] Enable ThriftServerQueryTestSuite #25868

Closed

MaxGekk reviewed Nov 15, 2019

View reviewed changes

[SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via Thrift Server #25567

[SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via Thrift Server #25567

Uh oh!

Conversation

wangyum commented Aug 23, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

wangyum commented Aug 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Aug 23, 2019

Uh oh!

wangyum commented Aug 23, 2019

Uh oh!

wangyum commented Aug 23, 2019

Uh oh!

SparkQA commented Aug 23, 2019

Uh oh!

wangyum commented Aug 23, 2019

Uh oh!

dongjoon-hyun commented Aug 24, 2019

Uh oh!

SparkQA commented Aug 24, 2019

Uh oh!

wangyum commented Aug 24, 2019

Uh oh!

SparkQA commented Aug 24, 2019

Uh oh!

wangyum commented Aug 24, 2019

Uh oh!

SparkQA commented Aug 24, 2019

Uh oh!

wangyum commented Aug 24, 2019

Uh oh!

SparkQA commented Aug 25, 2019

Uh oh!

HyukjinKwon commented Aug 25, 2019

Uh oh!

SparkQA commented Aug 25, 2019

Uh oh!

HyukjinKwon commented Aug 26, 2019

Uh oh!

SparkQA commented Aug 26, 2019

Uh oh!

wangyum commented Aug 26, 2019

Uh oh!

wangyum commented Aug 26, 2019

Uh oh!

SparkQA commented Aug 26, 2019

Uh oh!

wangyum commented Aug 26, 2019

Uh oh!

wangyum commented Aug 26, 2019

Uh oh!

SparkQA commented Aug 26, 2019

Uh oh!

wangyum commented Aug 26, 2019

Uh oh!

SparkQA commented Aug 26, 2019

Uh oh!

HyukjinKwon commented Aug 26, 2019

Uh oh!

dongjoon-hyun commented Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangyum commented Aug 27, 2019

Uh oh!

wangyum commented Aug 27, 2019

Uh oh!

dongjoon-hyun commented Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Aug 27, 2019

Uh oh!

wangyum commented Aug 23, 2019 •

edited

Loading

dongjoon-hyun commented Aug 27, 2019 •

edited

Loading

dongjoon-hyun commented Aug 27, 2019 •

edited

Loading

dongjoon-hyun commented Aug 27, 2019 •

edited

Loading