[SPARK-28527][SQL][TEST][test-maven] Re-run all the tests in SQLQueryTestSuite via Thrift Server #25574
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…a Thrift Server ## What changes were proposed in this pull request? This PR build a test framework that directly re-run all the tests in `SQLQueryTestSuite` via Thrift Server. But it's a little different from `SQLQueryTestSuite`: 1. Can not support [UDF testing](https://github.com/apache/spark/blob/44e607e9213bdceab970606fb15292db2fe157c2/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala#L293-L297). 2. Can not support `DESC` command and `SHOW` command because `SQLQueryTestSuite` [formatted the output](https://github.com/apache/spark/blob/1882912cca4921d3d8c8632b3bb34e69e8119791/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala#L38-L50.). When building this framework, found two bug: [SPARK-28624](https://issues.apache.org/jira/browse/SPARK-28624): `make_date` is inconsistent when reading from table [SPARK-28611](https://issues.apache.org/jira/browse/SPARK-28611): Histogram's height is different found two features that ThriftServer can not support: [SPARK-28636](https://issues.apache.org/jira/browse/SPARK-28636): ThriftServer can not support decimal type with negative scale [SPARK-28637](https://issues.apache.org/jira/browse/SPARK-28637): ThriftServer can not support interval type Also, found two inconsistent behavior: [SPARK-28620](https://issues.apache.org/jira/browse/SPARK-28620): Double type returned for float type in Beeline/JDBC [SPARK-28619](https://issues.apache.org/jira/browse/SPARK-28619): The golden result file is different when tested by `bin/spark-sql` ## How was this patch tested? N/A Closes apache#25373 from wangyum/SPARK-28527. Authored-by: Yuming Wang <[email protected]> Signed-off-by: gatorsmile <[email protected]>
HyukjinKwon
changed the title
Member
Author
|
retest this please |
Member
Author
|
Test build #109687 has finished for PR 25574 at commit
|
Member
|
OK. Thank you @HyukjinKwon |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR is a retry of #25373. The problem was found at #25373 (comment) due to different resource paths during the tests.
It works around just simply by always using sql-tests from the source.
Note that Seems similar way is already used when SPARK_GENERATE_GOLDEN_FILES is enabled, and referring
spark.homeproperty orSPARK_HOMEis a proper way to detect some paths too.See the original PR description:
This PR build a test framework that directly re-run all the tests in
SQLQueryTestSuitevia Thrift Server. But it's a little different fromSQLQueryTestSuite:DESCcommand andSHOWcommand becauseSQLQueryTestSuiteformatted the output.When building this framework, found two bug:
SPARK-28624:
make_dateis inconsistent when reading from tableSPARK-28611: Histogram's height is different
found two features that ThriftServer can not support:
SPARK-28636: ThriftServer can not support decimal type with negative scale
SPARK-28637: ThriftServer can not support interval type
Also, found two inconsistent behavior:
SPARK-28620: Double type returned for float type in Beeline/JDBC
SPARK-28619: The golden result file is different when tested by
bin/spark-sqlWhy are the changes needed?
To improve test coverage in thrift server and ensure same queries work.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manually tested via SBT and Maven roughly. Jenkins build is needed to verify if it works or not.