-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-19602][SQL][TESTS] Add tests for qualified column names #17067
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
5257983
Add tests to cover scenarios for the column resolution
skambha 685a53d
Add tests for global temp views
skambha 596b3e8
Add test with local view
skambha cdd1468
newline in end
skambha e95c33d
uppercase struct
skambha 715bc04
fix style and use ctas
skambha 040e7dd
Add column resolution tests to the SQLQueryTestSuite and also separat…
skambha feeb8a7
Remove the -ve cases from the SQLQueryTestSuite because the exprId ca…
skambha 129c0c4
Tests are moved to the SQLQueryTestSuite
skambha 4700f39
Remove empty lines
skambha 158016f
Remove unused import
skambha 1c5cc4b
Move view tests also to SQLQueryTestSuite
skambha 14ae125
Rebase with master
skambha 1accdcf
upper casing SQL
skambha 102e555
Add -ve tests to the SQLQueryTestSuite framework
skambha f01c884
This file isnt updated properly in Spark-19766
skambha 14fc6fd
Change to SQLQueryTestSuite to mask the id
skambha 5594eb0
Delete the ColumnResolutionSuite
skambha b2e411c
address comments
skambha File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
36 changes: 36 additions & 0 deletions
36
sql/core/src/test/resources/sql-tests/inputs/columnresolution-negative.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| -- Negative testcases for column resolution | ||
| CREATE DATABASE mydb1; | ||
| USE mydb1; | ||
| CREATE TABLE t1 USING parquet AS SELECT 1 AS i1; | ||
|
|
||
| CREATE DATABASE mydb2; | ||
| USE mydb2; | ||
| CREATE TABLE t1 USING parquet AS SELECT 20 AS i1; | ||
|
|
||
| -- Negative tests: column resolution scenarios with ambiguous cases in join queries | ||
| SET spark.sql.crossJoin.enabled = true; | ||
| USE mydb1; | ||
| SELECT i1 FROM t1, mydb1.t1; | ||
| SELECT t1.i1 FROM t1, mydb1.t1; | ||
| SELECT mydb1.t1.i1 FROM t1, mydb1.t1; | ||
| SELECT i1 FROM t1, mydb2.t1; | ||
| SELECT t1.i1 FROM t1, mydb2.t1; | ||
| USE mydb2; | ||
| SELECT i1 FROM t1, mydb1.t1; | ||
| SELECT t1.i1 FROM t1, mydb1.t1; | ||
| SELECT i1 FROM t1, mydb2.t1; | ||
| SELECT t1.i1 FROM t1, mydb2.t1; | ||
| SELECT db1.t1.i1 FROM t1, mydb2.t1; | ||
| SET spark.sql.crossJoin.enabled = false; | ||
|
|
||
| -- Negative tests | ||
| USE mydb1; | ||
| SELECT mydb1.t1 FROM t1; | ||
| SELECT t1.x.y.* FROM t1; | ||
| SELECT t1 FROM mydb1.t1; | ||
| USE mydb2; | ||
| SELECT mydb1.t1.i1 FROM t1; | ||
|
|
||
| -- reset | ||
| DROP DATABASE mydb1 CASCADE; | ||
| DROP DATABASE mydb2 CASCADE; |
25 changes: 25 additions & 0 deletions
25
sql/core/src/test/resources/sql-tests/inputs/columnresolution-views.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| -- Tests for qualified column names for the view code-path | ||
| -- Test scenario with Temporary view | ||
| CREATE OR REPLACE TEMPORARY VIEW view1 AS SELECT 2 AS i1; | ||
| SELECT view1.* FROM view1; | ||
| SELECT * FROM view1; | ||
| SELECT view1.i1 FROM view1; | ||
| SELECT i1 FROM view1; | ||
| SELECT a.i1 FROM view1 AS a; | ||
| SELECT i1 FROM view1 AS a; | ||
| -- cleanup | ||
| DROP VIEW view1; | ||
|
|
||
| -- Test scenario with Global Temp view | ||
| CREATE OR REPLACE GLOBAL TEMPORARY VIEW view1 as SELECT 1 as i1; | ||
| SELECT * FROM global_temp.view1; | ||
| -- TODO: Support this scenario | ||
| SELECT global_temp.view1.* FROM global_temp.view1; | ||
| SELECT i1 FROM global_temp.view1; | ||
| -- TODO: Support this scenario | ||
| SELECT global_temp.view1.i1 FROM global_temp.view1; | ||
| SELECT view1.i1 FROM global_temp.view1; | ||
| SELECT a.i1 FROM global_temp.view1 AS a; | ||
| SELECT i1 FROM global_temp.view1 AS a; | ||
| -- cleanup | ||
| DROP VIEW global_temp.view1; |
88 changes: 88 additions & 0 deletions
88
sql/core/src/test/resources/sql-tests/inputs/columnresolution.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| -- Tests covering different scenarios with qualified column names | ||
| -- Scenario: column resolution scenarios with datasource table | ||
| CREATE DATABASE mydb1; | ||
| USE mydb1; | ||
| CREATE TABLE t1 USING parquet AS SELECT 1 AS i1; | ||
|
|
||
| CREATE DATABASE mydb2; | ||
| USE mydb2; | ||
| CREATE TABLE t1 USING parquet AS SELECT 20 AS i1; | ||
|
|
||
| USE mydb1; | ||
| SELECT i1 FROM t1; | ||
| SELECT i1 FROM mydb1.t1; | ||
| SELECT t1.i1 FROM t1; | ||
| SELECT t1.i1 FROM mydb1.t1; | ||
|
|
||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t1.i1 FROM t1; | ||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t1.i1 FROM mydb1.t1; | ||
|
|
||
| USE mydb2; | ||
| SELECT i1 FROM t1; | ||
| SELECT i1 FROM mydb1.t1; | ||
| SELECT t1.i1 FROM t1; | ||
| SELECT t1.i1 FROM mydb1.t1; | ||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t1.i1 FROM mydb1.t1; | ||
|
|
||
| -- Scenario: resolve fully qualified table name in star expansion | ||
| USE mydb1; | ||
| SELECT t1.* FROM t1; | ||
| SELECT mydb1.t1.* FROM mydb1.t1; | ||
| SELECT t1.* FROM mydb1.t1; | ||
| USE mydb2; | ||
| SELECT t1.* FROM t1; | ||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t1.* FROM mydb1.t1; | ||
| SELECT t1.* FROM mydb1.t1; | ||
| SELECT a.* FROM mydb1.t1 AS a; | ||
|
|
||
| -- Scenario: resolve in case of subquery | ||
|
|
||
| USE mydb1; | ||
| CREATE TABLE t3 USING parquet AS SELECT * FROM VALUES (4,1), (3,1) AS t3(c1, c2); | ||
| CREATE TABLE t4 USING parquet AS SELECT * FROM VALUES (4,1), (2,1) AS t4(c2, c3); | ||
|
|
||
| SELECT * FROM t3 WHERE c1 IN (SELECT c2 FROM t4 WHERE t4.c3 = t3.c2); | ||
|
|
||
| -- TODO: Support this scenario | ||
| SELECT * FROM mydb1.t3 WHERE c1 IN | ||
| (SELECT mydb1.t4.c2 FROM mydb1.t4 WHERE mydb1.t4.c3 = mydb1.t3.c2); | ||
|
|
||
| -- Scenario: column resolution scenarios in join queries | ||
| SET spark.sql.crossJoin.enabled = true; | ||
|
|
||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t1.i1 FROM t1, mydb2.t1; | ||
|
|
||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t1.i1 FROM mydb1.t1, mydb2.t1; | ||
|
|
||
| USE mydb2; | ||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t1.i1 FROM t1, mydb1.t1; | ||
| SET spark.sql.crossJoin.enabled = false; | ||
|
|
||
| -- Scenario: Table with struct column | ||
| USE mydb1; | ||
| CREATE TABLE t5(i1 INT, t5 STRUCT<i1:INT, i2:INT>) USING parquet; | ||
| INSERT INTO t5 VALUES(1, (2, 3)); | ||
| SELECT t5.i1 FROM t5; | ||
| SELECT t5.t5.i1 FROM t5; | ||
| SELECT t5.t5.i1 FROM mydb1.t5; | ||
| SELECT t5.i1 FROM mydb1.t5; | ||
| SELECT t5.* FROM mydb1.t5; | ||
| SELECT t5.t5.* FROM mydb1.t5; | ||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t5.t5.i1 FROM mydb1.t5; | ||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t5.t5.i2 FROM mydb1.t5; | ||
| -- TODO: Support this scenario | ||
| SELECT mydb1.t5.* FROM mydb1.t5; | ||
|
|
||
| -- Cleanup and Reset | ||
| USE default; | ||
| DROP DATABASE mydb1 CASCADE; | ||
| DROP DATABASE mydb2 CASCADE; | ||
240 changes: 240 additions & 0 deletions
240
sql/core/src/test/resources/sql-tests/results/columnresolution-negative.sql.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,240 @@ | ||
| -- Automatically generated by SQLQueryTestSuite | ||
| -- Number of queries: 28 | ||
|
|
||
|
|
||
| -- !query 0 | ||
| CREATE DATABASE mydb1 | ||
| -- !query 0 schema | ||
| struct<> | ||
| -- !query 0 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 1 | ||
| USE mydb1 | ||
| -- !query 1 schema | ||
| struct<> | ||
| -- !query 1 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 2 | ||
| CREATE TABLE t1 USING parquet AS SELECT 1 AS i1 | ||
| -- !query 2 schema | ||
| struct<> | ||
| -- !query 2 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 3 | ||
| CREATE DATABASE mydb2 | ||
| -- !query 3 schema | ||
| struct<> | ||
| -- !query 3 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 4 | ||
| USE mydb2 | ||
| -- !query 4 schema | ||
| struct<> | ||
| -- !query 4 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 5 | ||
| CREATE TABLE t1 USING parquet AS SELECT 20 AS i1 | ||
| -- !query 5 schema | ||
| struct<> | ||
| -- !query 5 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 6 | ||
| SET spark.sql.crossJoin.enabled = true | ||
| -- !query 6 schema | ||
| struct<key:string,value:string> | ||
| -- !query 6 output | ||
| spark.sql.crossJoin.enabled true | ||
|
|
||
|
|
||
| -- !query 7 | ||
| USE mydb1 | ||
| -- !query 7 schema | ||
| struct<> | ||
| -- !query 7 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 8 | ||
| SELECT i1 FROM t1, mydb1.t1 | ||
| -- !query 8 schema | ||
| struct<> | ||
| -- !query 8 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 'i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 9 | ||
| SELECT t1.i1 FROM t1, mydb1.t1 | ||
| -- !query 9 schema | ||
| struct<> | ||
| -- !query 9 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 't1.i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To the other reviewer, this is the reason why we need to make a change in |
||
|
|
||
|
|
||
| -- !query 10 | ||
| SELECT mydb1.t1.i1 FROM t1, mydb1.t1 | ||
| -- !query 10 schema | ||
| struct<> | ||
| -- !query 10 output | ||
| org.apache.spark.sql.AnalysisException | ||
| cannot resolve '`mydb1.t1.i1`' given input columns: [i1, i1]; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 11 | ||
| SELECT i1 FROM t1, mydb2.t1 | ||
| -- !query 11 schema | ||
| struct<> | ||
| -- !query 11 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 'i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 12 | ||
| SELECT t1.i1 FROM t1, mydb2.t1 | ||
| -- !query 12 schema | ||
| struct<> | ||
| -- !query 12 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 't1.i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 13 | ||
| USE mydb2 | ||
| -- !query 13 schema | ||
| struct<> | ||
| -- !query 13 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 14 | ||
| SELECT i1 FROM t1, mydb1.t1 | ||
| -- !query 14 schema | ||
| struct<> | ||
| -- !query 14 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 'i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 15 | ||
| SELECT t1.i1 FROM t1, mydb1.t1 | ||
| -- !query 15 schema | ||
| struct<> | ||
| -- !query 15 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 't1.i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 16 | ||
| SELECT i1 FROM t1, mydb2.t1 | ||
| -- !query 16 schema | ||
| struct<> | ||
| -- !query 16 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 'i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 17 | ||
| SELECT t1.i1 FROM t1, mydb2.t1 | ||
| -- !query 17 schema | ||
| struct<> | ||
| -- !query 17 output | ||
| org.apache.spark.sql.AnalysisException | ||
| Reference 't1.i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 18 | ||
| SELECT db1.t1.i1 FROM t1, mydb2.t1 | ||
| -- !query 18 schema | ||
| struct<> | ||
| -- !query 18 output | ||
| org.apache.spark.sql.AnalysisException | ||
| cannot resolve '`db1.t1.i1`' given input columns: [i1, i1]; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 19 | ||
| SET spark.sql.crossJoin.enabled = false | ||
| -- !query 19 schema | ||
| struct<key:string,value:string> | ||
| -- !query 19 output | ||
| spark.sql.crossJoin.enabled false | ||
|
|
||
|
|
||
| -- !query 20 | ||
| USE mydb1 | ||
| -- !query 20 schema | ||
| struct<> | ||
| -- !query 20 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 21 | ||
| SELECT mydb1.t1 FROM t1 | ||
| -- !query 21 schema | ||
| struct<> | ||
| -- !query 21 output | ||
| org.apache.spark.sql.AnalysisException | ||
| cannot resolve '`mydb1.t1`' given input columns: [i1]; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 22 | ||
| SELECT t1.x.y.* FROM t1 | ||
| -- !query 22 schema | ||
| struct<> | ||
| -- !query 22 output | ||
| org.apache.spark.sql.AnalysisException | ||
| cannot resolve 't1.x.y.*' give input columns 'i1'; | ||
|
|
||
|
|
||
| -- !query 23 | ||
| SELECT t1 FROM mydb1.t1 | ||
| -- !query 23 schema | ||
| struct<> | ||
| -- !query 23 output | ||
| org.apache.spark.sql.AnalysisException | ||
| cannot resolve '`t1`' given input columns: [i1]; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 24 | ||
| USE mydb2 | ||
| -- !query 24 schema | ||
| struct<> | ||
| -- !query 24 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 25 | ||
| SELECT mydb1.t1.i1 FROM t1 | ||
| -- !query 25 schema | ||
| struct<> | ||
| -- !query 25 output | ||
| org.apache.spark.sql.AnalysisException | ||
| cannot resolve '`mydb1.t1.i1`' given input columns: [i1]; line 1 pos 7 | ||
|
|
||
|
|
||
| -- !query 26 | ||
| DROP DATABASE mydb1 CASCADE | ||
| -- !query 26 schema | ||
| struct<> | ||
| -- !query 26 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 27 | ||
| DROP DATABASE mydb2 CASCADE | ||
| -- !query 27 schema | ||
| struct<> | ||
| -- !query 27 output | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add two more cases for verifying
*