-
Notifications
You must be signed in to change notification settings - Fork 29.3k
[SPARK-32131][SQL] Fix AnalysisException messages at UNION/EXCEPT/MINUS operations #28951
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 19 commits
bb1efa2
1459d5b
88c40fe
df22083
0436611
ed80c84
39ca87c
c3b3c89
c3546eb
77a339a
a6b4f74
664277e
fd677c9
93b1f63
a5b5474
f4556a4
6071006
596b842
2865e45
1aa614d
984c652
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -831,4 +831,77 @@ class AnalysisSuite extends AnalysisTest with Matchers { | |
| } | ||
| } | ||
| } | ||
|
|
||
| test("SPARK-32131 Fix wrong column index when we have more than two columns" + | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks maropu ,btw, I did not see the test build start to build, do you know why? |
||
| " during union and set operations" ) { | ||
| val firstTable = LocalRelation( | ||
| AttributeReference("a", StringType)(), | ||
| AttributeReference("b", DoubleType)(), | ||
| AttributeReference("c", IntegerType)(), | ||
| AttributeReference("d", FloatType)()) | ||
|
|
||
| val secondTable = LocalRelation( | ||
| AttributeReference("a", StringType)(), | ||
| AttributeReference("b", TimestampType)(), | ||
| AttributeReference("c", IntegerType)(), | ||
| AttributeReference("d", FloatType)()) | ||
|
|
||
| val thirdTable = LocalRelation( | ||
| AttributeReference("a", StringType)(), | ||
| AttributeReference("b", DoubleType)(), | ||
| AttributeReference("c", TimestampType)(), | ||
| AttributeReference("d", FloatType)()) | ||
|
|
||
| val fourthTable = LocalRelation( | ||
| AttributeReference("a", StringType)(), | ||
| AttributeReference("b", DoubleType)(), | ||
| AttributeReference("c", IntegerType)(), | ||
| AttributeReference("d", TimestampType)()) | ||
|
|
||
| val a1 = firstTable.output(0) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @GuoPhilipse Variables a1, b1, c1, d1 not used ? Were you planning to use it in the test later ?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. em,,,let me remove it |
||
| val b1 = firstTable.output(1) | ||
| val c1 = firstTable.output(2) | ||
| val d1 = firstTable.output(3) | ||
|
|
||
| val a2 = secondTable.output(0) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ditto |
||
| val b2 = secondTable.output(1) | ||
| val c2 = secondTable.output(2) | ||
| val d2 = secondTable.output(3) | ||
|
|
||
| val a3 = thirdTable.output(0) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ditto |
||
| val b3 = thirdTable.output(1) | ||
| val c3 = thirdTable.output(2) | ||
| val d3 = thirdTable.output(3) | ||
|
|
||
| val a4 = fourthTable.output(0) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ditto
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks @dilipbiswal for your review. |
||
| val b4 = fourthTable.output(1) | ||
| val c4 = fourthTable.output(2) | ||
| val d4 = fourthTable.output(3) | ||
|
|
||
| val r1 = Union(firstTable, secondTable) | ||
| val r2 = Union(firstTable, thirdTable) | ||
| val r3 = Union(firstTable, fourthTable) | ||
| val r4 = Except(firstTable, secondTable, isAll = false) | ||
| val r5 = Intersect(firstTable, secondTable, isAll = false) | ||
|
|
||
| assertAnalysisError(r1, | ||
| Seq("Union can only be performed on tables with the compatible column types. " + | ||
| "timestamp <> double at the second column of the second table")) | ||
|
|
||
| assertAnalysisError(r2, | ||
| Seq("Union can only be performed on tables with the compatible column types. " + | ||
| "timestamp <> int at the third column of the second table")) | ||
|
|
||
| assertAnalysisError(r3, | ||
| Seq("Union can only be performed on tables with the compatible column types. " + | ||
| "timestamp <> float at the 4th column of the second table")) | ||
|
|
||
| assertAnalysisError(r4, | ||
| Seq("Except can only be performed on tables with the compatible column types. " + | ||
| "timestamp <> double at the second column of the second table")) | ||
|
|
||
| assertAnalysisError(r5, | ||
| Seq("Intersect can only be performed on tables with the compatible column types. " + | ||
| "timestamp <> double at the second column of the second table")) | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hehehe, nice catch ;)