Skip to content

[SPARK-24916][SQL] Fix type coercion for IN expression with subquery #21871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 7 commits into from
Closed

[SPARK-24916][SQL] Fix type coercion for IN expression with subquery #21871

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d60185b
default: findCommonTypeForBinaryComparison(l.dataType, r.dataType, co…
wangyum Jul 25, 2018
8a118b5
findCommonTypeForBinaryComparison(l.dataType, r.dataType, conf).orEls…
wangyum Jul 25, 2018
c306810
findWiderTypeForTwo(l.dataType, r.dataType)
wangyum Jul 25, 2018
daa120e
findCommonTypeForBinaryComparison(l.dataType, r.dataType, conf).orEls…
wangyum Jul 25, 2018
c84ba4d
findWiderTypeWithoutStringPromotionForTwo(l.dataType, r.dataType)
wangyum Jul 25, 2018
bc41b99
findCommonTypeForBinaryComparison(l.dataType, r.dataType, conf).orEls…
wangyum Jul 25, 2018
8ef142f
Fix type coercion for IN expression with subquery
wangyum Jul 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -469,7 +469,7 @@ object TypeCoercion {

val commonTypes = lhs.zip(rhs).flatMap { case (l, r) =>
findCommonTypeForBinaryComparison(l.dataType, r.dataType, conf)
.orElse(findTightestCommonType(l.dataType, r.dataType))
.orElse(findWiderTypeWithoutStringPromotionForTwo(l.dataType, r.dataType))
}

// The number of columns/expressions must match between LHS and RHS of an
Expand Down
2 changes: 1 addition & 1 deletion ...core/src/test/resources/sql-tests/inputs/subquery/negative-cases/subq-input-typecheck.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ CREATE TEMPORARY VIEW t3 AS SELECT * FROM VALUES
AS t3(t3a, t3b, t3c);

CREATE TEMPORARY VIEW t4 AS SELECT * FROM VALUES
(CAST(1 AS DOUBLE), CAST(2 AS STRING), CAST(3 AS STRING))
(CAST('2017-12-11' AS DATE), CAST(2 AS STRING), CAST('2017-12-11' AS DATE))
AS t1(t4a, t4b, t4c);

CREATE TEMPORARY VIEW t5 AS SELECT * FROM VALUES
Expand Down
156 changes: 156 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/inConversion.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,3 +328,159 @@ SELECT cast('2017-12-12 09:30:00' as date) in (cast('2017-12-12 09:30:00' as dat
SELECT cast('2017-12-12 09:30:00' as date) in (cast('2017-12-12 09:30:00' as date), cast(1 as boolean)) FROM t;
SELECT cast('2017-12-12 09:30:00' as date) in (cast('2017-12-12 09:30:00' as date), cast('2017-12-11 09:30:00.0' as timestamp)) FROM t;
SELECT cast('2017-12-12 09:30:00' as date) in (cast('2017-12-12 09:30:00' as date), cast('2017-12-11 09:30:00' as date)) FROM t;

SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as tinyint) FROM t);
Copy link
Member

@HyukjinKwon HyukjinKwon Jul 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to test all the combinations? We need most of such logics should be tested in findWiderTypeWithoutStringPromotionForTwo and we could have just few end to end tests.

SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as tinyint)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as smallint)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as int)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as bigint)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as float)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as double)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as decimal(10, 0))) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as string)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast('1' as binary)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast(1 as boolean)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00.0' as timestamp)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);

SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as tinyint) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as smallint) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as int) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as bigint) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as float) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as double) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as decimal(10, 0)) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as string) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast('1' as binary) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast(1 as boolean) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast('2017-12-11 09:30:00.0' as timestamp) FROM t);
SELECT * FROM t WHERE (cast('2017-12-11 09:30:00' as date)) IN (SELECT cast('2017-12-11 09:30:00' as date) FROM t);
6 changes: 3 additions & 3 deletions ...src/test/resources/sql-tests/results/subquery/negative-cases/subq-input-typecheck.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ struct<>

-- !query 3
CREATE TEMPORARY VIEW t4 AS SELECT * FROM VALUES
(CAST(1 AS DOUBLE), CAST(2 AS STRING), CAST(3 AS STRING))
(CAST('2017-12-11' AS DATE), CAST(2 AS STRING), CAST('2017-12-11' AS DATE))
AS t1(t4a, t4b, t4c)
-- !query 3 schema
struct<>
Expand Down Expand Up @@ -139,8 +139,8 @@ cannot resolve '(named_struct('t4a', t4.`t4a`, 't4b', t4.`t4b`, 't4c', t4.`t4c`)
The data type of one or more elements in the left hand side of an IN subquery
is not compatible with the data type of the output of the subquery
Mismatched columns:
[(t4.`t4a`:double, t5.`t5a`:decimal(18,0)), (t4.`t4c`:string, t5.`t5c`:bigint)]
[(t4.`t4a`:date, t5.`t5a`:decimal(18,0)), (t4.`t4c`:date, t5.`t5c`:bigint)]
Left side:
[double, string, string].
[date, string, date].
Right side:
[decimal(18,0), string, bigint].;
Loading