Skip to content

[SPARK-25056][SQL] Unify the InConversion and BinaryComparison behavior #22038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 15 commits into from
Closed

[SPARK-25056][SQL] Unify the InConversion and BinaryComparison behavior #22038

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
9459e6e
Unify the InConversion and BinaryComparison behaviour when InConversi…
wangyum Aug 8, 2018
c4775c4
findWiderTypeForTwo -> findWiderTypeWithoutStringPromotionForTwo
wangyum Aug 9, 2018
935ed36
Add findInCommonType
wangyum Aug 9, 2018
cb25b78
Fix test error.
wangyum Aug 10, 2018
4fd2143
Fix
wangyum Aug 12, 2018
48abf9f
Merge remote-tracking branch 'upstream/master' into SPARK-25056
wangyum Nov 14, 2019
20bdc95
Merge master
wangyum Nov 14, 2019
87ab27e
Fix indentation
wangyum Nov 14, 2019
80adb74
findCommonTypeForBinaryComparison -> findWiderTypeForDecimal -> findT…
wangyum Nov 14, 2019
7c56d38
Merge remote-tracking branch 'upstream/master' into SPARK-25056
wangyum Feb 2, 2020
232e42f
Merge upstream
wangyum Feb 2, 2020
8536af4
Merge remote-tracking branch 'upstream/master' into SPARK-25056
wangyum Feb 22, 2020
b1958dd
Add LEGACY_IN_PREDICATE_FOLLOW_BINARY_COMPARISON_TYPE_COERCION
wangyum Feb 22, 2020
f060efa
Merge remote-tracking branch 'upstream/master' into SPARK-25056
wangyum Mar 21, 2020
e60ff29
Fix
wangyum Mar 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -486,9 +486,17 @@ object TypeCoercion {
}

case i @ In(a, b) if b.exists(_.dataType != a.dataType) =>
findWiderCommonType(i.children.map(_.dataType)) match {
case Some(finalDataType) => i.withNewChildren(i.children.map(Cast(_, finalDataType)))
case None => i
if (b.map(_.dataType).distinct.size == 1) {
findCommonTypeForBinaryComparison(a.dataType, b.head.dataType, conf)
.orElse(findWiderTypeWithoutStringPromotionForTwo(a.dataType, b.head.dataType)) match {
case Some(finalDataType) => i.withNewChildren(i.children.map(Cast(_, finalDataType)))
case None => i
}
} else {
findWiderCommonType(i.children.map(_.dataType)) match {
case Some(finalDataType) => i.withNewChildren(i.children.map(Cast(_, finalDataType)))
case None => i
}
}
}
}
Expand Down
56 changes: 26 additions & 30 deletions sql/core/src/test/resources/sql-tests/results/typeCoercion/native/inConversion.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ true
-- !query 8
SELECT cast(1 as tinyint) in (cast(1 as string)) FROM t
-- !query 8 schema
struct<(CAST(CAST(1 AS TINYINT) AS STRING) IN (CAST(CAST(1 AS STRING) AS STRING))):boolean>
struct<(CAST(CAST(1 AS TINYINT) AS TINYINT) IN (CAST(CAST(1 AS STRING) AS TINYINT))):boolean>
Copy link
Member

@HyukjinKwon HyukjinKwon Sep 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should also update migration guide.

Copy link
Member Author

@wangyum wangyum Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the BinaryComparison behavior:

scala> spark.sql("explain SELECT cast(1 as tinyint) > (cast(1 as string))").show(false)
+---------------------------------------------------------------------------------------------------------------------------------------+
|plan                                                                                                                                   |
+---------------------------------------------------------------------------------------------------------------------------------------+
|== Physical Plan ==
*(1) Project [false AS (CAST(1 AS TINYINT) > CAST(CAST(1 AS STRING) AS TINYINT))#5]
+- *(1) Scan OneRowRelation[]

|
+---------------------------------------------------------------------------------------------------------------------------------------+

Copy link
Member

@maropu maropu Jan 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But, since this is a behaviour change in the existing in, I think its worth updating the guide.

-- !query 8 output
true

Expand Down Expand Up @@ -169,7 +169,7 @@ true
-- !query 20
SELECT cast(1 as smallint) in (cast(1 as string)) FROM t
-- !query 20 schema
struct<(CAST(CAST(1 AS SMALLINT) AS STRING) IN (CAST(CAST(1 AS STRING) AS STRING))):boolean>
struct<(CAST(CAST(1 AS SMALLINT) AS SMALLINT) IN (CAST(CAST(1 AS STRING) AS SMALLINT))):boolean>
-- !query 20 output
true

Expand Down Expand Up @@ -269,7 +269,7 @@ true
-- !query 32
SELECT cast(1 as int) in (cast(1 as string)) FROM t
-- !query 32 schema
struct<(CAST(CAST(1 AS INT) AS STRING) IN (CAST(CAST(1 AS STRING) AS STRING))):boolean>
struct<(CAST(CAST(1 AS INT) AS INT) IN (CAST(CAST(1 AS STRING) AS INT))):boolean>
-- !query 32 output
true

Expand Down Expand Up @@ -369,7 +369,7 @@ true
-- !query 44
SELECT cast(1 as bigint) in (cast(1 as string)) FROM t
-- !query 44 schema
struct<(CAST(CAST(1 AS BIGINT) AS STRING) IN (CAST(CAST(1 AS STRING) AS STRING))):boolean>
struct<(CAST(CAST(1 AS BIGINT) AS BIGINT) IN (CAST(CAST(1 AS STRING) AS BIGINT))):boolean>
-- !query 44 output
true

Expand Down Expand Up @@ -469,9 +469,9 @@ true
-- !query 56
SELECT cast(1 as float) in (cast(1 as string)) FROM t
-- !query 56 schema
struct<(CAST(CAST(1 AS FLOAT) AS STRING) IN (CAST(CAST(1 AS STRING) AS STRING))):boolean>
struct<(CAST(CAST(1 AS FLOAT) AS FLOAT) IN (CAST(CAST(1 AS STRING) AS FLOAT))):boolean>
-- !query 56 output
false
true


-- !query 57
Expand Down Expand Up @@ -569,9 +569,9 @@ true
-- !query 68
SELECT cast(1 as double) in (cast(1 as string)) FROM t
-- !query 68 schema
struct<(CAST(CAST(1 AS DOUBLE) AS STRING) IN (CAST(CAST(1 AS STRING) AS STRING))):boolean>
struct<(CAST(CAST(1 AS DOUBLE) AS DOUBLE) IN (CAST(CAST(1 AS STRING) AS DOUBLE))):boolean>
-- !query 68 output
false
true


-- !query 69
Expand Down Expand Up @@ -669,7 +669,7 @@ true
-- !query 80
SELECT cast(1 as decimal(10, 0)) in (cast(1 as string)) FROM t
-- !query 80 schema
struct<(CAST(CAST(1 AS DECIMAL(10,0)) AS STRING) IN (CAST(CAST(1 AS STRING) AS STRING))):boolean>
struct<(CAST(CAST(1 AS DECIMAL(10,0)) AS DOUBLE) IN (CAST(CAST(1 AS STRING) AS DOUBLE))):boolean>
-- !query 80 output
true

Expand Down Expand Up @@ -713,55 +713,55 @@ cannot resolve '(CAST(1 AS DECIMAL(10,0)) IN (CAST('2017-12-11 09:30:00' AS DATE
-- !query 85
SELECT cast(1 as string) in (cast(1 as tinyint)) FROM t
-- !query 85 schema
struct<(CAST(CAST(1 AS STRING) AS STRING) IN (CAST(CAST(1 AS TINYINT) AS STRING))):boolean>
struct<(CAST(CAST(1 AS STRING) AS TINYINT) IN (CAST(CAST(1 AS TINYINT) AS TINYINT))):boolean>
-- !query 85 output
true


-- !query 86
SELECT cast(1 as string) in (cast(1 as smallint)) FROM t
-- !query 86 schema
struct<(CAST(CAST(1 AS STRING) AS STRING) IN (CAST(CAST(1 AS SMALLINT) AS STRING))):boolean>
struct<(CAST(CAST(1 AS STRING) AS SMALLINT) IN (CAST(CAST(1 AS SMALLINT) AS SMALLINT))):boolean>
-- !query 86 output
true


-- !query 87
SELECT cast(1 as string) in (cast(1 as int)) FROM t
-- !query 87 schema
struct<(CAST(CAST(1 AS STRING) AS STRING) IN (CAST(CAST(1 AS INT) AS STRING))):boolean>
struct<(CAST(CAST(1 AS STRING) AS INT) IN (CAST(CAST(1 AS INT) AS INT))):boolean>
-- !query 87 output
true


-- !query 88
SELECT cast(1 as string) in (cast(1 as bigint)) FROM t
-- !query 88 schema
struct<(CAST(CAST(1 AS STRING) AS STRING) IN (CAST(CAST(1 AS BIGINT) AS STRING))):boolean>
struct<(CAST(CAST(1 AS STRING) AS BIGINT) IN (CAST(CAST(1 AS BIGINT) AS BIGINT))):boolean>
-- !query 88 output
true


-- !query 89
SELECT cast(1 as string) in (cast(1 as float)) FROM t
-- !query 89 schema
struct<(CAST(CAST(1 AS STRING) AS STRING) IN (CAST(CAST(1 AS FLOAT) AS STRING))):boolean>
struct<(CAST(CAST(1 AS STRING) AS FLOAT) IN (CAST(CAST(1 AS FLOAT) AS FLOAT))):boolean>
-- !query 89 output
false
true


-- !query 90
SELECT cast(1 as string) in (cast(1 as double)) FROM t
-- !query 90 schema
struct<(CAST(CAST(1 AS STRING) AS STRING) IN (CAST(CAST(1 AS DOUBLE) AS STRING))):boolean>
struct<(CAST(CAST(1 AS STRING) AS DOUBLE) IN (CAST(CAST(1 AS DOUBLE) AS DOUBLE))):boolean>
-- !query 90 output
false
true


-- !query 91
SELECT cast(1 as string) in (cast(1 as decimal(10, 0))) FROM t
-- !query 91 schema
struct<(CAST(CAST(1 AS STRING) AS STRING) IN (CAST(CAST(1 AS DECIMAL(10,0)) AS STRING))):boolean>
struct<(CAST(CAST(1 AS STRING) AS DOUBLE) IN (CAST(CAST(1 AS DECIMAL(10,0)) AS DOUBLE))):boolean>
-- !query 91 output
true

Expand All @@ -777,19 +777,17 @@ true
-- !query 93
SELECT cast(1 as string) in (cast('1' as binary)) FROM t
-- !query 93 schema
struct<>
struct<(CAST(CAST(1 AS STRING) AS BINARY) IN (CAST(CAST(1 AS BINARY) AS BINARY))):boolean>
-- !query 93 output
org.apache.spark.sql.AnalysisException
cannot resolve '(CAST(1 AS STRING) IN (CAST('1' AS BINARY)))' due to data type mismatch: Arguments must be same type but were: string != binary; line 1 pos 25
true


-- !query 94
SELECT cast(1 as string) in (cast(1 as boolean)) FROM t
-- !query 94 schema
struct<>
struct<(CAST(CAST(1 AS STRING) AS BOOLEAN) IN (CAST(CAST(1 AS BOOLEAN) AS BOOLEAN))):boolean>
-- !query 94 output
org.apache.spark.sql.AnalysisException
cannot resolve '(CAST(1 AS STRING) IN (CAST(1 AS BOOLEAN)))' due to data type mismatch: Arguments must be same type but were: string != boolean; line 1 pos 25
true


-- !query 95
Expand Down Expand Up @@ -874,10 +872,9 @@ cannot resolve '(CAST('1' AS BINARY) IN (CAST(1 AS DECIMAL(10,0))))' due to data
-- !query 104
SELECT cast('1' as binary) in (cast(1 as string)) FROM t
-- !query 104 schema
struct<>
struct<(CAST(CAST(1 AS BINARY) AS BINARY) IN (CAST(CAST(1 AS STRING) AS BINARY))):boolean>
-- !query 104 output
org.apache.spark.sql.AnalysisException
cannot resolve '(CAST('1' AS BINARY) IN (CAST(1 AS STRING)))' due to data type mismatch: Arguments must be same type but were: binary != string; line 1 pos 27
true


-- !query 105
Expand Down Expand Up @@ -981,10 +978,9 @@ cannot resolve '(true IN (CAST(1 AS DECIMAL(10,0))))' due to data type mismatch:
-- !query 116
SELECT true in (cast(1 as string)) FROM t
-- !query 116 schema
struct<>
struct<(CAST(true AS BOOLEAN) IN (CAST(CAST(1 AS STRING) AS BOOLEAN))):boolean>
-- !query 116 output
org.apache.spark.sql.AnalysisException
cannot resolve '(true IN (CAST(1 AS STRING)))' due to data type mismatch: Arguments must be same type but were: boolean != string; line 1 pos 12
true


-- !query 117
Expand Down