[SPARK-23054][SQL] Fix incorrect results of casting UserDefinedType to String #20246

maropu · 2018-01-12T08:08:19Z

What changes were proposed in this pull request?

This pr fixed the issue when casting UserDefinedTypes into strings;

>>> from pyspark.ml.classification import MultilayerPerceptronClassifier
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([(0.0, Vectors.dense([0.0, 0.0])), (1.0, Vectors.dense([0.0, 1.0]))], ["label", "features"])
>>> df.selectExpr("CAST(features AS STRING)").show(truncate = False)
+-------------------------------------------+
|features                                   |
+-------------------------------------------+
|[6,1,0,0,2800000020,2,0,0,0]               |
|[6,1,0,0,2800000020,2,0,0,3ff0000000000000]|
+-------------------------------------------+

The root cause is that Cast handles input data as UserDefinedType.sqlType(this is underlying storage type), so we should pass data into UserDefinedType.deserialize then toString.
This pr modified the result into;

+---------+                                                                     
|features |
+---------+
|[0.0,0.0]|
|[0.0,1.0]|
+---------+

How was this patch tested?

Added tests in UserDefinedTypeSuite .

ueshin · 2018-01-12T08:40:12Z

@maropu I guess you should elaborate the problem of the result string in the description.
The changes LGTM pending Jenkins, btw.

maropu · 2018-01-12T08:48:47Z

sure!

SparkQA · 2018-01-12T08:55:46Z

Test build #86027 has finished for PR 20246 at commit 137d85f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-01-12T08:57:24Z

retest this please

SparkQA · 2018-01-12T11:26:49Z

Test build #86033 has finished for PR 20246 at commit 137d85f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-01-12T11:44:00Z

retest this please

SparkQA · 2018-01-12T13:07:52Z

Test build #86032 has finished for PR 20246 at commit 137d85f.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-01-12T13:35:13Z

retest this please

cloud-fan · 2018-01-12T15:47:38Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

        builder.append("]")
        builder.build()
      })
+    case udt: UserDefinedType[_] =>


Is this the only place we miss UDT in Cast?

what about cast UDT to other types?

ok, I'll check

I think we can strip UDT at L608 and fix this problem entirely.

Here?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

Line 608 in cd9f49a

protected override def nullSafeEval(input: Any): Any = cast(input)

Since we can cast UDTs to other UDTs or strings only (by checking Cast.canCast), IIUC there is only a place to fix?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

Line 36 in cd9f49a

def canCast(from: DataType, to: DataType): Boolean = (from, to) match {

i see, makes sense

SparkQA · 2018-01-12T15:57:48Z

Test build #86034 has finished for PR 20246 at commit 137d85f.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-12T16:51:25Z

Test build #86036 has finished for PR 20246 at commit 137d85f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-01-13T01:48:32Z

retest this please

SparkQA · 2018-01-13T05:13:21Z

Test build #86070 has finished for PR 20246 at commit 137d85f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-01-14T20:38:50Z

Retest this please.

SparkQA · 2018-01-14T23:53:36Z

Test build #86126 has finished for PR 20246 at commit 137d85f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2018-01-15T02:56:14Z

thanks, merging to master/2.3

…o String ## What changes were proposed in this pull request? This pr fixed the issue when casting `UserDefinedType`s into strings; ``` >>> from pyspark.ml.classification import MultilayerPerceptronClassifier >>> from pyspark.ml.linalg import Vectors >>> df = spark.createDataFrame([(0.0, Vectors.dense([0.0, 0.0])), (1.0, Vectors.dense([0.0, 1.0]))], ["label", "features"]) >>> df.selectExpr("CAST(features AS STRING)").show(truncate = False) +-------------------------------------------+ |features | +-------------------------------------------+ |[6,1,0,0,2800000020,2,0,0,0] | |[6,1,0,0,2800000020,2,0,0,3ff0000000000000]| +-------------------------------------------+ ``` The root cause is that `Cast` handles input data as `UserDefinedType.sqlType`(this is underlying storage type), so we should pass data into `UserDefinedType.deserialize` then `toString`. This pr modified the result into; ``` +---------+ |features | +---------+ |[0.0,0.0]| |[0.0,1.0]| +---------+ ``` ## How was this patch tested? Added tests in `UserDefinedTypeSuite `. Author: Takeshi Yamamuro <[email protected]> Closes #20246 from maropu/SPARK-23054. (cherry picked from commit b98ffa4) Signed-off-by: Wenchen Fan <[email protected]>

…g PythonUserDefinedType to String. ## What changes were proposed in this pull request? This is a follow-up of #20246. If a UDT in Python doesn't have its corresponding Scala UDT, cast to string will be the raw string of the internal value, e.g. `"org.apache.spark.sql.catalyst.expressions.UnsafeArrayDataxxxxxxxx"` if the internal type is `ArrayType`. This pr fixes it by using its `sqlType` casting. ## How was this patch tested? Added a test and existing tests. Author: Takuya UESHIN <[email protected]> Closes #20306 from ueshin/issues/SPARK-23054/fup1. (cherry picked from commit 568055d) Signed-off-by: Wenchen Fan <[email protected]>

…g PythonUserDefinedType to String. ## What changes were proposed in this pull request? This is a follow-up of #20246. If a UDT in Python doesn't have its corresponding Scala UDT, cast to string will be the raw string of the internal value, e.g. `"org.apache.spark.sql.catalyst.expressions.UnsafeArrayDataxxxxxxxx"` if the internal type is `ArrayType`. This pr fixes it by using its `sqlType` casting. ## How was this patch tested? Added a test and existing tests. Author: Takuya UESHIN <[email protected]> Closes #20306 from ueshin/issues/SPARK-23054/fup1.

Cast user-defined data into strings

137d85f

maropu mentioned this pull request Jan 12, 2018

[SPARK-23023][SQL] Cast field data to strings in showString #20214

Closed

cloud-fan reviewed Jan 12, 2018

View reviewed changes

asfgit closed this in b98ffa4 Jan 15, 2018

ueshin mentioned this pull request Jan 18, 2018

[SPARK-23054][SQL][PYSPARK][FOLLOWUP] Use sqlType casting when casting PythonUserDefinedType to String. #20306

Closed

[SPARK-23054][SQL] Fix incorrect results of casting UserDefinedType to String #20246

[SPARK-23054][SQL] Fix incorrect results of casting UserDefinedType to String #20246

Uh oh!

Conversation

maropu commented Jan 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

ueshin commented Jan 12, 2018

Uh oh!

maropu commented Jan 12, 2018

Uh oh!

SparkQA commented Jan 12, 2018

Uh oh!

maropu commented Jan 12, 2018

Uh oh!

SparkQA commented Jan 12, 2018

Uh oh!

maropu commented Jan 12, 2018

Uh oh!

SparkQA commented Jan 12, 2018

Uh oh!

maropu commented Jan 12, 2018

Uh oh!

cloud-fan Jan 12, 2018

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jan 12, 2018

Choose a reason for hiding this comment

Uh oh!

maropu Jan 13, 2018

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jan 13, 2018

Choose a reason for hiding this comment

Uh oh!

maropu Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jan 15, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 12, 2018

Uh oh!

SparkQA commented Jan 12, 2018

Uh oh!

maropu commented Jan 13, 2018

Uh oh!

SparkQA commented Jan 13, 2018

Uh oh!

dongjoon-hyun commented Jan 14, 2018

Uh oh!

SparkQA commented Jan 14, 2018

Uh oh!

cloud-fan commented Jan 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

maropu commented Jan 12, 2018 •

edited

Loading

maropu Jan 13, 2018 •

edited

Loading