Commit 541dbc0
[SPARK-23054][SQL][PYSPARK][FOLLOWUP] Use sqlType casting when casting PythonUserDefinedType to String.
## What changes were proposed in this pull request?
This is a follow-up of #20246.
If a UDT in Python doesn't have its corresponding Scala UDT, cast to string will be the raw string of the internal value, e.g. `"org.apache.spark.sql.catalyst.expressions.UnsafeArrayDataxxxxxxxx"` if the internal type is `ArrayType`.
This pr fixes it by using its `sqlType` casting.
## How was this patch tested?
Added a test and existing tests.
Author: Takuya UESHIN <[email protected]>
Closes #20306 from ueshin/issues/SPARK-23054/fup1.
(cherry picked from commit 568055d)
Signed-off-by: Wenchen Fan <[email protected]>1 parent 225b1af commit 541dbc0
File tree
3 files changed
+15
-0
lines changed- python/pyspark/sql
- sql
- catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions
- core/src/main/scala/org/apache/spark/sql/test
3 files changed
+15
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1189 | 1189 | | |
1190 | 1190 | | |
1191 | 1191 | | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
1192 | 1203 | | |
1193 | 1204 | | |
1194 | 1205 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
| 285 | + | |
285 | 286 | | |
286 | 287 | | |
287 | 288 | | |
| |||
838 | 839 | | |
839 | 840 | | |
840 | 841 | | |
| 842 | + | |
841 | 843 | | |
842 | 844 | | |
843 | 845 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
0 commit comments