[SPARK-41238][CONNECT][PYTHON] Support more built-in datatypes #38770

zhengruifeng · 2022-11-23T12:32:58Z

What changes were proposed in this pull request?

1, in the sever side, make proto_datatype <-> catalyst_datatype conversion support all the built-in sql datatypes;

2, in the client side, make proto_datatype <-> pyspark_catalyst_datatype conversion support all the datatypes that are supported in pyspark now.

Why are the changes needed?

right now, only long, string, struct are supported

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Does not support convert float to connect proto types."
        debug_error_string = "{"created":"@1669206685.760099000","description":"Error received from peer ipv6:[::1]:15002","file":"src/core/lib/surface/call.cc","file_line":1064,"grpc_message":"Does not support convert float to connect proto types.","grpc_status":2}"

this PR make the schema and literal expr support more datatypes.

Does this PR introduce any user-facing change?

No

How was this patch tested?

added UT

HyukjinKwon

This is nice! LGTM, pending tests.

We should also probably fix up literal support together after this (see also #38462 (comment))

amaliujia · 2022-11-24T01:39:20Z

Thanks.

Can you test both nullable=true and nullable=false case? Only IIRC only ARRAY, MAP, STRUCT needs to care about it.

init

zhengruifeng · 2022-11-24T07:32:22Z

Can you test both nullable=true and nullable=false case?

the tests added in test_schema covers those cases @amaliujia

grundprinzip · 2022-11-24T08:02:22Z

Thank you! This has caused some challenges in demoing as we didn't support many types!

grundprinzip · 2022-11-24T08:04:17Z

connector/connect/src/main/protobuf/spark/connect/types.proto


 // This message describes the logical [[DataType]] of something. It does not carry the value
 // itself but only describes it.
 message DataType {


This is a severe breaking change, we need to make sure that at some point in time our protos become stable. In particular, here the order of the members changes.

maybe also need to add some UDTs(like VectorUDT) in the future, but now SQL types were completed.

grundprinzip · 2022-11-24T08:07:28Z

python/pyspark/sql/tests/connect/test_connect_basic.py

        )

+        # test FloatType, DoubleType, DecimalType, StringType, BooleanType, NullType
+        query = """


Love the SQL test for type checking :)

amaliujia · 2022-11-24T17:31:53Z

LGTM

There is a duplicate JIRA: https://issues.apache.org/jira/browse/SPARK-41057. Please remember mark that one as duplicate when this PR is merged.

zhengruifeng · 2022-11-25T01:26:57Z

merged into master

zhengruifeng · 2022-11-25T01:36:48Z

We should also probably fix up literal support together after this (see also #38462 (comment))

will take a look

…` in the client ### What changes were proposed in this pull request? Support `DayTimeIntervalType` in the client ### Why are the changes needed? In #38770, I forgot to deal with `DayTimeIntervalType` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added test case, the schema should be ``` In [1]: query = """ SELECT INTERVAL '100 10:30' DAY TO MINUTE AS interval """ In [2]: spark.sql(query).schema Out[2]: StructType([StructField('interval', DayTimeIntervalType(0, 2), False)]) ``` Closes #38818 from zhengruifeng/connect_type_time_stamp. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

### What changes were proposed in this pull request? 1, in the sever side, make `proto_datatype` <-> `catalyst_datatype` conversion support all the built-in sql datatypes; 2, in the client side, make `proto_datatype` <-> `pyspark_catalyst_datatype` conversion support [all the datatypes that are supported in pyspark now.](https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L60-L83) ### Why are the changes needed? right now, only `long`, `string`, `struct` are supported ``` grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Does not support convert float to connect proto types." debug_error_string = "{"created":"1669206685.760099000","description":"Error received from peer ipv6:[::1]:15002","file":"src/core/lib/surface/call.cc","file_line":1064,"grpc_message":"Does not support convert float to connect proto types.","grpc_status":2}" ``` this PR make the schema and literal expr support more datatypes. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added UT Closes apache#38770 from zhengruifeng/connect_support_more_datatypes. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

…` in the client ### What changes were proposed in this pull request? Support `DayTimeIntervalType` in the client ### Why are the changes needed? In apache#38770, I forgot to deal with `DayTimeIntervalType` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added test case, the schema should be ``` In [1]: query = """ SELECT INTERVAL '100 10:30' DAY TO MINUTE AS interval """ In [2]: spark.sql(query).schema Out[2]: StructType([StructField('interval', DayTimeIntervalType(0, 2), False)]) ``` Closes apache#38818 from zhengruifeng/connect_type_time_stamp. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

### What changes were proposed in this pull request? 1, in the sever side, make `proto_datatype` <-> `catalyst_datatype` conversion support all the built-in sql datatypes; 2, in the client side, make `proto_datatype` <-> `pyspark_catalyst_datatype` conversion support [all the datatypes that are supported in pyspark now.](https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L60-L83) ### Why are the changes needed? right now, only `long`, `string`, `struct` are supported ``` grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Does not support convert float to connect proto types." debug_error_string = "{"created":"1669206685.760099000","description":"Error received from peer ipv6:[::1]:15002","file":"src/core/lib/surface/call.cc","file_line":1064,"grpc_message":"Does not support convert float to connect proto types.","grpc_status":2}" ``` this PR make the schema and literal expr support more datatypes. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added UT Closes apache#38770 from zhengruifeng/connect_support_more_datatypes. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

…` in the client ### What changes were proposed in this pull request? Support `DayTimeIntervalType` in the client ### Why are the changes needed? In apache#38770, I forgot to deal with `DayTimeIntervalType` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added test case, the schema should be ``` In [1]: query = """ SELECT INTERVAL '100 10:30' DAY TO MINUTE AS interval """ In [2]: spark.sql(query).schema Out[2]: StructType([StructField('interval', DayTimeIntervalType(0, 2), False)]) ``` Closes apache#38818 from zhengruifeng/connect_type_time_stamp. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

github-actions bot added CONNECT CORE PYTHON SQL labels Nov 23, 2022

HyukjinKwon approved these changes Nov 24, 2022

View reviewed changes

zhengruifeng added 2 commits November 24, 2022 10:50

init

ef26f8d

init

fix test

78e7541

zhengruifeng changed the title ~~[SPARK-41238][CONNECT][PYTHON] Support more datatypes~~ [SPARK-41238][CONNECT][PYTHON] Support more built-in datatypes Nov 24, 2022

address comments

4c5f279

zhengruifeng force-pushed the connect_support_more_datatypes branch from aa72c69 to 4c5f279 Compare November 24, 2022 04:10

zhengruifeng added 2 commits November 24, 2022 12:49

rename numeric types

f5194c8

reformat

cddf350

grundprinzip reviewed Nov 24, 2022

View reviewed changes

grundprinzip approved these changes Nov 24, 2022

View reviewed changes

zhengruifeng closed this in 4006d19 Nov 25, 2022

zhengruifeng deleted the connect_support_more_datatypes branch November 25, 2022 01:27

zhengruifeng mentioned this pull request Nov 28, 2022

[SPARK-41238][CONNECT][PYTHON][FOLLOWUP] Support DayTimeIntervalType in the client #38818

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-41238][CONNECT][PYTHON] Support more built-in datatypes #38770

[SPARK-41238][CONNECT][PYTHON] Support more built-in datatypes #38770

Uh oh!

zhengruifeng commented Nov 23, 2022 •

edited

Loading

Uh oh!

HyukjinKwon left a comment •

edited

Loading

Uh oh!

amaliujia commented Nov 24, 2022 •

edited

Loading

Uh oh!

zhengruifeng commented Nov 24, 2022 •

edited

Loading

Uh oh!

grundprinzip commented Nov 24, 2022

Uh oh!

grundprinzip Nov 24, 2022

Uh oh!

zhengruifeng Nov 25, 2022

Uh oh!

grundprinzip Nov 24, 2022

Uh oh!

amaliujia commented Nov 24, 2022 •

edited

Loading

Uh oh!

zhengruifeng commented Nov 25, 2022

Uh oh!

zhengruifeng commented Nov 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-41238][CONNECT][PYTHON] Support more built-in datatypes #38770

[SPARK-41238][CONNECT][PYTHON] Support more built-in datatypes #38770

Uh oh!

Conversation

zhengruifeng commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amaliujia commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengruifeng commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grundprinzip commented Nov 24, 2022

Uh oh!

grundprinzip Nov 24, 2022

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Nov 25, 2022

Choose a reason for hiding this comment

Uh oh!

grundprinzip Nov 24, 2022

Choose a reason for hiding this comment

Uh oh!

amaliujia commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengruifeng commented Nov 25, 2022

Uh oh!

zhengruifeng commented Nov 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhengruifeng commented Nov 23, 2022 •

edited

Loading

HyukjinKwon left a comment •

edited

Loading

amaliujia commented Nov 24, 2022 •

edited

Loading

zhengruifeng commented Nov 24, 2022 •

edited

Loading

amaliujia commented Nov 24, 2022 •

edited

Loading