[SPARK-52779][SQL][CONNECT] Support TimeType literal in Connect #51462
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
ping @MaxGekk to take a look, please. |
dengziming
changed the title
Member
|
@peter-toth Could you review this PR since you are working on similar one: #51464 |
peter-toth
reviewed
Jul 15, 2025
Contributor
There was a problem hiding this comment.
Can we use SparkDateTimeUtils.localTimeToNanos() here?
peter-toth
reviewed
Jul 15, 2025
peter-toth
approved these changes
Jul 15, 2025
Contributor
peter-toth
left a comment
peter-toth
left a comment
There was a problem hiding this comment.
LGTM, just minor nits.
HyukjinKwon
approved these changes
Jul 16, 2025
MaxGekk
approved these changes
Jul 16, 2025
Member
|
+1, LGTM. Merging to master. |
Contributor
|
quick question: does it works with python client? |
Member
Author
Thanks for pointing out this @zhengruifeng . I think it works with Python client, but I need to check it and add some PySpark tests. |
zhengruifeng
pushed a commit
that referenced
this pull request
Jul 24, 2025
### What changes were proposed in this pull request? This is the follow-up of #51462 to support TimeType literal in pyspark connect. ### Why are the changes needed? To align the Python Connect client with the Java/Scala Connect client. ### Does this PR introduce _any_ user-facing change? Yes, we can use TimeType literal in several ways, for example, `PySparkSession.sql("SELECT TIME '12:13:14'")` and `pyspark.sql.connect.functions.lit(datetime.time(12, 13, 14))`. ### How was this patch tested? 1. Add some local literal convert tests 2. Add pyspark SQL tests ### Was this patch authored or co-authored using generative AI tooling? No Closes #51515 from dengziming/SPARK-52779. Authored-by: dengziming <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
dongjoon-hyun
added a commit
to apache/spark-connect-swift
that referenced
this pull request
Oct 1, 2025
…th `4.1.0-preview2` ### What changes were proposed in this pull request? This PR aims to update Spark Connect-generated Swift source code with Apache Spark `4.1.0-preview2`. ### Why are the changes needed? There are many changes from Apache Spark 4.1.0. - apache/spark#52342 - apache/spark#52256 - apache/spark#52271 - apache/spark#52242 - apache/spark#51473 - apache/spark#51653 - apache/spark#52072 - apache/spark#51561 - apache/spark#51563 - apache/spark#51489 - apache/spark#51507 - apache/spark#51462 - apache/spark#51464 - apache/spark#51442 To use the latest bug fixes and new messages to develop for new features of `4.1.0-preview2`. ``` $ git clone -b v4.1.0-preview2 https://github.com/apache/spark.git $ cd spark/sql/connect/common/src/main/protobuf/ $ protoc --swift_out=. spark/connect/*.proto $ protoc --grpc-swift_out=. spark/connect/*.proto // Remove empty GRPC files $ cd spark/connect $ grep 'This file contained no services' * catalog.grpc.swift:// This file contained no services. commands.grpc.swift:// This file contained no services. common.grpc.swift:// This file contained no services. example_plugins.grpc.swift:// This file contained no services. expressions.grpc.swift:// This file contained no services. ml_common.grpc.swift:// This file contained no services. ml.grpc.swift:// This file contained no services. pipelines.grpc.swift:// This file contained no services. relations.grpc.swift:// This file contained no services. types.grpc.swift:// This file contained no services. $ rm catalog.grpc.swift commands.grpc.swift common.grpc.swift example_plugins.grpc.swift expressions.grpc.swift ml_common.grpc.swift ml.grpc.swift pipelines.grpc.swift relations.grpc.swift types.grpc.swift ``` ### Does this PR introduce _any_ user-facing change? Pass the CIs. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #250 from dongjoon-hyun/SPARK-53777. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Support TimeType literal in Spark Connect.
Why are the changes needed?
Part of SPARK-51162 to introduce TimeType
Does this PR introduce any user-facing change?
Yes, TimeType literals are supported in Connect, we can use functions such as
org.apache.spark.sql.functions.lit()to pass a time literal.How was this patch tested?
Added some unit tests.
Was this patch authored or co-authored using generative AI tooling?
No.