[SPARK-53112][SQL][PYTHON][CONNECT] Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark #51831

uros-db · 2025-08-04T21:23:43Z

What changes were proposed in this pull request?

Implement the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark & PySpark Connect API.

Why are the changes needed?

Expand API support for the make_timestamp_ntz and try_make_timestamp_ntz functions.

Does this PR introduce any user-facing change?

Yes, the new functions are now available in Python API.

How was this patch tested?

Added appropriate Python function tests.

pyspark.sql.tests.test_functions
pyspark.sql.tests.connect.test_parity_functions

Was this patch authored or co-authored using generative AI tooling?

No.

uros-db

Putting this PR on pause until the corresponding Scala PR is merged: #51828.

zhengruifeng · 2025-09-17T07:35:29Z

python/pyspark/sql/connect/functions/builtin.py

+
+
+def make_timestamp_ntz(  # type: ignore[misc]
+    yearsOrDate: "ColumnOrName",


can we change the argument names?
does it break this?

make_timestamp_ntz(years=..., months=..., days=..., hours=..., mins=..., secs=...)

yeah let's don't change the name

reverted this change. using keyword arguments for new date and time arguments.

uros-db · 2025-09-17T07:44:36Z

Adding @Yicong-Huang to take over this task and address lint failures & outstanding comments.

…standardizing formatting

…ed clarity

…ent handling and type casting

Yicong-Huang · 2025-09-23T01:00:34Z

@zhengruifeng could you please review this again? I've updated it with setting date and time as keyword arguments.

…ions with improved argument handling and error messaging

…p_ntz functions for clarity and consistency

…_ntz for flexibility

…t tests and clarifying positional arguments

zhengruifeng · 2025-09-24T02:59:19Z

python/pyspark/sql/connect/functions/builtin.py

+
+
+def make_timestamp_ntz(  # type: ignore[misc]
+    *args: "ColumnOrName",


shall we remove the *args: "ColumnOrName" and only support following patterns? @HyukjinKwon

make_timestamp_ntz(years=y, months=mo, days=d, hours=h, mins=mi, secs=s) make_timestamp_ntz(y, mo, d, h, mins=mi, secs=s) make_timestamp_ntz(y, mo, d, h, mi, s) make_timestamp_ntz(date=d, time=t) <- for the new code path

If we keep *args: "ColumnOrName", we need a helper function to preprocess the arguments so that it can be reused in all the places.

def _preprocess_make_timestampe_args(...) -> Tuple[ColumnOrName, ..., ColumnOrName] which returns a 8-tuple for the 8 arguments

we need the *args to match the positional arguments. Those are from the original use case. If we remove it, I am afraid that we'd cause breaking change?

I think it won't break positional cases like following?

make_timestamp_ntz(y, mo, d, h, mi, s) make_timestamp_ntz(y, mo, d, h, mins=mi, secs=s)

I think you are right. I did one more refactor to remove *args and now we explicitly support the following 4 cases:

make_timestamp_ntz(years=y, months=mo, days=d, hours=h, mins=mi, secs=s) make_timestamp_ntz(y, mo, d, h, mins=mi, secs=s) make_timestamp_ntz(y, mo, d, h, mi, s) make_timestamp_ntz(date=d, time=t)

Could you please check again?

…nctions by removing overloads and enhancing argument handling

zhengruifeng · 2025-09-24T07:27:48Z

python/pyspark/errors/error-conditions.json

      "Value for `<arg_name>` must be between <lower_bound> and <upper_bound> (inclusive), got <actual>"
    ]
  },
+  "WRONG_NUM_ARGS": {


it seems we can reuse error class CANNOT_SET_TOGETHER:

raise PySparkValueError( errorClass="CANNOT_SET_TOGETHER", messageParameters={"arg_list": "years|months|days|hours|mins|secs and date|time"}, )

zhengruifeng · 2025-09-24T07:28:27Z

python/pyspark/sql/connect/functions/builtin.py

+
+@overload
+def make_timestamp_ntz(
+    *,


do we still need *?

Yes. This is to mark the date and time (anything after *) are keyword only. Otherwise it will be marked as positional as well, then mypy would complain.

zhengruifeng · 2025-09-24T07:28:34Z

python/pyspark/sql/connect/functions/builtin.py

+    hours: Optional["ColumnOrName"] = None,
+    mins: Optional["ColumnOrName"] = None,
+    secs: Optional["ColumnOrName"] = None,
+    *,


do we still need *?

zhengruifeng · 2025-09-24T07:35:49Z

python/pyspark/sql/connect/functions/builtin.py

+        and date is None
+        and time is None
+    ):
+        from typing import cast


cast is already imported at the top of this file

removed redundant imports (cast, PySparkValueError).

zhengruifeng · 2025-09-24T07:43:38Z

python/pyspark/sql/connect/functions/builtin.py

+    date: Optional["ColumnOrName"] = None,
+    time: Optional["ColumnOrName"] = None,
+) -> Column:
+    # 6 positional/keyword arguments: years, months, days, hours, mins, secs


I think we can simplify it a bit:

if year is not None: if any(arg is not None for arg in [date, time]): raise CANNOT_SET_TOGETHER return _invoke_function_over_columns("make_timestamp_ntz", years, ...) else: if any(arg is not None for arg in [years, months, days, hours, mins, secs]): raise CANNOT_SET_TOGETHER return _invoke_function_over_columns("make_timestamp_ntz", date, time)

_invoke_function_over_columns will raise NOT_COLUMN_OR_STR if a input is None

Got it. I have it too complicated to check all arguments. I've followed your suggestion to push other validations to _invoke_function_over_columns

zhengruifeng · 2025-09-24T07:45:06Z

python/pyspark/sql/connect/functions/builtin.py

+    # Invalid argument combinations - return NULL for try_ functions
+    # For try_ functions, invalid inputs should return NULL
+    from pyspark.sql.connect.functions import lit
+
+    return lit(None)


I think we should not allow this

I've changed to use default values when inputs are None. Please check.

…ET_TOGETHER for conflicting arguments

…stamp_ntz and try_make_timestamp_ntz functions

…meters gracefully by returning NULL

…combinations, edge cases, and error handling scenarios

…generic exceptions for invalid inputs

…ner code

…mproved readability

zhengruifeng · 2025-09-25T00:20:04Z

python/pyspark/sql/connect/functions/builtin.py

+        _months = lit(0) if months is None else months
+        _days = lit(0) if days is None else days
+        _hours = lit(0) if hours is None else hours
+        _mins = lit(0) if mins is None else mins
+        _secs = lit(0) if secs is None else secs


why support None => lit(0) here?

…arameters and enhance test cases for required arguments

…ueError for conflicting argument combinations

zhengruifeng · 2025-09-25T03:18:45Z

merged to master

zhengruifeng · 2025-09-26T10:00:38Z

@Yicong-Huang some newly added tests actually depends on ANSI, and failed when ANSI is off.
see https://github.com/apache/spark/actions/runs/18025276597/job/51291154866

I am going to remove them in #52466

…p_ntz and try_make_timestamp_ntz functions in PySpark ### What changes were proposed in this pull request? Implement the `make_timestamp_ntz` and `try_make_timestamp_ntz` functions in PySpark & PySpark Connect API. ### Why are the changes needed? Expand API support for the `make_timestamp_ntz` and `try_make_timestamp_ntz` functions. ### Does this PR introduce _any_ user-facing change? Yes, the new functions are now available in Python API. ### How was this patch tested? Added appropriate Python function tests. - pyspark.sql.tests.test_functions - pyspark.sql.tests.connect.test_parity_functions ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#51831 from uros-db/python-try_make_timestamp_ntz. Lead-authored-by: Yicong-Huang <[email protected]> Co-authored-by: Uros Bojanic <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

Initial commit

ff9c6d6

github-actions bot added SQL PYTHON CONNECT labels Aug 4, 2025

uros-db commented Aug 4, 2025

View reviewed changes

uros-db marked this pull request as draft August 4, 2025 21:30

uros-db added 2 commits August 18, 2025 14:53

Cleanup

4e79dd8

Merge branch 'apache:master' into python-try_make_timestamp_ntz

a6bf88d

uros-db marked this pull request as ready for review August 27, 2025 07:18

uros-db added 4 commits August 27, 2025 13:41

Python linter

c707a1e

Update tests

53b30e9

Python lint fixes

87c2327

Python lint fixes

3a6e03a

Yicong-Huang mentioned this pull request Sep 16, 2025

[SPARK-53547] [PYTHON] Add make_timestamp_ntz overload with date/time parameters #52329

Closed

fix: doc-string

8180018

zhengruifeng requested a review from HyukjinKwon September 17, 2025 07:30

zhengruifeng reviewed Sep 17, 2025

View reviewed changes

zhengruifeng requested a review from ueshin September 17, 2025 07:35

Yicong-Huang added 8 commits September 17, 2025 10:58

fix: doc-string

853b9c8

fix: doc-string

46f6b10

wip: use keyword arguments only

eadad20

use keyword arguments for date and time

ce643c2

refactor: clean up docstrings for make_timestamp_ntz function

b5936cf

refactor: improve code readability in make_timestamp_ntz function by …

91740e6

…standardizing formatting

refactor: replace __builtins__ sum calls with built-in sum for improv…

da60214

…ed clarity

refactor: enhance try_make_timestamp_ntz function with improved argum…

338a23d

…ent handling and type casting

Yicong-Huang added 2 commits September 23, 2025 12:27

refactor: enhance make_timestamp_ntz and try_make_timestamp_ntz funct…

d581527

…ions with improved argument handling and error messaging

refactor: update comments in make_timestamp_ntz and try_make_timestam…

a868cf2

…p_ntz functions for clarity and consistency

Yicong-Huang added 3 commits September 23, 2025 17:22

refactor: replace keyword-only arguments with *args in make_timestamp…

860fc53

…_ntz for flexibility

refactor: swap error message keys for clarity in error-conditions.json

260df29

refactor: simplify make_timestamp_ntz usage by removing mixed argumen…

93c99bd

…t tests and clarifying positional arguments

zhengruifeng reviewed Sep 24, 2025

View reviewed changes

refactor: streamline make_timestamp_ntz and try_make_timestamp_ntz fu…

873cf4a

…nctions by removing overloads and enhancing argument handling

zhengruifeng reviewed Sep 24, 2025

View reviewed changes

Yicong-Huang added 9 commits September 24, 2025 09:27

refactor: update error handling in make_timestamp_ntz to use CANNOT_S…

66e1685

…ET_TOGETHER for conflicting arguments

refactor: improve argument validation and error handling in make_time…

bb42aeb

…stamp_ntz and try_make_timestamp_ntz functions

refactor: enhance try_make_timestamp_ntz function to handle None para…

31f33a9

…meters gracefully by returning NULL

test: expand make_timestamp_ntz test cases to cover various argument …

0696390

…combinations, edge cases, and error handling scenarios

fix: shorten doc string

6fb47d8

doc: add more examples in doc-string

a704c54

test: update exception handling in make_timestamp_ntz tests to raise …

d028c14

…generic exceptions for invalid inputs

test: remove unused imports in make_timestamp_ntz test cases for clea…

107ac4f

…ner code

doc: format output examples in make_timestamp_ntz documentation for i…

5acd80c

…mproved readability

zhengruifeng reviewed Sep 25, 2025

View reviewed changes

Yicong-Huang added 2 commits September 24, 2025 20:48

refactor: update try_make_timestamp_ntz function to use casting for p…

a3be61c

…arameters and enhance test cases for required arguments

refactor: enhance try_make_timestamp_ntz function to raise PySparkVal…

b824f78

…ueError for conflicting argument combinations

zhengruifeng approved these changes Sep 25, 2025

View reviewed changes

zhengruifeng closed this in a8cdae8 Sep 25, 2025

Yicong-Huang mentioned this pull request Oct 20, 2025

[SPARK-53956][PYTHON] Support TIME in the try_make_timestamp function in PySpark #52666

Closed



		def make_timestamp_ntz( # type: ignore[misc]
		yearsOrDate: "ColumnOrName",



		def make_timestamp_ntz( # type: ignore[misc]
		*args: "ColumnOrName",

[SPARK-53112][SQL][PYTHON][CONNECT] Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark #51831

[SPARK-53112][SQL][PYTHON][CONNECT] Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark #51831

Uh oh!

Conversation

uros-db commented Aug 4, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

uros-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uros-db commented Sep 17, 2025

Uh oh!

Yicong-Huang commented Sep 23, 2025

Uh oh!

zhengruifeng Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng commented Sep 25, 2025

Uh oh!

zhengruifeng commented Sep 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhengruifeng Sep 24, 2025 •

edited

Loading

zhengruifeng Sep 24, 2025 •

edited

Loading

zhengruifeng Sep 24, 2025 •

edited

Loading