Skip to content

[SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE pandas_udf with keyword args #20900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mstewart141 wants to merge 5 commits into from
Closed

[SPARK-23645][MINOR][DOCS][PYTHON] Add docs RE pandas_udf with keyword args #20900

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
048570f
[SPARK-23645][PYTHON] Allow python udfs to be called with keyword arg…
mstewart141 Mar 11, 2018
9ea2595
Incomplete / Show issue with partial fn in pandas_udf
mstewart141 Mar 18, 2018
acd1cbe
Add note RE no keyword args in python UDFs
mstewart141 Mar 18, 2018
bc49c3c
Address comments
mstewart141 Mar 24, 2018
a3da39c
Update wording
mstewart141 Mar 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions python/pyspark/sql/functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2155,6 +2155,8 @@ def udf(f=None, returnType=StringType()):
in boolean expressions and it ends up with being executed all internally. If the functions
can fail on special rows, the workaround is to incorporate the condition into the functions.

.. note:: The user-defined functions do not take keyword arguments.
Copy link
Member

@HyukjinKwon HyukjinKwon Mar 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add on the calling side (I meant not on function declaration side) at the end for clarification.


:param f: python function if used as a standalone function
:param returnType: the return type of the user-defined function. The value can be either a
:class:`pyspark.sql.types.DataType` object or a DDL-formatted type string.
Expand Down Expand Up @@ -2338,6 +2340,8 @@ def pandas_udf(f=None, returnType=None, functionType=None):
.. note:: The user-defined functions do not support conditional expressions or short circuiting
in boolean expressions and it ends up with being executed all internally. If the functions
can fail on special rows, the workaround is to incorporate the condition into the functions.

.. note:: The user-defined functions do not take keyword arguments.
"""
# decorator @pandas_udf(returnType, functionType)
is_decorator = f is None or isinstance(f, (str, DataType))
Expand Down