Skip to content

[SPARK-24392][PYTHON] Label pandas_udf as Experimental #21435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BryanCutler wants to merge 8 commits into from
Closed

[SPARK-24392][PYTHON] Label pandas_udf as Experimental #21435

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
376149a
add experimental to pandas_udf
BryanCutler May 25, 2018
fb05282
add experimental note to other arrow functionality
BryanCutler May 26, 2018
0bce522
moved placement of note in pandas udf
BryanCutler May 26, 2018
a1e70aa
added line in migration guide
BryanCutler May 26, 2018
38a765d
Made section in migration guide for 2.3.1+
BryanCutler May 27, 2018
f977293
forgot newline
BryanCutler May 27, 2018
84b52f3
use backticks for conf
BryanCutler May 27, 2018
6b87330
fix case in heading
BryanCutler May 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions python/pyspark/sql/functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
.. note:: Experimental

@HyukjinKwon HyukjinKwon May 26, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a big deal but can we put this right before doctest just for consistency?

@BryanCutler BryanCutler May 26, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, it was hard to find a consistent place to put it. I changed this one to be after the param definitions, before doctest. hopefully that's better.


@HyukjinKwon HyukjinKwon May 25, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BryanCutler, can we add a short node for createDataFrame and to Pandas with Pandas DataFrame input/output?

@BryanCutler BryanCutler May 25, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I could have sworn that was added before but I don't see it anywhere. Would it make sense to add the note to the SQLconf description?

@HyukjinKwon HyukjinKwon May 26, 2018 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anywhere is fine IMHO if it looks making sense. Another option is things like:

.. versionchanged:: 2.3
  Arrow optimization added
  .. note:: Experimental

I just wanted to make sure this feature is experimental.

@HyukjinKwon HyukjinKwon May 26, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, don't forget to update group.apply().

@BryanCutler BryanCutler May 26, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to add the note under versionchanged ends up with everything on one line and the note isn't highlighted like usual with Experimental, so I just made it it's own note.

Creates a vectorized user defined function (UDF).

:param f: user-defined function. A python function if used as a standalone function
Expand Down