Skip to content

Conversation

@HyukjinKwon
Copy link
Member

What changes were proposed in this pull request?

This PR is a small followup of #28793 and proposes to use is_categorical_dtype instead of deprecated is_categorical.

is_categorical_dtype exists from minimum pandas version we support (https://github.com/pandas-dev/pandas/blob/v0.23.2/pandas/core/dtypes/api.py), and is_categorical was deprecated from pandas 1.1.0 (pandas-dev/pandas@87a1cc2).

Why are the changes needed?

To avoid using deprecated APIs, and remove warnings.

Does this PR introduce any user-facing change?

Yes, it will remove warnings that says is_categorical is deprecated.

How was this patch tested?

By running any pandas UDF with pandas 1.1.0+:

import pandas as pd
from pyspark.sql.functions import pandas_udf

def func(x: pd.Series) -> pd.Series:
    return x

spark.range(10).select(pandas_udf(func, "long")("id")).show()

Before:

/.../python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py:151: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
...

After:

...

@SparkQA
Copy link

SparkQA commented Oct 21, 2020

Test build #130068 has finished for PR 30114 at commit 4b1ef1f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 21, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34677/

@SparkQA
Copy link

SparkQA commented Oct 21, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34677/

Copy link
Member

@BryanCutler BryanCutler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for catching this @HyukjinKwon !

@BryanCutler
Copy link
Member

merged to master.

@HyukjinKwon
Copy link
Member Author

Thanks, @BryanCutler and @srowen.

@HyukjinKwon HyukjinKwon deleted the replace-deprecated-is_categorical branch December 7, 2020 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants