[SPARK-31963][PYSPARK][SQL] Support both pandas 0.23 and 1.0 in serializers.py

[williamhyun]

What changes were proposed in this pull request?

This PR aims to support both pandas 0.23 and 1.0.

Why are the changes needed?

$ pip install pandas==0.23.2

$ python -c "import pandas.CategoricalDtype"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas.CategoricalDtype'

$ python -c "from pandas.api.types import CategoricalDtype"

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins.

$ pip freeze | grep pandas
pandas==0.23.2

$ python/run-tests.py --python-executables python --modules pyspark-sql
...
Tests passed in 359 seconds

[SparkQA]

Test build #123782 has finished for PR 28789 at commit 160f16f.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Since the PR builder doesn't test Pandas test, I verified pyspark-sql module locally with Python 3.7.7 and Pandas 0.23.2.

cc @BryanCutler and @HyukjinKwon

[SparkQA]

Test build #123783 has finished for PR 28789 at commit e6586c3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Thank you, @williamhyun .
Merged to master.

[BryanCutler]

Thanks @williamhyun and @dongjoon-hyun LGTM

[dongjoon-hyun]

Thanks, @BryanCutler .

[HyukjinKwon]

Lgtm too. There's is_categorical_type actually :) but no biggie