BUG: read_parquet not preserving type of partition_cols

[galipremsagar]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'a': [1, 2, 3, 10, 11, 1, 2, 3, 4, 10, 11], 'b':['a']*11})

In [3]: mkdir "tmp_dir"

In [4]: df.to_parquet("tmp_dir", partition_cols=['a'])

In [5]: df.dtypes
Out[5]: 
a     int64
b    object
dtype: object

In [6]: pd.read_parquet("tmp_dir")
Out[6]: 
    b   a
0   a   1
1   a   1
2   a  10
3   a  10
4   a  11
5   a  11
6   a   2
7   a   2
8   a   3
9   a   3
10  a   4

In [7]: pd.read_parquet("tmp_dir").dtypes
Out[7]: 
b      object
a    category
dtype: object

In [8]: pd.read_parquet("tmp_dir")['a']
Out[8]: 
0      1
1      1
2     10
3     10
4     11
5     11
6      2
7      2
8      3
9      3
10     4
Name: a, dtype: category
Categories (6, int32): [1, 10, 11, 2, 3, 4]

Issue Description

When we partition a dataframe into multiple files using a column(a) in this case, and we read it back, the partitioned column is converted to categorical type of the original type. This is expected, but with pandas-2.0, the categorical type seems to be modifying the underlying type from int64 to int32. See example above & expected behavior.

Expected Behavior

In [8]: pd.read_parquet("tmp_dir")['a']
Out[8]: 
0      1
1      1
2     10
3     10
4     11
5     11
6      2
7      2
8      3
9      3
10     4
Name: a, dtype: category
Categories (6, int64): [1, 10, 11, 2, 3, 4]

Installed Versions

In [9]: pd.show_versions()
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : 37ea63d
python : 3.10.11.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-76-generic
Version : #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.2
Cython : 0.29.34
pytest : 7.3.1
hypothesis : 6.75.3
sphinx : 5.3.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.13.2
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli :
fastparquet : None
fsspec : 2023.5.0
gcsfs : None
matplotlib : None
numba : 0.56.4
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : 2023.5.0
scipy : 1.10.1
snappy :
sqlalchemy : 2.0.15
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[mroeschke]

It looks like pyarrow is reading the partitions into int32 values, so maybe pandas maintaining this is unintended but better behavior?

In [3]: import pyarrow.parquet as pq

In [4]: pq.read_table("tmp_dir")
Out[4]: 
pyarrow.Table
b: string
a: dictionary<values=int32, indices=int32, ordered=0>
----
b: [["a","a"],["a","a"],...,["a","a"],["a"]]
a: [  -- dictionary:
[1,10,11,2,3,4]  -- indices:
[0,0],  -- dictionary:
[1,10,11,2,3,4]  -- indices:
[1,1],...,  -- dictionary:
[1,10,11,2,3,4]  -- indices:
[4,4],  -- dictionary:
[1,10,11,2,3,4]  -- indices:
[5]]

[alippai]

Also see #53008