Skip to content

Conversation

@itholic
Copy link
Contributor

@itholic itholic commented Sep 27, 2022

What changes were proposed in this pull request?

The PR proposes to fix CategoricalIndex.append to match the behavior with pandas.

Why are the changes needed?

Because the current behavior is different from pandas 1.5.0.

Does this PR introduce any user-facing change?

The behavior of API is changed as below:

Before

>>> psidx1 = ps.CategoricalIndex(["x", "y", "z"], categories=["z", "y", "x", "w"])
>>> psidx3 = ps.Index(["y", "x", "w", "z"])
>>> psidx1.append(psidx3.astype("category"))
CategoricalIndex(['x', 'y', 'z', 'y', 'x', 'w', 'z'], categories=['z', 'y', 'x', 'w'], ordered=False, dtype='category')

After

>>> psidx1 = ps.CategoricalIndex(["x", "y", "z"], categories=["z", "y", "x", "w"])
>>> psidx3 = ps.Index(["y", "x", "w", "z"])
>>> psidx1.append(psidx3.astype("category"))
CategoricalIndex(['x', 'y', 'z', 'x', 'y', 'z', 'w'], categories=['z', 'y', 'x', 'w'], ordered=False, dtype='category')

How was this patch tested?

Manually check the existing test is passed with pandas 1.5.0.

@itholic itholic changed the title [SPARK-40577][PS] Fix CategoricalIndex.append to match pandas 1.5.0 [SPARK-40577][PS] Fix CategoricalIndex.append to match pandas 1.5.0 Sep 27, 2022
@HyukjinKwon
Copy link
Member

Merged to master.

@zhengruifeng
Copy link
Contributor

Merged to master.

@HyukjinKwon this PR seems not merged?

@HyukjinKwon
Copy link
Member

oops

@itholic itholic deleted the SPARK-40577 branch April 22, 2023 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants