Upgrade Pandas dependency to 2.1 #31185
Conversation
|
Assigning reviewers. If you would like to opt out of this review, comment R: @riteshghorse for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| @@ -103,6 +109,7 @@ def test_ndframe_tests(self): | |||
| 'pandas.core.generic.NDFrame.get': ['*'], | |||
| }, | |||
| not_implemented_ok={ | |||
There was a problem hiding this comment.
Is the distinction here that there is a list of operations that we 100% won't implement, and then there are the operations that are just not-yet-implemented but could be?
There was a problem hiding this comment.
yes, that was the original intent. Order-sensitive ops are an example of 'Won't implement', although there is a proposal https://s.apache.org/order-sensitive-dataframe-operations to possibly support them later. The 'Not implemented' ops raise an exception that says, we didn't get around to support it but let us know if you care. See: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/dataframe/frame_base.py#L426-L442
This PR increases the upper bound to allow Pandas 2.1 and makes necessary changes to for Beam dataframe doctests pass.
Expanding the list of
ELEMENTWISE_STRING_METHODSenables support of additional dataframe operations in Beam by running them per-element:center, decode, encode, find, index, lfind, ljust, normalize, rfind, rindex, rjust, translate.Most of the changes exclude newly added doctests tests to Pandas 2.1, which exercise Dataframe operations that are not
supported in Beam, such as order-sensitive operations operations that are not applicable to a distributed, unordered deferred dataframe.
Excluded tests don't appear to be regressions in Pandas 2.1, they were also not supported in Pandas 1.x.
The PR also adds a capability to skip an example from all tests that include it, so that we don't need to exclude the same (unsupported) example multiple times in several tests.
Remaining issue: include a compat tests suite for Pandas 2.1. This is deferred, since current compat test suite uses Python 3.8, for which Pandas 2.1 is not available. In short term, Pandas 2.1 will be on tests against master. We should add a compat test suite for Pandas 2.1 before switching to Pandas 2.2.
fixes: #31198
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.