maint: pandas 2.0 forward compatible changes #540
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Julien Jerphanion <[email protected]>
Required since pandas 2.0. Signed-off-by: Julien Jerphanion <[email protected]>
pd.DataFrame.append does not exist anymore as of pandas 2.0. Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
force-pushed
the
pandas-2.0-support
branch
from
a75405e to
a4e12ad
Compare
Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
force-pushed
the
pandas-2.0-support
branch
from
a4e12ad to
2825838
Compare
jjerphan
changed the title
For now, some dtypes checks are skipped. Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
Comparison of tz-naive and tz-aware datetime-like objects has been removed in Pandas 2.0. See: pandas-dev/pandas#49492 Signed-off-by: Julien Jerphanion <[email protected]>
We might need to unify the behavior then. Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
added a commit
that referenced
this pull request
Jul 12, 2023
Extracted from #540. Remove warnings, prerequites for the support of Pandas 2.0. Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
force-pushed
the
pandas-2.0-support
branch
from
e4244f0 to
ce3edf1
Compare
jjerphan
force-pushed
the
pandas-2.0-support
branch
from
3921c85 to
c9b4bc9
Compare
Signed-off-by: Julien Jerphanion <[email protected]>
Looking at the error message, the dtypes are apparently identical but aren't in practice. Signed-off-by: Julien Jerphanion <[email protected]>
The normalizaton and denormalization of empty Series and empty DataFrame is tied and has to slightly change with pandas 2.0. Thus, we can't satisfy the original test_fallback_to_pickle (which test empty DataFrame normalization and denormalization) and test_write_no_rows (which test empty Series' normalization and denormalization). The current choice of Pandas 2.0 allows for DataFrame and Series not to be pickled to be stored, which is currently the case with Pandas 1.0. Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
force-pushed
the
pandas-2.0-support
branch
from
c9b4bc9 to
07c720b
Compare
|
I've added the pin on pandas again and changed the title of this PR as discussed. The last comment of this PR can be addressed in a subsequent one after #646 is merged. |
qc00
reviewed
Aug 9, 2023
Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
vasil-pashov
added a commit
that referenced
this pull request
Aug 16, 2023
<!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> #### Reference Issues/PRs Resolves #639 Relates to: #540 #224 <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> #### What does this implement/fix? Explain your changes. Change the C++ to support a special undefined type that can be changed into any other type in a subsequent data key. #### Any other comments? Only the C++ layers is changed, it might need additional python layer integration. --------- Co-authored-by: willdealtry <[email protected]> Co-authored-by: Vasil Pashov <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]>
jjerphan
force-pushed
the
pandas-2.0-support
branch
from
f6743cf to
94e2f1b
Compare
Signed-off-by: Julien Jerphanion <[email protected]>
This reverts commit 9c3a56c.
This reverts commit 621ebc9.
Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
changed the title
Signed-off-by: Julien Jerphanion <[email protected]>
Signed-off-by: Julien Jerphanion <[email protected]>
jjerphan
changed the title
willdealtry
approved these changes
Aug 22, 2023
|
FYI, I have added references to the remaining threads there in #634 (comment). As discussed with Alex and William, let's merge this PR and treat them in dedicated PRs. |
jjerphan
dismissed
qc00’s stale review
Alex, William and I agree to treat the remaining points raised by Qi's in subsequent PRs.
This was referenced Aug 25, 2023
Merged
Merged
vasil-pashov
added a commit
that referenced
this pull request
Oct 18, 2023
<!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> Resolves #639 Relates to: #540 #224 <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> Change the C++ to support a special undefined type that can be changed into any other type in a subsequent data key. Only the C++ layers is changed, it might need additional python layer integration. --------- Co-authored-by: willdealtry <[email protected]> Co-authored-by: Vasil Pashov <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]>
vasil-pashov
pushed a commit
that referenced
this pull request
Oct 18, 2023
Signed-off-by: Julien Jerphanion <[email protected]> Co-authored-by: Qi Chen <[email protected]>
grusev
pushed a commit
that referenced
this pull request
Nov 25, 2024
<!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> #### Reference Issues/PRs Resolves #639 Relates to: #540 #224 <!-- Example: Fixes #1234. See also #3456. Please use keywords (e.g., Fixes) to create link to the issues or pull requests you resolved, so that they will automatically be closed when your pull request is merged. See: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue --> #### What does this implement/fix? Explain your changes. Change the C++ to support a special undefined type that can be changed into any other type in a subsequent data key. #### Any other comments? Only the C++ layers is changed, it might need additional python layer integration. --------- Co-authored-by: willdealtry <[email protected]> Co-authored-by: Vasil Pashov <[email protected]> Co-authored-by: Julien Jerphanion <[email protected]>
grusev
pushed a commit
that referenced
this pull request
Nov 25, 2024
Signed-off-by: Julien Jerphanion <[email protected]> Co-authored-by: Qi Chen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Fixes #237.
Follow-up of #343.
What does this implement/fix? Explain your changes.
This PR brings forward compatible changes for pandas 2.0. by:
pandas.Seriesandpandas.DataFrameAny other comments?
We need to discuss how to properly support some changes of behavior of pandas with respect to:
datetime64ortimedelta64datetime64andtimedelta64conversion for their resolution introduced by API: .astype to non-nano return the specified dtype pandas-dev/pandas#48928Edges cases which I think can be treated subsequently:
See
NOTEandTODOin the diff of this PR.