Use Pyarrow 19 decimal32/64 objects in cudf Python - #18035
Conversation
mroeschke
added
Python
| }, | ||
| { | ||
| "dtype": "decimal64", | ||
| "dtype": "decimal128", |
There was a problem hiding this comment.
Since this data is generated from the pyarrow representation of our type in addition to apache/arrow#45570, this data generation step will fail.
Just changing this to test decimal128 as it shouldn't drastically change the spirit of the test
|
Responding to #17922 (comment), that's a great point. Let's not bump the minimum requirement so quickly. Let's do conditional handling of decimal types based on the pyarrow version for the moment. libcudf can be built assuming that there is 32/64 bit decimal support in the spec, so no need for conditionals in our C++, and in Python we can branch on the pyarrow version at runtime. |
Broken off (the non-breaking parts) from #18035 as that PR will probably not move forward since it would require a pyarrow minimum version bump to 19 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #18111
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Should we close this and just hold it off until whenever we upgrade? |
|
Sure, sounds good. I'll rebase this branch every so often locally |
2 participants
Description
For
Decimal32DtypeandDecimal64Dtype, we currently usepyarrow.Decimal128Typeas their pyarrow representation. As of pyarrow 19,pyarrow.Decimal32Typeandpyarrow.Decimal64Typetypes (and their arrays) exist allow us align their representation with their pyarrow representaions and with the libcudf decimal32/64 changes in #17422 to avoid conversions in the Python layer.These changes do assume
pyarrow>=19The breaking changes are:
Decimal32/64Dtype.to_arrowreturnpyarrow.Decimal32/64Typerespectively instead ofpyarrow.Decimal128TypeDecimal32/64Column.from_arrowonly acceptpyarrow.Decimal32/64Arrayrespectively instead ofpyarrow.Decimal128ArrayChecklist