ARROW-10768: [Python] pass ndim to pandas make_block #8957
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on JIRA? Then could you also rename pull request title in the following format? See also: |
jorisvandenbossche
changed the title
python/pyarrow/pandas_compat.py
Outdated
| elif 'timezone' in item: | ||
| dtype = make_datetimetz(item['timezone']) | ||
| cls = dtype.construct_array_type() | ||
| block_arr = cls._simple_new(block_arr, dtype=dtype) |
There was a problem hiding this comment.
this line seems to cause the failures on latest pandas. So if the previous way was working, I would keep it that way (in addition, although pyarrow is already using private APIs here, _simple_new is even more a private implementation detail, so I would prefer not using it)
There was a problem hiding this comment.
In addition, the dtype.construct_array_type() line is failing on older pandas releases. So reverting this change might solve that as well.
There was a problem hiding this comment.
IIRC the motivation here was to stop passing dtype to make_block, since that kwarg isnt used within pandas and AFAICT this usage is the blocker to deprecating/removing it. could be considered separate from the matter at hand, but while im in the neighborhood
There was a problem hiding this comment.
OK, I understand.
Now, next problem: also DatetimeArray is not yet available on the older pandas versions ..
There was a problem hiding this comment.
(now we could do something else depending on the pandas version)
|
@jbrockmendel Do you want to update this PR? |
python/pyarrow/pandas_compat.py
Outdated
| except ImportError: | ||
| # older pandas versions | ||
| from pandas import DatetimeIndex | ||
| block_arr = DatetimeIndex(block_arr, dtype=dtype) |
There was a problem hiding this comment.
This is also failing on older pandas. I assume because DatetimeArray and DatetimeIndex might be interpreting datetime64 differently? (wall time vs unix time? didn't check, but I remember discussions about it)
jorgecarleitao
force-pushed
the
master
branch
from
d4608a9 to
356c300
Compare
|
closing, as pandas can now deal with this internally. |
2 participants
xref pandas-dev/pandas#38134 pandas will require passing ndim to make_block, this updates to the new usage
cc @jorisvandenbossche