ARROW-10151: [Python] Add support for MapArray conversion to Pandas #8337
Conversation
|
Ping @pitrou @jorisvandenbossche to please take a look, thanks! |
pitrou
left a comment
pitrou
left a comment
There was a problem hiding this comment.
Thanks for doing this. Here are a couple comments.
There was a problem hiding this comment.
Isn't the line below adding another reference to that object?
There was a problem hiding this comment.
Why do you need this? I would expect py_items to contain a None entry already...?
There was a problem hiding this comment.
Good point, let me try that
There was a problem hiding this comment.
Actually, this doesn't seem to quite work. The backing array is numpy so if the dtype is an object, it will return None, but if numeric, it will be nan. I think we want the conversion to have None for all null values right?
There was a problem hiding this comment.
Is this required? I would expect the above to be always successful?
There was a problem hiding this comment.
Hmm, I'm not an expert on the python/numpy c-apis, so I was following ConvertStruct here. I thought that PyArray_GETITEM might set an error, but looks like it would return null on failure. Is it possible the reset of the item_value PyObject could cause an error?
There was a problem hiding this comment.
Ah, I though PyArray_GETITEM was a simple macro but it seems more involved. So it can return an error indeed.
|
Also, you'll want to rebase. |
BryanCutler
force-pushed
the
map-type-to-pandas
branch
from
28fb789 to
2ac0282
Compare
|
Thanks for reviewing @pitrou , I updated but still had a couple questions and I'm not totally sure I got the reference counting right. I'll have to take a closer look tomorrow. |
There was a problem hiding this comment.
*out_values = list_item.detach(), simply.
There was a problem hiding this comment.
Ah yes, thanks for fixing that up
pitrou
force-pushed
the
map-type-to-pandas
branch
from
2ac0282 to
090177e
Compare
|
Will merge if CI is ok, thank you. |
|
Thanks for all the help @pitrou ! |
2 participants
This change adds conversion for a
pyarrow.MapArrayto Pandas as a column of lists of tuples, where each tuple is a key/item pair. Unit tests were added for python to verify conversion for Pandas round-trip, chunked arrays andMapArraywith NULL map and NULL items.