[Python] Support converting map to dict (instead of list of tuples) in scalar conversion / to_pylist

[adriangb]

Describe the bug, including details regarding any error messages, version, and platform.

import pyarrow as pa

schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]

I understand that this may be how it's stored internally but I feel like it should be converted to a dict.

Using struct is not an option because my keys are not fixed.

Component(s)

Python

[jorisvandenbossche]

The reason that a list of tuples was used as default in the past is that a dict cannot always represent the data of a map array, because a map allows duplicate keys.

But of course very often that is not an issue and you want dicts. And therefore a new keyword maps_as_pydicts was recently added (#34730), but only to the conversion path to pandas.

Using your example:

schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
batch = pa.RecordBatch.from_pylist(data, schema=schema)

this can now be converted to a pandas DataFrame (or numpy array) using dicts:

In [8]: batch.to_pandas(maps_as_pydicts="strict")
Out[8]: 
          x
0  {'a': 1}

However, the to_pylist() method takes a different code path, where this is not yet supported. Under the hood, this just calls as_py() on the individual MapScalar objects in a for loop.
But I think it certainly make sense to add this same keyword to this scalar conversion code path as well.

[adriangb]

Makes sense!

[jonded94]

Hey, we recently invested much time switching to parquet in our internal data processing tooling and stumbled over this exact issue. This is somewhat bad for us since we try to use parquet and pyarrow natively as much as possible, and then this workaround through pandas is a bit silly and also not optimal from a performance point of view.

Is there any traction currently around adding such a maps_as_pydicts parameter to to_pylist? Can one support in any way? Do you think this would be much work?

EDIT: This issue right now got even worse for us, since we noticed that the conversion to pandas dataframes has effects on our data:

• nullable integer types are converted to floats
• list types are converted to numpy arrays (e.g. of object type)

[jonded94]

Hey @jorisvandenbossche, I attached some PR that would already fix the problem which fundamentally blocks us right now. A new maps_as_pydicts parameter is included in hopefully all relevant methods.

Please let me know what you think, I'm very willing to do any adjustments as needed as fast as possible, as this is currently a large issue for us.