You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Arrow has field names for list elements and map elements. arrow-rs can cast lists with one name for elements (like elements) to another name like `items). But this is not supported for the map type which has field names, one for the elements, one for the keys and one for the values. This can be a limitation for anyone using arrays of Maps from different sources.
Describe the solution you'd like
Maps to mimic the behavior of lists where casting would be supported where the elements of the maps contain the same structure but the names for the elements, keys and values of the fields may be different. Keys are always considered the first field according the arrow spec and values are considered the second field.
Describe alternatives you've considered
Some alternatives would be:
Rather than using the field names encoded in files like parquet for lists and maps, always convert to the same field when reading into an arrow record batch. There would be a small loss of information using this approach as what the field names before writing to Parquet would be lost, not sure how much this matters in practice. The proposed solution also does not prevent this approach from being implemented in the future.
Require third parties outside of arrow core to do the casting. This allows the consumer of arrow-rs to control what type of casting would be supported based on their needs. But in most cases consumers of arrow-rs will want the same type of casting.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Arrow has field names for list elements and map elements. arrow-rs can cast lists with one name for elements (like
elements
) to another name like `items). But this is not supported for the map type which has field names, one for the elements, one for the keys and one for the values. This can be a limitation for anyone using arrays of Maps from different sources.Describe the solution you'd like
Maps to mimic the behavior of lists where casting would be supported where the elements of the maps contain the same structure but the names for the elements, keys and values of the fields may be different. Keys are always considered the first field according the arrow spec and values are considered the second field.
Describe alternatives you've considered
Some alternatives would be:
The text was updated successfully, but these errors were encountered: