Spatial data support

[Folyd]

Is your feature request related to a problem or challenge?

Currently, datafusion does not support spatial data, any plan to support this?

Describe the solution you'd like

Similar to duckdb: https://duckdb.org/docs/extensions/spatial.html

Describe alternatives you've considered

Duckdb

Additional context

https://cloud.google.com/bigquery/docs/geospatial-data

[alamb]

There are no plans that i know of yet.

In theory, we should be able to create an extension package, much like the duckdb model, rather than extending the core DataFusion engine.

I suspect there would be certain things that are not yet feasible (like adding a GEOMETRY type / alias for example) but otherwise the existiing extension points for DataFusion should be sufficient (ScalarUDFs, AggregateUDFs, etc)

Perhaps we can do something similar for the JSON/BSON support we are discussing in #7845

[Folyd]

Thanks @alamb.

Here is my example of handling geometry parquet data: apache/arrow-rs#4945
There are Geoparquet format in the community: https://geoparquet.org/
Also see: https://getindata.com/blog/introducing-geoparquet-data-format/

[wjones127]

Since it hasn't been mentioned yet, I'd add there is already a project for Arrow extension types for geospatial data: https://github.com/geoarrow/geoarrow/blob/main/extension-types.md

This is related to the GeoParquet project.

[alamb]

Thanks @wjones127 -- I had forgotten about extension types.

Maybe we could add support for extension types in DataFusion's core and use that extension point to implement a geospatial package on top of DataFusion 🤔

Having a good first use case (Geospatial and possible JSON) to drive the requirements seems like a good idea.

If you agree, I can try and write up a larger project description

[yukkit]

@alamb I have the same requirement as well and hope to initiate it as soon as possible. If possible, I can also contribute code for this.

[alamb]

@alamb I have the same requirement as well and hope to initiate it as soon as possible. If possible, I can also contribute code for this.

That is great news @yukkit -- I don't think I have the bandwidth to try and organize an effort to add Geospatial support to DataFusion in the near term. I wonder if anyone has the bandwidth to help organize an effort to add extension type support? I don't know enough about how this works to really do so without additional research, and sadly I don't have the time at the moment to devote there

[yukkit]

@alamb Ok, if possible, I plan to support UDT (user-defined type) in datafusion, I will paste my ideas in the next few days for anyone to discuss

[alamb]

I would love to see a design proposal for user defined types. ❤️ -- thank you!

[yukkit]

I would love to see a design proposal for user defined types. ❤️ -- thank you!

Of course, it's absolutely essential!

[kylebarron]

My goal is to enable spatial support in projects such as datafusion via https://github.com/geoarrow/geoarrow-rs

[kylebarron]

I'd argue that spatial data support is pretty much blocked until datafusion has support for user-defined types, since it's pretty crucial to pass along GeoArrow metadata, so it's really exciting to see #11513 / #11160 !