[SPARK-53760][Geo][SQL] Introduce GeometryType and GeographyType #52491

uros-db · 2025-09-30T15:21:13Z

What changes were proposed in this pull request?

Introduce two new logical types to Spark:

GeographyType
GeometryType

This PR also adds appropriate SQL parsing and JSON serialization logic for the new types.

Why are the changes needed?

Kicking off https://issues.apache.org/jira/browse/SPARK-51658 by adding GEOMETRY and GEOGRAPHY types to Spark.

Does this PR introduce any user-facing change?

Yes, two new logical data types are added as Experimental.

How was this patch tested?

Added new tests to:

GeographyTypeSuite
GeometryTypeSuite

Also, added appropriate test cases to:

DataTypeSuite

Was this patch authored or co-authored using generative AI tooling?

No.

uros-db

@mkaravel Please review.

szehon-ho

Thanks @uros-db for kicking it off! left some comments/questions as i read it

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

sql/api/src/main/scala/org/apache/spark/sql/types/CrsMappings.java

sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeometryType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeometryType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

uros-db · 2025-10-07T08:14:40Z

Hi folks, please re-review. @szehon-ho @mkaravel

mkaravel

Generally LGTM. Please take a look at the comments I posted.
Thank you for this PR!

common/utils/src/main/resources/error/error-conditions.json

sql/api/src/main/scala/org/apache/spark/sql/types/SpatialReferenceMapper.java

mkaravel

@uros-db Thank you for addressing the comments!

LGTM.

uros-db · 2025-10-09T17:54:21Z

@cloud-fan @szehon-ho Please review. Also cc @srielau (we have some new error classes here).

szehon-ho

lgtm, be nice to document the expectation of the GeographyType and GeometryType argument (what is expected, not expected and only internal), as its a user facing API: ref: #52491 (comment)

jiayuasu · 2025-10-10T05:33:21Z

Great work. The Apache Sedona team was not aware of or informed by this PR. We are now looking into this.

jiayuasu · 2025-10-10T05:39:57Z

We will review and comment on this PR. This is to ensure it does not create incompatibility with Apache Sedona.

uros-db · 2025-10-10T07:12:21Z

Thank you @jiayuasu. We will hold the merge until you have enough time to review. Please feel free to let me know if you have any questions or concerns.

paleolimbot

Very cool! Just a few notes from the GeoArrow/Parquet compatibility angle where most CRSes are written by spatial tools with a PROJJSON or authority:code CRS.

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

uros-db · 2025-10-13T13:15:19Z

Hi folks @paleolimbot @jiayuasu, do you have any other concerns about this PR? cc @cloud-fan

dongjoon-hyun · 2025-10-15T03:16:23Z

Thank you, @uros-db , @cloud-fan and all.

…patial types ### What changes were proposed in this pull request? - Fix the `ST_INVALID_SRID_VALUE` error message. - Fix the `SpatialType` doc comment. ### Why are the changes needed? Followup PR to clean up original issues introduced in: #52491. ### Does this PR introduce _any_ user-facing change? Yes, the `ST_INVALID_SRID_VALUE` error message is updated. ### How was this patch tested? Existing tests are sufficient. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52632 from uros-db/geo-logical-types-FOLLOWUP. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…o PySpark API ### What changes were proposed in this pull request? Introduce two new geospatial data types to PySpark API: - `GeographyType` - `GeometryType` This PR also adds appropriate JSON serialization logic for the new types in PySpark. Note that the GEOMETRY and GEOGRAPHY logical types were recently included to Spark SQL as part of: #52491. ### Why are the changes needed? Expanding on GEOMETRY and GEOGRAPHY type support across all of the supported APIs. ### Does this PR introduce _any_ user-facing change? Yes, two new data types are now available to users of the PySpark API. ### How was this patch tested? Added new tests to: - `test_geographytype.py` - `test_geometrytype.py` Also, added appropriate test cases to: - `test_types.py` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52627 from uros-db/geo-python-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

### What changes were proposed in this pull request? Introduce two new physical types to Spark: - `PhysicalGeographyType` - `PhysicalGeometryType` This PR also adds appropriate mapping from the logical geospatial types (introduced in: #52491) to the new physical types. ### Why are the changes needed? Extending the implementation of GEOMETRY and GEOGRAPHY types in Spark, laying the groundwork for full geospatial data type support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new tests to: - `GeographyValSuite` - `GeometryValSuite` Also, added appropriate test cases to: - `GeographyTypeSuite` - `GeographyTypeSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52629 from uros-db/geo-physical-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…ava API ### What changes were proposed in this pull request? Introduce two new geospatial data types to Java API: - `GeographyType` - `GeometryType` Note that the GEOMETRY and GEOGRAPHY logical types were recently included to Spark SQL as part of: #52491. ### Why are the changes needed? Expanding on GEOMETRY and GEOGRAPHY type support across all of the supported APIs. ### Does this PR introduce _any_ user-facing change? Yes, two new data types are now available to users of the Java API. ### How was this patch tested? Added new tests to: - `JavaGeographyTypeSuite` - `JavaGeometryTypeSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52623 from uros-db/geo-java-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…ReferenceSystemMapper ### What changes were proposed in this pull request? Extend the `SpatialReferenceSystemMapper` class to offer SRID <-> CRS mappings for both `GeographyType` and `GeometryType`. The `SpatialReferenceSystemMapper` class was introduced originally as part of: #52491. ### Why are the changes needed? Avoid manual checks for GEOGRAPHY type, and use the centralized SRS mapping logic. Also, this PR will make it easier to support additional SRID/CRS values for spatial types soon. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new tests to: - `SpatialReferenceSystemMapperSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52667 from uros-db/geo-srsMapper. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…n PySpark ### What changes were proposed in this pull request? Implement the Spatial Reference System Mapper classes to offer SRID <-> CRS mappings in PySpark, for both `GeographyType` and `GeometryType`. On JVM / Scala side, the `SpatialReferenceSystemMapper` class was introduced originally as part of: #52491, and then subsequently extended in: #52667. ### Why are the changes needed? Use centralized SRS mapping logic in PySpark, same as we currently do in Scala types. Also, this PR will make it easier to support additional SRID/CRS values for spatial types soon. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Updated tests in: - `test_geographytype` - `test_geometrytype` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52799 from uros-db/geo-srsMapper-python. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

…er data types ### What changes were proposed in this pull request? This PR disallows casting `GEOGRAPHY` and `GEOMETRY` to/from other data types in the system (such as `BOOLEAN`, `BINARY`, `STRING`, and others). Note that these types were recently introduced as part of: #52491. ### Why are the changes needed? `GeographyType` and `GeometryType` are not interoperable with any other non-geospatial data types, so we need to forbid such casts explicitly in order to avoid unexpected failures. ### Does this PR introduce _any_ user-facing change? Yes, casts are disallowed for newly introduced geospatial types. ### How was this patch tested? Added new unit tests to verify the expected behaviour: - `CastWithAnsiOffSuite` - `CastWithAnsiOnSuite` Added new appropriate end-to-end SQL tests: - `st-functions` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52806 from uros-db/geo-cast-disallow. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? Introduce two new logical types to Spark: - `GeographyType` - `GeometryType` This PR also adds appropriate SQL parsing and JSON serialization logic for the new types. ### Why are the changes needed? Kicking off https://issues.apache.org/jira/browse/SPARK-51658 by adding GEOMETRY and GEOGRAPHY types to Spark. ### Does this PR introduce _any_ user-facing change? Yes, two new logical data types are added as `Experimental`. ### How was this patch tested? Added new tests to: - `GeographyTypeSuite` - `GeometryTypeSuite` Also, added appropriate test cases to: - `DataTypeSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52491 from uros-db/geo-logical-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…patial types ### What changes were proposed in this pull request? - Fix the `ST_INVALID_SRID_VALUE` error message. - Fix the `SpatialType` doc comment. ### Why are the changes needed? Followup PR to clean up original issues introduced in: apache#52491. ### Does this PR introduce _any_ user-facing change? Yes, the `ST_INVALID_SRID_VALUE` error message is updated. ### How was this patch tested? Existing tests are sufficient. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52632 from uros-db/geo-logical-types-FOLLOWUP. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…o PySpark API ### What changes were proposed in this pull request? Introduce two new geospatial data types to PySpark API: - `GeographyType` - `GeometryType` This PR also adds appropriate JSON serialization logic for the new types in PySpark. Note that the GEOMETRY and GEOGRAPHY logical types were recently included to Spark SQL as part of: apache#52491. ### Why are the changes needed? Expanding on GEOMETRY and GEOGRAPHY type support across all of the supported APIs. ### Does this PR introduce _any_ user-facing change? Yes, two new data types are now available to users of the PySpark API. ### How was this patch tested? Added new tests to: - `test_geographytype.py` - `test_geometrytype.py` Also, added appropriate test cases to: - `test_types.py` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52627 from uros-db/geo-python-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

### What changes were proposed in this pull request? Introduce two new physical types to Spark: - `PhysicalGeographyType` - `PhysicalGeometryType` This PR also adds appropriate mapping from the logical geospatial types (introduced in: apache#52491) to the new physical types. ### Why are the changes needed? Extending the implementation of GEOMETRY and GEOGRAPHY types in Spark, laying the groundwork for full geospatial data type support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new tests to: - `GeographyValSuite` - `GeometryValSuite` Also, added appropriate test cases to: - `GeographyTypeSuite` - `GeographyTypeSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52629 from uros-db/geo-physical-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…ava API ### What changes were proposed in this pull request? Introduce two new geospatial data types to Java API: - `GeographyType` - `GeometryType` Note that the GEOMETRY and GEOGRAPHY logical types were recently included to Spark SQL as part of: apache#52491. ### Why are the changes needed? Expanding on GEOMETRY and GEOGRAPHY type support across all of the supported APIs. ### Does this PR introduce _any_ user-facing change? Yes, two new data types are now available to users of the Java API. ### How was this patch tested? Added new tests to: - `JavaGeographyTypeSuite` - `JavaGeometryTypeSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52623 from uros-db/geo-java-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…ReferenceSystemMapper ### What changes were proposed in this pull request? Extend the `SpatialReferenceSystemMapper` class to offer SRID <-> CRS mappings for both `GeographyType` and `GeometryType`. The `SpatialReferenceSystemMapper` class was introduced originally as part of: apache#52491. ### Why are the changes needed? Avoid manual checks for GEOGRAPHY type, and use the centralized SRS mapping logic. Also, this PR will make it easier to support additional SRID/CRS values for spatial types soon. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new tests to: - `SpatialReferenceSystemMapperSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52667 from uros-db/geo-srsMapper. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…n PySpark ### What changes were proposed in this pull request? Implement the Spatial Reference System Mapper classes to offer SRID <-> CRS mappings in PySpark, for both `GeographyType` and `GeometryType`. On JVM / Scala side, the `SpatialReferenceSystemMapper` class was introduced originally as part of: apache#52491, and then subsequently extended in: apache#52667. ### Why are the changes needed? Use centralized SRS mapping logic in PySpark, same as we currently do in Scala types. Also, this PR will make it easier to support additional SRID/CRS values for spatial types soon. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Updated tests in: - `test_geographytype` - `test_geometrytype` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52799 from uros-db/geo-srsMapper-python. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

…er data types ### What changes were proposed in this pull request? This PR disallows casting `GEOGRAPHY` and `GEOMETRY` to/from other data types in the system (such as `BOOLEAN`, `BINARY`, `STRING`, and others). Note that these types were recently introduced as part of: apache#52491. ### Why are the changes needed? `GeographyType` and `GeometryType` are not interoperable with any other non-geospatial data types, so we need to forbid such casts explicitly in order to avoid unexpected failures. ### Does this PR introduce _any_ user-facing change? Yes, casts are disallowed for newly introduced geospatial types. ### How was this patch tested? Added new unit tests to verify the expected behaviour: - `CastWithAnsiOffSuite` - `CastWithAnsiOnSuite` Added new appropriate end-to-end SQL tests: - `st-functions` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52806 from uros-db/geo-cast-disallow. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

Initial commit

3985d5a

github-actions bot added the SQL label Sep 30, 2025

uros-db commented Sep 30, 2025

View reviewed changes

uros-db added 2 commits October 1, 2025 04:35

Fix scalastyle

1124ff5

Regenerate golden files

3fa63c8

szehon-ho reviewed Oct 1, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala Show resolved Hide resolved

uros-db added 4 commits October 6, 2025 11:20

Address comments

4be6774

Run scalafmt

fcae2a1

Merge branch 'apache:master' into geo-logical-types

fa62491

Refactor classes

e2dfe8e

uros-db requested a review from szehon-ho October 6, 2025 11:08

Fix SQL keywords

e6ceef9

github-actions bot added the DOCS label Oct 6, 2025

uros-db added 3 commits October 6, 2025 17:30

Fix failing tests

3085dc0

Cleanup comment

a25550d

Fix Java lint

27fb2d2

mkaravel reviewed Oct 9, 2025

View reviewed changes

Address comments

b777086

uros-db requested a review from mkaravel October 9, 2025 17:51

mkaravel approved these changes Oct 9, 2025

View reviewed changes

szehon-ho approved these changes Oct 9, 2025

View reviewed changes

paleolimbot reviewed Oct 10, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala Show resolved Hide resolved

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala Outdated Show resolved Hide resolved

uros-db requested a review from paleolimbot October 10, 2025 17:21

cloud-fan closed this in 8ff9feb Oct 15, 2025

uros-db mentioned this pull request Oct 20, 2025

[SPARK-53957][Geo][SQL] Support GEOGRAPHY and GEOMETRY in the SpatialReferenceSystemMapper #52667

Closed

This was referenced Oct 30, 2025

[SPARK-54096][Geo][PYTHON] Support Spatial Reference System mapping in PySpark #52799

Closed

[SPARK-54104][Geo][SQL] Disallow casting geospatial types to/from other data types #52806

Closed

[SPARK-53760][Geo][SQL] Introduce GeometryType and GeographyType #52491

[SPARK-53760][Geo][SQL] Introduce GeometryType and GeographyType #52491

Uh oh!

Conversation

uros-db commented Sep 30, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

uros-db left a comment

Choose a reason for hiding this comment

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uros-db commented Oct 7, 2025

Uh oh!

mkaravel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkaravel left a comment

Choose a reason for hiding this comment

Uh oh!

uros-db commented Oct 9, 2025

Uh oh!

szehon-ho left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiayuasu commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiayuasu commented Oct 10, 2025

Uh oh!

uros-db commented Oct 10, 2025

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

uros-db commented Oct 13, 2025

Uh oh!

dongjoon-hyun commented Oct 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

szehon-ho left a comment •

edited

Loading

jiayuasu commented Oct 10, 2025 •

edited

Loading