[SPARK-53922][Geo][SQL] Introduce physical Geometry and Geography types #52629
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
uros-db
commented
Oct 16, 2025
Contributor
Author
uros-db
left a comment
uros-db
left a comment
There was a problem hiding this comment.
@mkaravel @szehon-ho Please review.
szehon-ho
reviewed
Oct 21, 2025
Member
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
looks mostly good, some comment/questions
common/unsafe/src/main/java/org/apache/spark/unsafe/types/GeographyVal.java
Show resolved
Hide resolved
common/unsafe/src/main/java/org/apache/spark/unsafe/types/GeometryVal.java
Outdated
Show resolved
Hide resolved
sql/core/src/test/scala/org/apache/spark/sql/types/GeographyTypeSuite.scala
Show resolved
Hide resolved
szehon-ho
approved these changes
Oct 22, 2025
cloud-fan
approved these changes
Oct 23, 2025
Contributor
|
thanks, merging to master! |
This was referenced Oct 24, 2025
cloud-fan
pushed a commit
that referenced
this pull request
Oct 28, 2025
…ross Catalyst ### What changes were proposed in this pull request? Added Geography and Geometry accessors to core row/column interfaces, extended codegen and physical type handling to properly recognize geospatial types, enabled writing/reading of Geography and Geometry values in unsafe writer, and added other necessary plumbing for Geography and Geometry in projection/row utilities in order to thread through the new accessors. Note that the GEOMETRY and GEOGRAPHY physical types were recently included to Spark SQL as part of: #52629. ### Why are the changes needed? To provide first-class support for GEOGRAPHY and GEOMETRY within Catalyst. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new tests to: - `GenerateUnsafeProjectionSuite.scala` - `UnsafeRowWriterSuite.scala` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52723 from uros-db/geo-interfaces. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
cloud-fan
pushed a commit
that referenced
this pull request
Oct 28, 2025
…cution classes ### What changes were proposed in this pull request? Introduce internal server-side `Geography` and `Geometry` execution classes in catalyst. Note that the corresponding low-level physical holders (`GeographyVal` and `GeometryVal`) for geospatial types have been previously added as part of #52629. ### Why are the changes needed? Establishing a clear internal execution layer for geospatial operations in catalyst and unblocking downstream work for implementing built-in ST functions and geospatial storage support. ### Does this PR introduce _any_ user-facing change? No. This PR does not introduce any new public API, catalyst expressions, nor user-facing SQL functions. Those will be added in the future. ### How was this patch tested? Added new Java test suites for the execution classes: - `GeographyExecutionSuite` - `GeometryExecutionSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52737 from uros-db/geo-server-classes. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
huangxiaopingRD
pushed a commit
to huangxiaopingRD/spark
that referenced
this pull request
Nov 25, 2025
### What changes were proposed in this pull request? Introduce two new physical types to Spark: - `PhysicalGeographyType` - `PhysicalGeometryType` This PR also adds appropriate mapping from the logical geospatial types (introduced in: apache#52491) to the new physical types. ### Why are the changes needed? Extending the implementation of GEOMETRY and GEOGRAPHY types in Spark, laying the groundwork for full geospatial data type support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new tests to: - `GeographyValSuite` - `GeometryValSuite` Also, added appropriate test cases to: - `GeographyTypeSuite` - `GeographyTypeSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52629 from uros-db/geo-physical-types. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
huangxiaopingRD
pushed a commit
to huangxiaopingRD/spark
that referenced
this pull request
Nov 25, 2025
…ross Catalyst ### What changes were proposed in this pull request? Added Geography and Geometry accessors to core row/column interfaces, extended codegen and physical type handling to properly recognize geospatial types, enabled writing/reading of Geography and Geometry values in unsafe writer, and added other necessary plumbing for Geography and Geometry in projection/row utilities in order to thread through the new accessors. Note that the GEOMETRY and GEOGRAPHY physical types were recently included to Spark SQL as part of: apache#52629. ### Why are the changes needed? To provide first-class support for GEOGRAPHY and GEOMETRY within Catalyst. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added new tests to: - `GenerateUnsafeProjectionSuite.scala` - `UnsafeRowWriterSuite.scala` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52723 from uros-db/geo-interfaces. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
huangxiaopingRD
pushed a commit
to huangxiaopingRD/spark
that referenced
this pull request
Nov 25, 2025
…cution classes ### What changes were proposed in this pull request? Introduce internal server-side `Geography` and `Geometry` execution classes in catalyst. Note that the corresponding low-level physical holders (`GeographyVal` and `GeometryVal`) for geospatial types have been previously added as part of apache#52629. ### Why are the changes needed? Establishing a clear internal execution layer for geospatial operations in catalyst and unblocking downstream work for implementing built-in ST functions and geospatial storage support. ### Does this PR introduce _any_ user-facing change? No. This PR does not introduce any new public API, catalyst expressions, nor user-facing SQL functions. Those will be added in the future. ### How was this patch tested? Added new Java test suites for the execution classes: - `GeographyExecutionSuite` - `GeometryExecutionSuite` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52737 from uros-db/geo-server-classes. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Introduce two new physical types to Spark:
PhysicalGeographyTypePhysicalGeometryTypeThis PR also adds appropriate mapping from the logical geospatial types (introduced in: #52491) to the new physical types.
Why are the changes needed?
Extending the implementation of GEOMETRY and GEOGRAPHY types in Spark, laying the groundwork for full geospatial data type support.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added new tests to:
GeographyValSuiteGeometryValSuiteAlso, added appropriate test cases to:
GeographyTypeSuiteGeographyTypeSuiteWas this patch authored or co-authored using generative AI tooling?
No.