Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PARQUET-2417: Add support for geometry logical type #2971

Open
wants to merge 57 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
6a2e051
PARQUET-2471: Add support for geometry logical type
zhangfengcdt Jul 23, 2024
d354012
fix types
zhangfengcdt Jul 23, 2024
969b696
Refactor BoundingBox and GeometryTypes
zhangfengcdt Jul 23, 2024
87ee8ea
revert naming changes
zhangfengcdt Jul 24, 2024
d67e03b
revert TestDecimalUtils
zhangfengcdt Jul 24, 2024
ccf1c4a
revert more
zhangfengcdt Jul 24, 2024
35342c2
refactor statistics
zhangfengcdt Jul 24, 2024
bcfefb7
modify EnvelopeCovering
zhangfengcdt Jul 24, 2024
cf615f6
add more unit tests
zhangfengcdt Jul 25, 2024
c6ae733
update comments
zhangfengcdt Jul 25, 2024
80a629e
add comment for envelope converging expand calculation
zhangfengcdt Jul 26, 2024
e0ec9ef
Fix the boundingbox initial values in constructor
zhangfengcdt Jul 29, 2024
7c728b1
Update the poc implementation for the changes to the spec
zhangfengcdt Aug 7, 2024
fa36d06
remove print
zhangfengcdt Aug 7, 2024
2a1c62a
implement evelop covering for spherical coordinates
zhangfengcdt Aug 8, 2024
d0e7d3d
throw a not-implemented exception for the covering statistics when th…
zhangfengcdt Aug 12, 2024
30d64be
Merge branch 'master' of github.com:apache/parquet-java into feature-…
zhangfengcdt Aug 12, 2024
29a86b5
remove unused comment codes
zhangfengcdt Aug 12, 2024
0c4b8b4
address some review comments
zhangfengcdt Aug 19, 2024
a56e9ad
revert changes that are not desired
zhangfengcdt Aug 20, 2024
1ae3d99
refactor toString and remove test scope of jts-core in parquet-hadoop…
zhangfengcdt Aug 20, 2024
198642d
refactor the converings to be map to avoid ordering issues in stats m…
zhangfengcdt Aug 23, 2024
0cf2de9
address review comments
zhangfengcdt Sep 13, 2024
4f0f7ed
fix formating
zhangfengcdt Sep 13, 2024
a378a9a
address comments to remove string parsing (to be consistent with spec)
zhangfengcdt Sep 23, 2024
698325a
update according to the changes to the upstream pqrquet-format pr
zhangfengcdt Oct 17, 2024
d65ba8e
remove coverings
zhangfengcdt Oct 17, 2024
1b0f5b9
add GeometryStatistics to ColumnMetaData
zhangfengcdt Oct 17, 2024
342e400
more code cleanup for covering
zhangfengcdt Oct 17, 2024
3cc9b68
add toParquetGeometryStatistics
zhangfengcdt Oct 18, 2024
dc05cfd
fix check errors
zhangfengcdt Oct 18, 2024
6f1d586
Merge branch 'master' of github.com:apache/parquet-java into feature-…
zhangfengcdt Nov 13, 2024
e4e3cae
Merge branch 'master' of github.com:apache/parquet-java into feature-…
zhangfengcdt Feb 7, 2025
69d950f
change and remove the encoding and edges from geometry type (spec cha…
zhangfengcdt Feb 7, 2025
d296c6a
fix unit tests
zhangfengcdt Feb 7, 2025
93e28b7
handle the wraparound case for X values
zhangfengcdt Feb 8, 2025
e688d05
support GEOGRAPHY type
zhangfengcdt Feb 8, 2025
01ac560
revert import changes
zhangfengcdt Feb 8, 2025
f9585cd
address pr review comments
zhangfengcdt Feb 19, 2025
b805cf4
fix formatting issue
zhangfengcdt Feb 19, 2025
6aa7028
refactor geography logic type
zhangfengcdt Feb 19, 2025
4536e5f
revert the edge algorithm to use string to avoid loop dependency
zhangfengcdt Feb 19, 2025
f188341
add enum EdgeInterpolationAlgorithm for the geography edge algorithm
zhangfengcdt Feb 21, 2025
214fd39
address pr comments: rename, hash, and equal
zhangfengcdt Feb 21, 2025
907acce
remove column index changes
zhangfengcdt Feb 21, 2025
ef83a2e
revert unnecessary changes
zhangfengcdt Feb 21, 2025
c5d542b
refactor GeospatialStatistics
zhangfengcdt Feb 24, 2025
c1fb37b
refactor writers
zhangfengcdt Feb 24, 2025
dac9ad0
refactor to use colomn value collector instead of binary statistics
zhangfengcdt Feb 24, 2025
e0a319e
revert getLogicalTypeAnnotation
zhangfengcdt Feb 24, 2025
4fd177d
fix TestColumnChunkPageWriteStore
zhangfengcdt Feb 24, 2025
9f0482e
remove DummyBoundingBox
zhangfengcdt Feb 24, 2025
ebd45a9
handle noop builder and null values
zhangfengcdt Feb 25, 2025
0114701
refactor normalizeLongitude
zhangfengcdt Feb 25, 2025
4cd8dfd
add ShowGeospatialStatisticsCommand and tests
zhangfengcdt Feb 25, 2025
5ec6546
add shouldNormalizeLongitude
zhangfengcdt Feb 28, 2025
23af52f
add allowWraparound
zhangfengcdt Mar 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add comment for envelope converging expand calculation
zhangfengcdt committed Jul 26, 2024
commit 80a629e345173649e5b86272f6b6a20c06fdf949
Original file line number Diff line number Diff line change
@@ -49,8 +49,17 @@ void update(Geometry geom) {
Geometry existingGeometry = reader.read(geometry.array());
Envelope existingEnvelope = existingGeometry.getEnvelopeInternal();
Envelope newEnvelope = geom.getEnvelopeInternal();

// The following expandToInclude method works correctly for planar coordinate systems.
// It simply extends the existing envelope to include the new envelope by adjusting
// the min and max values of x and y coordinates.
existingEnvelope.expandToInclude(newEnvelope);

// However, this approach is not accurate for spherical coordinate systems.
// In spherical coordinates, the Earth's curvature needs to be taken into account,
// and simple min/max bounds may not correctly represent the spatial extent.
// More sophisticated geodetic calculations are required for spherical envelopes.

Geometry envelopePolygon = createPolygonFromEnvelope(existingEnvelope);

geometry = ByteBuffer.wrap(writer.write(envelopePolygon));