Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GeometryPipeline #1118

Merged
merged 23 commits into from
Dec 15, 2024
Merged

GeometryPipeline #1118

merged 23 commits into from
Dec 15, 2024

Conversation

msbarry
Copy link
Contributor

@msbarry msbarry commented Nov 27, 2024

This PR adds a new GeometryPipeline interface for operations that transform a Geometry to another Geometry and also:

  • add new feature.transformScaledGeometry(GeometryPipeline) which tells planetiler to apply that transform to the feature each time it is scaled to tile coordinates (where 1=width of tile at that zoom level) before slicing the feature into tiles at that zoom
  • add new feature.transformScaledGeometryByZoom((zoom) -> GeometryPipeline) which lets you dynamically change the transform applied to scaled geometries at each zoom level
  • update FeatureMerge mergeLineStrings() and mergeNearbyPolygons to accept a geometry pipeline to apply to each merged feature in tile coordinates (where 256=width of the tile)
  • Adapt DouglasPeuckerSimplifier and VWSimplifier to implement GeometryPipeline
  • Add MidpointSmoother to implement midpoint smoothing
  • Add DualMidpointSmoother to implement Chaikin smoothing
  • Add static factory methods to GeometryPipeline for these common transforms (simplifyDP, simplifyVW, smoothChaikin, smoothMidpoint)

You can also pass simple functions where a GeometryPipeline is expected, for example:

feature.transformScaledGeometry(GeometryFixer::fix)
feature.transformScaledGeometry(Geometry::getCentroid)
feature.transformScaledGeometry(Geometry::getInteriorPoint)

And you can chain transforms using andThen:

feature.transformScaledGeometry(
  GeometryPipeline.simplifyVW(1).setWeight(0.9)
    .andThen(GeometryPipeline.simplifyDP(1))
);

⚠️ transformScaledGeometry replaces default simplification with the provided transform, so you need to manually add simplification back if you want it with .andThen(GeometryPipeline.defaultSimplify(feature))
⚠️ transformScaledGeometry and FeatureMerge work with geometries at different scales (tile width=1 in transformScaledGeometry vs. tile width=256 for FeatureMerge) so you can't reuse a pipeline between feature processing and tile post-processing

Here is a joke example using Visvalingam Whyatt simplification and Chaikin smoothing on OSM ocean polygons:

image

Copy link

github-actions bot commented Nov 27, 2024

This Branch 442dcf3 Base fd7b79a
0:01:11 DEB [archive] - Tile stats:
0:01:11 DEB [archive] - Biggest tiles (gzipped)
1. 14/4942/6092 (157k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:85k)
2. 9/154/190 (144k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k)
3. 10/308/380 (136k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k)
4. 10/308/381 (136k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k)
5. 14/4941/6092 (113k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:65k)
6. 14/4941/6093 (112k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k)
7. 14/4940/6092 (101k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k)
8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k)
9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k)
10. 11/616/761 (95k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k)
0:01:11 DEB [archive] - Max tile sizes
                      z0    z1    z2    z3    z4    z5    z6    z7    z8    z9   z10   z11   z12   z13   z14   all
           boundary  151   336   409   544   872   332   437   552   802  1.6k    2k  6.9k  6.2k  5.6k  4.5k  6.9k
              water 7.7k  3.7k  8.6k  5.5k  2.6k  5.1k   15k   18k   16k   26k   15k   13k   17k   15k   12k   26k
              place    0     0   441   441   441   640   714    1k  1.6k  3.1k  5.8k  3.4k  1.7k   803   948  5.8k
            landuse    0     0     0     0   549   695  1.6k  6.7k   17k   44k   59k   50k   38k   19k   12k   59k
     transportation    0     0     0     0   311   776  1.2k    4k  5.6k   17k   13k   17k   62k   47k   33k   62k
           waterway    0     0     0     0   112   119     0     0     0    3k  2.3k    2k  2.1k  4.9k  2.4k  4.9k
               park    0     0     0     0     0     0  1.3k  4.3k  9.7k   18k   13k  8.2k  3.7k  3.4k  4.4k   18k
transportation_name    0     0     0     0     0     0   287   364  1.1k  1.9k  5.5k  4.7k  3.9k  3.4k   18k   18k
          landcover    0     0     0     0     0     0     0  9.9k   29k   85k   72k   81k   53k   30k   25k   85k
      mountain_peak    0     0     0     0     0     0     0  1.1k  1.8k  3.4k  4.3k  2.8k  1.4k  1.4k   869  4.3k
         water_name    0     0     0     0     0     0     0     0     0   486   461   433   452  1.2k  1.5k  1.5k
    aerodrome_label    0     0     0     0     0     0     0     0     0     0   666   328   273   221   221   666
            aeroway    0     0     0     0     0     0     0     0     0     0  1.6k  2.1k    3k  3.4k  2.8k  3.4k
                poi    0     0     0     0     0     0     0     0     0     0   585   398   711   646   85k   85k
           building    0     0     0     0     0     0     0     0     0     0     0     0     0   59k   92k   92k
        housenumber    0     0     0     0     0     0     0     0     0     0     0     0     0     0   35k   35k
          full tile 7.9k    4k  9.5k  6.4k  3.7k    6k   20k   41k   82k  195k  182k  135k  113k  128k  247k  247k
            gzipped 6.2k  3.5k  7.1k  5.2k  3.1k  4.8k   14k   29k   59k  144k  136k   99k   83k   92k  157k  157k
0:01:11 DEB [archive] -    Max tile: 247k (gzipped: 157k)
0:01:11 DEB [archive] -    Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic
0:01:11 DEB [archive] -     # tiles: 4,115,039
0:01:11 DEB [archive] -  # features: 5,519,144
0:01:11 INF [archive] - Finished in 19s cpu:1m10s avg:3.7
0:01:11 INF [archive] -   read    1x(3% 0.6s wait:17s done:1s)
0:01:11 INF [archive] -   encode  4x(57% 11s wait:2s)
0:01:11 INF [archive] -   write   1x(22% 4s wait:13s)
0:01:11 INF [archive] - Finished in 1m12s cpu:3m39s gc:1s avg:3
0:01:11 INF [archive] - FINISHED!
0:01:11 INF [archive] - 
0:01:11 INF [archive] - ----------------------------------------
0:01:11 INF [archive] - data errors:
0:01:11 INF [archive] - 	render_snap_fix_input	16,733
0:01:11 INF [archive] - 	osm_multipolygon_missing_way	360
0:01:11 INF [archive] - 	osm_boundary_missing_way	55
0:01:11 INF [archive] - 	merge_snap_fix_input	12
0:01:11 INF [archive] - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:01:11 INF [archive] - 	render_snap_fix_input2	1
0:01:11 INF [archive] - 	omt_fix_water_before_ne_intersect	1
0:01:11 INF [archive] - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	1
0:01:11 INF [archive] - 	feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix	1
0:01:11 INF [archive] - ----------------------------------------
0:01:11 INF [archive] - 	overall          1m12s cpu:3m39s gc:1s avg:3
0:01:11 INF [archive] - 	lake_centerlines 4s cpu:6s avg:1.4
0:01:11 INF [archive] - 	  read     1x(11% 0.5s done:4s)
0:01:11 INF [archive] - 	  process  4x(0% 0s done:4s)
0:01:11 INF [archive] - 	  write    1x(0% 0s done:4s)
0:01:11 INF [archive] - 	water_polygons   15s cpu:41s avg:2.8
0:01:11 INF [archive] - 	  read     1x(41% 6s done:7s)
0:01:11 INF [archive] - 	  process  4x(26% 4s wait:4s done:5s)
0:01:11 INF [archive] - 	  write    1x(4% 0.5s wait:9s done:5s)
0:01:11 INF [archive] - 	natural_earth    11s cpu:17s avg:1.6
0:01:11 INF [archive] - 	  read     1x(57% 6s done:5s)
0:01:11 INF [archive] - 	  process  4x(8% 0.8s wait:6s done:5s)
0:01:11 INF [archive] - 	  write    1x(0% 0s wait:6s done:5s)
0:01:11 INF [archive] - 	osm_pass1        2s cpu:6s avg:3.3
0:01:11 INF [archive] - 	  read     1x(2% 0s wait:2s)
0:01:11 INF [archive] - 	  parse    4x(32% 0.6s)
0:01:11 INF [archive] - 	  process  1x(72% 1s)
0:01:11 INF [archive] - 	osm_pass2        19s cpu:1m13s avg:3.9
0:01:11 INF [archive] - 	  read     1x(0% 0s wait:11s done:7s)
0:01:11 INF [archive] - 	  process  4x(77% 14s)
0:01:11 INF [archive] - 	  write    1x(2% 0.4s wait:18s)
0:01:11 INF [archive] - 	ne_lakes         0s cpu:0s avg:0
0:01:11 INF [archive] - 	boundaries       0s cpu:0s avg:1.4
0:01:11 INF [archive] - 	agg_stop         0s cpu:0s avg:0
0:01:11 INF [archive] - 	sort             1s cpu:3s avg:2.6
0:01:11 INF [archive] - 	  worker  1x(49% 0.7s)
0:01:11 INF [archive] - 	archive          19s cpu:1m10s avg:3.7
0:01:11 INF [archive] - 	  read    1x(3% 0.6s wait:17s done:1s)
0:01:11 INF [archive] - 	  encode  4x(57% 11s wait:2s)
0:01:11 INF [archive] - 	  write   1x(22% 4s wait:13s)
0:01:11 INF [archive] - ----------------------------------------
0:01:11 INF [archive] - 	archive	108MB
0:01:11 INF [archive] - 	features	284MB
-rw-r--r-- 1 runner docker 87M Dec 15 12:09 run.jar
0:01:03 DEB [archive] - Tile stats:
0:01:03 DEB [archive] - Biggest tiles (gzipped)
1. 14/4942/6092 (157k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:85k)
2. 9/154/190 (144k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k)
3. 10/308/380 (136k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k)
4. 10/308/381 (135k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k)
5. 14/4941/6092 (113k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:65k)
6. 14/4941/6093 (112k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k)
7. 14/4940/6092 (101k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k)
8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k)
9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k)
10. 11/616/761 (95k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k)
0:01:03 DEB [archive] - Max tile sizes
                      z0    z1    z2    z3    z4    z5    z6    z7    z8    z9   z10   z11   z12   z13   z14   all
           boundary  151   336   409   544   872   332   437   552   802  1.6k    2k  6.9k  6.2k  5.6k  4.5k  6.9k
              water 7.7k  3.7k  8.6k  5.5k  2.6k  5.1k   15k   18k   16k   26k   15k   13k   17k   15k   12k   26k
              place    0     0   441   441   441   640   714    1k  1.6k  3.1k  5.8k  3.4k  1.7k   803   948  5.8k
            landuse    0     0     0     0   549   695  1.6k  6.7k   17k   44k   59k   50k   38k   19k   12k   59k
     transportation    0     0     0     0   311   776  1.2k    4k  5.6k   17k   13k   17k   62k   47k   33k   62k
           waterway    0     0     0     0   112   119     0     0     0    3k  2.3k    2k  2.1k  4.9k  2.4k  4.9k
               park    0     0     0     0     0     0  1.3k  4.3k  9.7k   18k   13k  8.2k  3.7k  3.4k  4.4k   18k
transportation_name    0     0     0     0     0     0   287   364  1.1k  1.9k  5.5k  4.7k  3.9k  3.4k   18k   18k
          landcover    0     0     0     0     0     0     0  9.9k   29k   85k   72k   81k   53k   30k   25k   85k
      mountain_peak    0     0     0     0     0     0     0  1.1k  1.8k  3.4k  4.3k  2.8k  1.4k  1.4k   869  4.3k
         water_name    0     0     0     0     0     0     0     0     0   486   461   433   452  1.2k  1.5k  1.5k
    aerodrome_label    0     0     0     0     0     0     0     0     0     0   666   328   273   221   221   666
            aeroway    0     0     0     0     0     0     0     0     0     0  1.6k  2.1k    3k  3.4k  2.8k  3.4k
                poi    0     0     0     0     0     0     0     0     0     0     0     0   568   565   85k   85k
           building    0     0     0     0     0     0     0     0     0     0     0     0     0   59k   92k   92k
        housenumber    0     0     0     0     0     0     0     0     0     0     0     0     0     0   35k   35k
          full tile 7.9k    4k  9.5k  6.4k  3.7k    6k   20k   41k   82k  195k  182k  135k  113k  127k  247k  247k
            gzipped 6.2k  3.5k  7.1k  5.2k  3.1k  4.8k   14k   29k   59k  144k  136k   99k   83k   92k  157k  157k
0:01:03 DEB [archive] -    Max tile: 247k (gzipped: 157k)
0:01:03 DEB [archive] -    Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic
0:01:03 DEB [archive] -     # tiles: 4,115,039
0:01:03 DEB [archive] -  # features: 5,519,083
0:01:03 INF [archive] - Finished in 19s cpu:1m12s avg:3.7
0:01:03 INF [archive] -   read    1x(3% 0.6s wait:18s done:1s)
0:01:03 INF [archive] -   encode  4x(56% 11s wait:2s done:1s)
0:01:03 INF [archive] -   write   1x(21% 4s wait:13s done:1s)
0:01:03 INF [archive] - Finished in 1m4s cpu:3m29s gc:1s avg:3.3
0:01:03 INF [archive] - FINISHED!
0:01:03 INF [archive] - 
0:01:03 INF [archive] - ----------------------------------------
0:01:03 INF [archive] - data errors:
0:01:03 INF [archive] - 	render_snap_fix_input	16,733
0:01:03 INF [archive] - 	osm_multipolygon_missing_way	360
0:01:03 INF [archive] - 	osm_boundary_missing_way	55
0:01:03 INF [archive] - 	merge_snap_fix_input	12
0:01:03 INF [archive] - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:01:03 INF [archive] - 	render_snap_fix_input2	1
0:01:03 INF [archive] - 	omt_fix_water_before_ne_intersect	1
0:01:03 INF [archive] - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	1
0:01:03 INF [archive] - 	feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix	1
0:01:03 INF [archive] - ----------------------------------------
0:01:03 INF [archive] - 	overall          1m4s cpu:3m29s gc:1s avg:3.3
0:01:03 INF [archive] - 	lake_centerlines 2s cpu:5s avg:2.3
0:01:03 INF [archive] - 	  read     1x(22% 0.5s done:2s)
0:01:03 INF [archive] - 	  process  4x(0% 0s done:2s)
0:01:03 INF [archive] - 	  write    1x(0% 0s done:2s)
0:01:03 INF [archive] - 	water_polygons   14s cpu:38s avg:2.7
0:01:03 INF [archive] - 	  read     1x(43% 6s done:7s)
0:01:03 INF [archive] - 	  process  4x(28% 4s wait:3s done:5s)
0:01:03 INF [archive] - 	  write    1x(4% 0.5s wait:9s done:5s)
0:01:03 INF [archive] - 	natural_earth    6s cpu:13s avg:2
0:01:03 INF [archive] - 	  read     1x(96% 6s)
0:01:03 INF [archive] - 	  process  4x(13% 0.8s wait:6s)
0:01:03 INF [archive] - 	  write    1x(0% 0s wait:6s)
0:01:03 INF [archive] - 	osm_pass1        2s cpu:6s avg:3.3
0:01:03 INF [archive] - 	  read     1x(2% 0s wait:2s)
0:01:03 INF [archive] - 	  parse    4x(33% 0.6s)
0:01:03 INF [archive] - 	  process  1x(70% 1s)
0:01:03 INF [archive] - 	osm_pass2        18s cpu:1m10s avg:4
0:01:03 INF [archive] - 	  read     1x(0% 0s wait:10s done:8s)
0:01:03 INF [archive] - 	  process  4x(77% 14s)
0:01:03 INF [archive] - 	  write    1x(2% 0.4s wait:17s)
0:01:03 INF [archive] - 	ne_lakes         0s cpu:0s avg:12.8
0:01:03 INF [archive] - 	boundaries       0s cpu:0s avg:1.4
0:01:03 INF [archive] - 	agg_stop         0s cpu:0s avg:0
0:01:03 INF [archive] - 	sort             1s cpu:3s avg:2.6
0:01:03 INF [archive] - 	  worker  1x(51% 0.7s)
0:01:03 INF [archive] - 	archive          19s cpu:1m12s avg:3.7
0:01:03 INF [archive] - 	  read    1x(3% 0.6s wait:18s done:1s)
0:01:03 INF [archive] - 	  encode  4x(56% 11s wait:2s done:1s)
0:01:03 INF [archive] - 	  write   1x(21% 4s wait:13s done:1s)
0:01:03 INF [archive] - ----------------------------------------
0:01:03 INF [archive] - 	archive	108MB
0:01:03 INF [archive] - 	features	284MB
-rw-r--r-- 1 runner docker 87M Dec 15 12:10 run.jar

Full logs: https://github.com/onthegomap/planetiler/actions/runs/12338616800

@msbarry msbarry marked this pull request as ready for review November 27, 2024 12:19
@wipfli
Copy link
Contributor

wipfli commented Dec 6, 2024

Amazing, thanks for introducing this! I am playing a bit around with it and post things that come to mind below...

@wipfli
Copy link
Contributor

wipfli commented Dec 6, 2024

First question, if I do

features.polygon("forest")
    .setPixelTolerance(0.0)
    .setMinPixelSize(0.0)
    .transformScaledGeometry(GeometryPipeline.simplifyDP(2.0 / 256));

in the processFeature function, will setPixelTolerance have any effect?

@wipfli
Copy link
Contributor

wipfli commented Dec 6, 2024

Chaikin works nicely

features.polygon("waterway")
    .setPixelTolerance(0.0)
    .transformScaledGeometry(GeometryPipeline.smoothChaikin(3))
    .setMinPixelSize(0.0);

chaikin

@wipfli
Copy link
Contributor

wipfli commented Dec 6, 2024

One thing that I always found a little bit disturbing about Douglas Peucker simplification was that you get these sharp corners all over the place. Would the recommended way to remove those be something like this?

features.polygon("waterway")
    .setPixelTolerance(0.0)
    .transformScaledGeometry(
        GeometryPipeline.simplifyDP(1/256d)
            .andThen(GeometryPipeline.smoothChaikin(1))
    )
    .setMinPixelSize(0.0);
}

@msbarry
Copy link
Contributor Author

msbarry commented Dec 7, 2024

Thanks for taking a look!

First question, if I do

features.polygon("forest")
    .setPixelTolerance(0.0)
    .setMinPixelSize(0.0)
    .transformScaledGeometry(GeometryPipeline.simplifyDP(2.0 / 256));

in the processFeature function, will setPixelTolerance have any effect?

transformScaledGeometry replaces simplification, but I added a utility so you can do pipeline.andThen(GeometryPipeline.defaultSimplify(feature)) if you want to add back in the simplification that would have happened automatically. Do you think that makes sense?

One thing that I always found a little bit disturbing about Douglas Peucker simplification was that you get these sharp corners all over the place. Would the recommended way to remove those be something like this?

features.polygon("waterway")
    .setPixelTolerance(0.0)
    .transformScaledGeometry(
        GeometryPipeline.simplifyDP(1/256d)
            .andThen(GeometryPipeline.smoothChaikin(1))
    )
    .setMinPixelSize(0.0);
}

Yes that could work! You end up adding points back to remove the sharp corners. You could also add points back by just using a smaller DP tolerance, but adding them back with chaikin smoothing results in a globbier shape.

You could also try setMaxVertexArea/setMaxVertexOffset which limit how far chaikin smoothing is allowed to move the smoothed shape away from the original shape, and setMinVertexArea/setMinVertexOffset to stop smoothing when the corners get too small instead of using a fixed number of iterations.

@msbarry
Copy link
Contributor Author

msbarry commented Dec 15, 2024

@wipfli I'm going to merge this now, let me know if you have any feedback using it on main branch and we can adjust if necessary!

@msbarry msbarry merged commit cdd7839 into main Dec 15, 2024
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants