Open Street Map Import

Creating base maps for Weather data visualization

Motivation

We have some specific needs for when various data sets are shown to users. Those are:

1. Roads at a lower zoom (more zoomed out) - Vaisala Xweather provides best in the world road weather modeling and having an overview available in a map is beneficial
2. Electrical Infrastructure - We help protect and model various parts of energy infrastructure and we need to show this information in a visual way.
3. Provide a way to create high-resolution spatial subsets of data. For example, from zoom 8 to 14 for a specific area.
4. Import other data sets with different licensing, like Natural Earth

Key Concepts

Following our standard Xweather approach - how can we make data make sense at each processing step?

1. Importing data into Postgres
   1. Prefix data with source ID like osm_, natearth_, etc
   2. Analyze data - use the THEMEPARK_DEBUG=on flag to import all properties into the DB into an HSTORE type hash column (all keys and values are strings! Value can be NULL)
      1. For each database, iterate over each key/value pair in the tags HSTORE column and run stats on it - # of unique values, number of nulls, top 10 frequent values
   3. Iterate and decide which values to store directly in the DB for this application
2. Create QGIS viewing style to view all layers and properties
3. Create Tegola views for each use case
4. Run pmtiles import to pull from Tegola and create an output pmtiles for the map.

Under the hood, this is a Postgres database. Features with different values we want to keep, need to go into different tables. When querying features back out, we can transform that data further, only selecting what we want at each step.

Tools

There are a lot of tools, we have found:

1. osm2pgsql - Seemingly recommended by Open Street Maps group, imports data to a postgres database
2. planetiler - A large java project that dumps directly to pmtiles. It's fast, goes right into tiles
   1. pmtiles uses this for their generation https://github.com/protomaps/basemaps
3. imposm3 - Dies during large global imports, has Go segfaults

Tile Schemas

OpenStreetMap data can be presented in any way, it's just polygons, lines, and points with metadata. There are some existing conventions on how to present that data, however:

1. OpenMapTiles Layer Schema (OMT)

   1. Layer list:
      1. aerodrome_label
      2. aeroway
      3. boundary
      4. building
      5. globallandcover
      6. housenumber
      7. landcover
      8. landuse
      9. mountain_peak
      10. park
      11. place
      12. poi
      13. transportation
      14. transportation_name
      15. water
      16. water_name
      17. waterway
   2. An example Tegola file: https://github.com/dechristopher/tegola-omt/blob/master/config.toml
   3. Seth's Opinion
      1. things they do well:
         1. Water stuff - putting together oceans + lakes, separating out streams into waterway
         2. landcover and landuse - see https://oceanservice.noaa.gov/facts/lclu.html
         3. aeroway is everything airports/heliports
         4. globallandcover is a nice zoomed out representation of forest, scrub, farmland
         5. mountainpeak - Ridges and points identifying mountain features
         6. Names for busy layers, like transportation, are separated into a different layer
      2. sillythings
         1. housenumber is a bad name, it's addresses
         2. aerowaydromes (airport areas) are really landuse aeren't they?
         3. parks should be landuse
         4. mountain_peak could be a sub type in poi
         5. Place vs poi is confusing

2. Tilezen Vector Tile Format

   1. This is a popular format like on Pmtiles' Demo Page

3. Shortbread Layer Standard v1

   1. This is a published spec of how the data is separated and layered, a spec is better than no spec.
   2. Seth's Opinion - These guys split all sorts of things in kind of weird ways...

4. Mapbox Streets

   1. Layer List:
      1. admin
      2. aeroway
      3. airport_label
      4. building
      5. depth
      6. hillshade
      7. housenum_label
      8. landcover
      9. landuse
      10. landuse_overlay - Just national-park
      11. motorway_junction
      12. natural_label
      13. place_label
      14. poi_label
      15. road
      16. structure
      17. transit_stop_label
      18. water
      19. waterway

Additional tools

1. Tegola - Query for layers from a postgis database and organize layers

TODO

• Make a gen for larger buildings
• Make a gen for sites
• Move terminal to buildings
• Move aerodrome pavements and stuff to "land", taxiways and stuff