Latest versions of proj share data are huge

[jeroen]

The binary packages for windows and mac include a copy of the proj-data. However as of proj-data-1.9.tar.gz (included with proj 9) this data has grown a lot in size, it is now over 500mb. I think we can no longer include a full copy with the R package.

What is the best way to approach this? Can you identify a subset of the data that we should be including with sf/rgdal?

[rsbivand]

Ignore these archives, sf and rgdal provide access to the on-demand content download network as a more efficient alternative. Does that sound OK? @edzer @rhijmans (for terra).

[edzer]

Yes.

[jeroen]

Oh i was not aware of that. Both the windows and mac binary packages from CRAN still include a proj dir. So the entire dance with --with-data-copy and PROJ_GDAL_DATA_COPY is now redundant?

[rsbivand]

No, still needed, but it should only include what is built when PROJ is built, especially proj.db. Other packagers are muddled, OSGeo4W includes all 600MB, Fedora splits the 600MB into country parts, pyproj bundles it all too, as does the Windows GRASS stand-alone.

[jeroen]

OK thank you for the clarification.

[rsbivand]

For the record, for PROJ 9 installed from source on Fedora 35 I have:

$ ls -l /usr/local/share/proj
total 7960
-rw-r--r--. 1 root root    1183 Feb 15 21:08 CH
-rw-r--r--. 1 root root   17671 Feb 15 21:08 deformation_model.schema.json
-rw-r--r--. 1 root root     728 Feb 15 21:08 GL27
-rw-r--r--. 1 root root    2099 Feb 15 21:08 ITRF2000
-rw-r--r--. 1 root root    5680 Feb 15 21:08 ITRF2008
-rw-r--r--. 1 root root    3468 Feb 15 21:08 ITRF2014
-rw-r--r--. 1 root root   19535 Feb 15 21:08 nad27
-rw-r--r--. 1 root root   16593 Feb 15 21:08 nad83
-rw-r--r--. 1 root root    6385 Feb 15 21:08 nad.lst
-rw-r--r--. 1 root root    3915 Feb 15 21:08 other.extra
-rw-r--r--. 1 root root 7995392 Feb 16 10:06 proj.db
-rw-r--r--. 1 root root    1050 Feb 15 21:08 proj.ini
-rw-r--r--. 1 root root   32662 Feb 15 21:08 projjson.schema.json
-rw-r--r--. 1 root root    8403 Feb 15 21:08 triangulation.schema.json
-rw-r--r--. 1 root root    7079 Feb 15 21:08 world
$ proj
Rel. 9.0.0, March 1st, 2022

[rsbivand]

This, for example @rhijmans shows that terra::project() can use the grid CDN if the PROJ_NETWORK environment variable is set before terra is loaded:

td <- tempfile()
dir.create(td)
Sys.setenv("PROJ_USER_WRITABLE_DIRECTORY"=td)
Sys.setenv("PROJ_NETWORK"="ON")
f <- system.file("gpkg/b_pump.gpkg", package="sf")
v <- terra::vect(f)
file.size(file.path(td, "cache.db"))
p <- terra::project(v, "OGC:CRS84")
file.size(file.path(td, "cache.db"))
terra::ext(p)
v1 <- sf::st_read(f, quiet=TRUE)
sf::st_bbox(sf::st_transform(v1, "OGC:CRS84"))
o1 <- sf::sf_proj_pipelines("EPSG:27700", "OGC:CRS84")
(o1_best <- o1$definition[which.min(o1$accuracy)])
sf::st_bbox(sf::st_transform(v1, "OGC:CRS84", pipeline=o1_best))

I haven't yet debugged the small difference between the automated and explicit pipeline choice in sf. This also works with just Sys.setenv("PROJ_NETWORK"="ON"), and should work when setting the network on in share/proj/proj.ini. I haven't yet tried on flight-mode to see how PROJ reacts through curl to no network. @jeroen: it is OK to drop the inclusion of the bulky grid files, CDN works in sf, terra and rgdal. @edzer I'll try to dig up the example from the Toronto CN Tower (NB Feb 21):

#Sys.setenv("PROJ_NETWORK"="ON")
library(sf)
sf_proj_search_paths()
sf_proj_network()
sf_proj_network(TRUE)
sf_proj_network()
CN <- st_point (c(-79.387139, 43.642567))
CN_sfc <- st_sfc(CN, crs = "OGC:CRS84")
(CN_sfc_proj <- st_transform(CN_sfc, "EPSG:2019"))
CN_sfc_proj1 <- CN_sfc_proj
st_crs(CN_sfc_proj1) <- st_crs(st_crs(CN_sfc_proj)$proj4string)
(good <- st_transform(CN_sfc_proj, "OGC:CRS84"))
(bad <- st_transform(CN_sfc_proj1, "OGC:CRS84"))
library(mapview)
mapviewOptions(fgb = FALSE)
mapview(CN_sfc) + mapview(good) + mapview(bad)

This case is instructive because you need the active CDN to get better than ballpark accuracy, as there are no Helmert or other transformation variants available in proj.db lookup, unlike the London Broad Street pump example above:

o <- sf::sf_proj_pipelines("EPSG:2019", "OGC:CRS84")
o$accuracy
o$definition

[rsbivand]

In terra, one starts the CDN by Sys.setenv("PROJ_NETWORK"="ON") before loading terra:

Sys.setenv("PROJ_NETWORK"="ON")
library(terra)
CN <- vect(matrix(c(-79.387139, 43.642567), ncol=2), crs = "OGC:CRS84")
crds(project(CN, "EPSG:2019"))
#             x       y
# [1,] 313887.1 4833409

library(terra)
CN <- vect(matrix(c(-79.387139, 43.642567), ncol=2), crs = "OGC:CRS84")
crds(project(CN, "EPSG:2019"))
#             x       y
# [1,] 313905.7 4833415