Skip to content

proj-data: add proj-data package and integration support#290643

Draft
imincik wants to merge 1 commit intoNixOS:masterfrom
imincik:proj-data-init2
Draft

proj-data: add proj-data package and integration support#290643
imincik wants to merge 1 commit intoNixOS:masterfrom
imincik:proj-data-init2

Conversation

@imincik
Copy link
Contributor

@imincik imincik commented Feb 22, 2024

Description of changes

  • add proj-data package to proj

  • add optional support for installing proj-data grid files to proj using withProjData passthru function

Implementation notes

AFAIK, installation of proj-data resource files in share/proj directory is the only user-friendly way of enabling them for all programs using proj. Another alternative methods are:

  • setting PROJ_DATA variable to ${proj}/share/proj:${proj-data}
  • adding them to user home directories

which can be very tricky for other applications using proj in containers or multi-user environments and can be error-prone.
For more information, see: https://proj.org/en/9.3/resource_files.html

Installation of proj-data files to proj is optional, because proj-data repository size is large (can reach 1GB of size soon) and proj-data support is not always needed.

Example usage

  • Example of building proj package without proj-data support:
$ nix-build -A proj

ls ./result/share/proj
CH  deformation_model.schema.json  GL27  ITRF2000  ITRF2008  ITRF2014  nad27  nad83  nad.lst  other.extra  proj.db  proj.ini  projjson.schema.json  triangulation.schema.json  world
  • Example of building proj package containing proj-data grids provided by nz_linz:
$ nix-build -I nixpkgs=. -E 'with import <nixpkgs> {}; proj.withProjData [ "nz_linz" ]'

$ ls ./result/share/proj
CH                               nz_linz_gisbht1926-nzvd2016.tif         nz_linz_nzgd2000-c220110222-grid01.tif   nz_linz_nzgd2000-ka20161114-grid03.tif   nz_linz_nzgd2000-mq20041223-grid012.tif  nz_linz_stisht1977-nzvd2016.tif
deformation_model.schema.json    nz_linz_lyttht1937-nzvd2016.tif         nz_linz_nzgd2000-c320110613-grid01.tif   nz_linz_nzgd2000-ka20161114-grid04.tif   nz_linz_nzgd2000-mq20041223-grid013.tif  nz_linz_taraht1970-nzvd2016.tif
GL27                             nz_linz_motuht1953-nzvd2016.tif         nz_linz_nzgd2000-c420111223-grid01.tif   nz_linz_nzgd2000-ka20161114-grid05.tif   nz_linz_nzgd2000-mq20041223-grid014.tif  nz_linz_wellht1953-nzvd2016.tif
ITRF2000                         nz_linz_napiht1962-nzvd2016.tif         nz_linz_nzgd2000-ch20160214-grid01.tif   nz_linz_nzgd2000-ka20161114-grid06.tif   nz_linz_nzgd2000-mq20041223-grid015.tif  other.extra
ITRF2008                         nz_linz_nelsht1955-nzvd2016.tif         nz_linz_nzgd2000-cs20130721-grid01.tif   nz_linz_nzgd2000-ka20161114-grid07.tif   nz_linz_nzgd2000-mq20041223-grid016.tif  proj.db
ITRF2014                         nz_linz_nzgd2000-20000101.json          nz_linz_nzgd2000-cs20130721-grid02.tif   nz_linz_nzgd2000-ka20161114-grid08.tif   nz_linz_nzgd2000-ndm-grid01.tif          proj.ini
nad27                            nz_linz_nzgd2000-20130801.json          nz_linz_nzgd2000-ds20090715-grid011.tif  nz_linz_nzgd2000-ka20161114-grid09.tif   nz_linz_nzgd2000-ndm-grid02.tif          projjson.schema.json
nad83                            nz_linz_nzgd2000-20140201.json          nz_linz_nzgd2000-ds20090715-grid012.tif  nz_linz_nzgd2000-ka20161114-grid10.tif   nz_linz_nzgd2000-si20030821-grid01.tif   README-PROJ-DATA.md
nad.lst                          nz_linz_nzgd2000-20150101.json          nz_linz_nzgd2000-ds20090715-grid013.tif  nz_linz_nzgd2000-ka20161114-grid11.tif   nz_linz_nzgd2kgrid0005.tif               triangulation.schema.json
nz_linz_auckht1946-nzvd2016.tif  nz_linz_nzgd2000-20160701.json          nz_linz_nzgd2000-ds20090715-grid014.tif  nz_linz_nzgd2000-ka20161114-grid12.tif   nz_linz_nzgeoid2009.tif                  world
nz_linz_blufht1955-nzvd2016.tif  nz_linz_nzgd2000-20171201.json          nz_linz_nzgd2000-gs20071016-grid01.tif   nz_linz_nzgd2000-lg20130816-grid01.tif   nz_linz_nzgeoid2016.tif
nz_linz_dublht1960-nzvd2016.tif  nz_linz_nzgd2000-20180701.json          nz_linz_nzgd2000-ka20161114-grid01.tif   nz_linz_nzgd2000-lg20130816-grid02.tif   nz_linz_ontpht1964-nzvd2016.tif
nz_linz_duneht1958-nzvd2016.tif  nz_linz_nzgd2000-c120100904-grid01.tif  nz_linz_nzgd2000-ka20161114-grid02.tif   nz_linz_nzgd2000-mq20041223-grid011.tif  nz_linz_README.txt

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@imincik imincik changed the title proj-data: init at 1.16.0 proj-data: init at 1.16.0 (take 2) Feb 22, 2024
@imincik imincik mentioned this pull request Feb 22, 2024
13 tasks
@ofborg ofborg bot added the 8.has: package (new) This PR adds a new package label Feb 22, 2024
@ofborg ofborg bot added 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Feb 22, 2024
@imincik imincik marked this pull request as draft February 22, 2024 16:28
@imincik imincik changed the title proj-data: init at 1.16.0 (take 2) proj-data: add proj-data package and integration support Feb 22, 2024
@imincik imincik marked this pull request as ready for review February 22, 2024 17:27
@imincik imincik requested a review from l0b0 February 22, 2024 17:28
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. and removed 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Feb 22, 2024
proj = finalAttrs.finalPackage;
in
{
withProjData = gridPackages:
Copy link
Contributor

@l0b0 l0b0 Feb 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't most packages implement optional add-ons as parameters to default.nix? That is, rather than installing pkgs.foo.overrides.withBar I'd install (pkgs.foo.override { withBar = true; }). See for example 1, 2.

proj.overrideAttrs (final: prev: {
pname = prev.pname + "-with-grid-packages";
postInstall = ''
cp --recursive ${proj-data}/* $out/share/proj/
Copy link
Contributor

@l0b0 l0b0 Feb 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than copying the data across, would it be possible to instead install proj-data and link to it from this package? That way there's only one way to install proj-data (and only ever one copy of each version), but proj can refer to that package.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this is not possible because $out/share/proj/ already contains data installed by proj. Otherwise I would do a symlink.


shopt -u extglob
runHook postInstall
'';
Copy link
Contributor

@l0b0 l0b0 Feb 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind running this through shellcheck --shell=bash -? It does have a handful of issues. Ditto for the other shell scripts.

@imincik imincik mentioned this pull request Mar 4, 2024
13 tasks
@dotlambda
Copy link
Member

Can 4c88316 be reverted once this is merged?

@imincik
Copy link
Contributor Author

imincik commented Mar 7, 2024

Can 4c88316 be reverted once this is merged?

No, I don't think so. You have to explicitly enable required proj-data grids using proj.withProjData [ "<PROJ-DATA-GRID>" ]' to use them and we don't have any grids enabled during pyproj tests.

@imincik imincik marked this pull request as draft May 22, 2024 17:39
@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 4, 2024
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. 11.by: package-maintainer This PR was created by a maintainer of all the package it changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants