Skip to content

Improvements to qgis and grass packaging, introduce qgis-ltr derivation #150286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mpickering wants to merge 10 commits into from
Closed

Improvements to qgis and grass packaging, introduce qgis-ltr derivation #150286

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
24696ce
grass: 7.6.1 -> 7.8.6
mpickering Dec 6, 2021
90b748f
grass: Add pdal dependency
mpickering Dec 6, 2021
a0cdf4d
grass: Add missing feature flags
mpickering Dec 6, 2021
5c7230d
grass: wrap grass with wrapGAppsHook
mpickering Dec 7, 2021
3f24658
qgis: add grass to wrapper
mpickering Dec 7, 2021
dbb14ec
qgis: Enable 3D viewer
mpickering Dec 7, 2021
b4cb8c2
qgis: Enable grass
mpickering Dec 7, 2021
8489816
Add qtlocation to pyqt
mpickering Dec 7, 2021
f707c48
qgis: Introduce qgis-ltr and add up to date qgis
mpickering Dec 7, 2021
ca2311b
qgis: apply grass wrapper in unwrapped derivation
mpickering Dec 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 18 additions & 13 deletions pkgs/applications/gis/grass/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,25 @@
{ lib, stdenv, fetchFromGitHub, flex, bison, pkg-config, zlib, libtiff, libpng, fftw
, cairo, readline, ffmpeg_3, makeWrapper, wxGTK30, netcdf, blas
, proj, gdal, geos, sqlite, postgresql, libmysqlclient, python2Packages, libLAS, proj-datumgrid
, cairo, readline, ffmpeg, makeWrapper, wxGTK30, netcdf, blas
, proj, gdal, geos, sqlite, postgresql, libmysqlclient, python3Packages, libLAS, proj-datumgrid
, zstd, pdal, wrapGAppsHook
}:

stdenv.mkDerivation rec {
name = "grass";
version = "7.6.1";
version = "7.8.6";

src = with lib; fetchFromGitHub {
owner = "OSGeo";
repo = "grass";
rev = "${name}_${replaceStrings ["."] ["_"] version}";
sha256 = "1amjk9rz7vw5ha7nyl5j2bfwj5if9w62nlwx5qbp1x7spldimlll";
rev = version;
sha256 = "sha256-zvZqFWuxNyA+hu+NMiRbQVdzzrQPsZrdGdfVB17+SbM=";
};

nativeBuildInputs = [ pkg-config ];
buildInputs = [ flex bison zlib proj gdal libtiff libpng fftw sqlite cairo proj
readline ffmpeg_3 makeWrapper wxGTK30 netcdf geos postgresql libmysqlclient blas
libLAS proj-datumgrid ]
++ (with python2Packages; [ python python-dateutil wxPython30 numpy ]);
buildInputs = [ flex bison zlib proj gdal libtiff libpng fftw sqlite cairo
readline ffmpeg makeWrapper wxGTK30 netcdf geos postgresql libmysqlclient blas
libLAS proj-datumgrid zstd pdal wrapGAppsHook ]
++ (with python3Packages; [ python python-dateutil wxPython_4_1 numpy ]);
Copy link
Contributor

@willcohen willcohen Dec 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can wxPython_4_1 either be switched to wxPython_4_0 or put with logic similar to this:

    ++ (with python3Packages; [ python python-dateutil numpy ] ++
                              lib.optional stdenv.isDarwin wxPython_4_0 ++
                              lib.optional stdenv.isLinux wxPython_4_1);

For reasons not entirely clear to me with #150908, wxGTK30 builds on darwin x86, but wxGTK31 fails with the error that --enable-universal-binaries (found here) isn't a valid option. I don't totally understand why, since it also exists in 3.0.


# On Darwin the installer tries to symlink the help files into a system
# directory
Expand All @@ -37,6 +38,7 @@ stdenv.mkDerivation rec {
"--with-readline"
"--with-wxwidgets"
"--with-netcdf"
"--with-pdal"
Copy link
Contributor

@willcohen willcohen Dec 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On darwin, pdal should also similarly be disabled here pending a resolution to asmaloney/libE57Format#48

"--with-geos"
"--with-postgres"
"--with-postgres-libs=${postgresql.lib}/lib/"
Expand All @@ -46,6 +48,9 @@ stdenv.mkDerivation rec {
"--with-mysql-libs=${libmysqlclient}/lib/mysql"
"--with-blas"
"--with-liblas=${libLAS}/bin/liblas-config"
"--with-zstd"
"--with-fftw"
"--with-pthread"
];

# Otherwise a very confusing "Can't load GDAL library" error
Expand All @@ -62,6 +67,7 @@ stdenv.mkDerivation rec {
scripts/g.extension.all/g.extension.all.py \
scripts/r.drain/r.drain.py \
scripts/r.pack/r.pack.py \
scripts/r.import/r.import.py \
scripts/r.tileset/r.tileset.py \
scripts/r.unpack/r.unpack.py \
scripts/v.clip/v.clip.py \
Expand All @@ -79,18 +85,17 @@ stdenv.mkDerivation rec {
temporal/t.rast.algebra/t.rast.algebra.py \
temporal/t.rast3d.algebra/t.rast3d.algebra.py \
temporal/t.vect.algebra/t.vect.algebra.py \
temporal/t.downgrade/t.downgrade.py \
temporal/t.select/t.select.py
for d in gui lib scripts temporal tools; do
patchShebangs $d
done
'';

NIX_CFLAGS_COMPILE = "-DACCEPT_USE_OF_DEPRECATED_PROJ_API_H=1";

postInstall = ''
wrapProgram $out/bin/grass76 \
wrapProgram $out/bin/grass78 \
--set PYTHONPATH $PYTHONPATH \
--set GRASS_PYTHON ${python2Packages.python}/bin/${python2Packages.python.executable} \
--set GRASS_PYTHON ${python3Packages.python}/bin/${python3Packages.python.executable} \
--suffix LD_LIBRARY_PATH ':' '${gdal}/lib'
ln -s $out/grass*/lib $out/lib
ln -s $out/grass*/include $out/include
Expand Down
27 changes: 27 additions & 0 deletions pkgs/applications/gis/qgis/ltr.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{ lib, makeWrapper, symlinkJoin
, qgis-ltr-unwrapped, extraPythonPackages ? (ps: [ ])
}:
with lib;
symlinkJoin rec {
inherit (qgis-ltr-unwrapped) version;
name = "qgis-${version}";

paths = [ qgis-ltr-unwrapped ];

nativeBuildInputs = [ makeWrapper qgis-ltr-unwrapped.python3Packages.wrapPython ];

# extend to add to the python environment of QGIS without rebuilding QGIS application.
pythonInputs = qgis-ltr-unwrapped.pythonBuildInputs ++ (extraPythonPackages qgis-ltr-unwrapped.python3Packages);

postBuild = ''
# unpackPhase

Comment on lines +17 to +18
Copy link
Member

@SuperSandro2000 SuperSandro2000 Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# unpackPhase

buildPythonPath "$pythonInputs"

wrapProgram $out/bin/qgis \
--prefix PATH : $program_PATH \
--set PYTHONPATH $program_PYTHONPATH
'';

meta = qgis-ltr-unwrapped.meta;
Copy link
Member

@SuperSandro2000 SuperSandro2000 Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
meta = qgis-ltr-unwrapped.meta;
inherit (qgis-ltr-unwrapped) meta;

}
140 changes: 140 additions & 0 deletions pkgs/applications/gis/qgis/unwrapped-ltr.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
{ lib
, mkDerivation
, fetchFromGitHub
, cmake
, ninja
, flex
, bison
, proj
, geos
, xlibsWrapper
, sqlite
, gsl
, qwt
, fcgi
, python3Packages
, libspatialindex
, libspatialite
, postgresql
, txt2tags
, openssl
, libzip
, hdf5
, netcdf
, exiv2
, protobuf
, qtbase
, qtsensors
, qca-qt5
, qtkeychain
, qt3d
, qscintilla
, qtserialport
, qtxmlpatterns
, withGrass ? true
, grass
, withWebKit ? true
, qtwebkit
, makeWrapper
}:

let
pythonBuildInputs = with python3Packages; [
qscintilla-qt5
gdal
jinja2
numpy
psycopg2
chardet
python-dateutil
pyyaml
pytz
requests
urllib3
pygments
pyqt5
sip_4
owslib
six
];
in mkDerivation rec {
version = "3.16.14";
Copy link
Member

@sikmir sikmir Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.16.15 is out

pname = "qgis-ltr-unwrapped";

src = fetchFromGitHub {
owner = "qgis";
repo = "QGIS";
rev = "final-${lib.replaceStrings [ "." ] [ "_" ] version}";
sha256 = "sha256-3FUGSBdlhJhhpTPtYuzKOznsC7PJV3kRL9Il2Yryi1Q=";
};

passthru = {
inherit pythonBuildInputs;
inherit python3Packages;
};

buildInputs = [
openssl
proj
geos
xlibsWrapper
sqlite
gsl
qwt
exiv2
protobuf
fcgi
libspatialindex
libspatialite
postgresql
txt2tags
libzip
hdf5
netcdf
qtbase
qtsensors
qca-qt5
qtkeychain
qscintilla
qtserialport
qtxmlpatterns
qt3d
] ++ lib.optional withGrass grass
++ lib.optional withWebKit qtwebkit
++ pythonBuildInputs;

nativeBuildInputs = [ makeWrapper cmake flex bison ninja ];

# Force this pyqt_sip_dir variable to point to the sip dir in PyQt5
#
# TODO: Correct PyQt5 to provide the expected directory and fix
# build to use PYQT5_SIP_DIR consistently.
postPatch = ''
substituteInPlace cmake/FindPyQt5.py \
--replace 'sip_dir = cfg.default_sip_dir' 'sip_dir = "${python3Packages.pyqt5}/${python3Packages.python.sitePackages}/PyQt5/bindings"'
'';

cmakeFlags = [
"-DCMAKE_SKIP_BUILD_RPATH=OFF"
"-DWITH_3D=True"
"-DPYQT5_SIP_DIR=${python3Packages.pyqt5}/${python3Packages.python.sitePackages}/PyQt5/bindings"
"-DQSCI_SIP_DIR=${python3Packages.qscintilla-qt5}/share/sip/PyQt5"
Copy link
Contributor

@willcohen willcohen Jan 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"-DQSCI_SIP_DIR=${python3Packages.qscintilla-qt5}/share/sip/PyQt5"
"-DQSCI_SIP_DIR=${python3Packages.qscintilla-qt5}/${python3Packages.python.sitePackages}/PyQt5/bindings"

Per #150595, now merged

] ++ lib.optional (!withWebKit) "-DWITH_QTWEBKIT=OFF"
++ lib.optional withGrass "-DGRASS_PREFIX7=${grass}/grass78";

postFixup = ''
# unpackPhase
# grass has to be availble on the command line even though we baked in
# the path at build time using GRASS_PREFIX
wrapProgram $out/bin/qgis \
--prefix PATH : ${lib.makeBinPath (lib.optional withGrass grass)}
'';
Comment on lines +125 to +131
Copy link
Member

@SuperSandro2000 SuperSandro2000 Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
postFixup = ''
# unpackPhase
# grass has to be availble on the command line even though we baked in
# the path at build time using GRASS_PREFIX
wrapProgram $out/bin/qgis \
--prefix PATH : ${lib.makeBinPath (lib.optional withGrass grass)}
'';
postFixup = lib.optionalString withGrass ''
# grass has to be availble on the command line even though we baked in
# the path at build time using GRASS_PREFIX
wrapProgram $out/bin/qgis \
--prefix PATH : ${lib.makeBinPath [ grass ]}
'';

We do not want to empty wrap it.


meta = with lib; {
description = "A Free and Open Source Geographic Information System";
homepage = "https://www.qgis.org";
license = licenses.gpl2Plus;
platforms = platforms.linux;
maintainers = with maintainers; [ lsix sikmir erictapen ];
};
}
32 changes: 25 additions & 7 deletions pkgs/applications/gis/qgis/unwrapped.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,21 @@
, qtsensors
, qca-qt5
, qtkeychain
, qt3d
, qscintilla
, qtserialport
, qtxmlpatterns
, withGrass ? true
, grass
, withWebKit ? true
, qtwebkit
, pdal
, zstd
, makeWrapper
}:

let

pythonBuildInputs = with python3Packages; [
qscintilla-qt5
gdal
Expand All @@ -50,20 +55,20 @@ let
requests
urllib3
pygments
pyqt5
pyqt5_with_qtlocation
sip_4
owslib
six
];
in mkDerivation rec {
version = "3.16.14";
version = "3.22.1";
Copy link
Member

@sikmir sikmir Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.22.2 is out

pname = "qgis-unwrapped";

src = fetchFromGitHub {
owner = "qgis";
repo = "QGIS";
rev = "final-${lib.replaceStrings [ "." ] [ "_" ] version}";
sha256 = "sha256-3FUGSBdlhJhhpTPtYuzKOznsC7PJV3kRL9Il2Yryi1Q=";
sha256 = "sha256:0hpbbv84lh1m7vrxv8d8x5kxgxcf0dydsvr3r2brgv3b40lpavd4";
Copy link
Member

@SuperSandro2000 SuperSandro2000 Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sha256 = "sha256:0hpbbv84lh1m7vrxv8d8x5kxgxcf0dydsvr3r2brgv3b40lpavd4";
sha256 = "0hpbbv84lh1m7vrxv8d8x5kxgxcf0dydsvr3r2brgv3b40lpavd4";

};

passthru = {
Expand Down Expand Up @@ -96,27 +101,40 @@ in mkDerivation rec {
qscintilla
qtserialport
qtxmlpatterns
qt3d
pdal
zstd
] ++ lib.optional withGrass grass
Copy link
Contributor

@willcohen willcohen Dec 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grass is currently broken on darwin, due to #126101. Suggest withGrass -> (!stdenv.isDarwin && withGrass) in all cases for now.

Copy link
Contributor

@willcohen willcohen Dec 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upon further thought, it's not totally clear to me why grass should fundamentally be dependent on webkitgtk. I suppose this is still a stopgap to get qgis to even build on darwin, while it might be possible to go deeper into grass/wxwidgets/etc in parallel to figure out how darwin can still keep grass even while leaving out webkitgtk while it's broken.

Copy link
Contributor

@willcohen willcohen Dec 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It turns out it's actually a pretty small fix: wxwidgets just shouldn't ever build webkitgtk while webkitgtk is broken (

nixpkgs/pkgs/development/libraries/wxwidgets/3.1/default.nix

Line 62 in a0dbe47

++ optional withWebKit webkitgtk
). I'll submit a PR for that so that no workaround here is needed.

Copy link
Contributor

@willcohen willcohen Dec 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #150908 for the wxwidgets fix. There's still a few other issues with grass, so if those can't get resolved then it still may be better to disable grass to get QGIS to still build at all.

In addition, pdal also doesn't work on darwin, due to its dependence on https://github.com/asmaloney/libE57Format, which currently is linux only (see asmaloney/libE57Format#48). I'm not very versed in how static versus dynamic libraries play a role in nix, so I'm not sure what is needed to make that derivation cross-platform compatible. Until then, pdal should probably be disabled and excluded as a build input for darwin.

++ lib.optional withWebKit qtwebkit
++ pythonBuildInputs;

nativeBuildInputs = [ cmake flex bison ninja ];
nativeBuildInputs = [ makeWrapper cmake flex bison ninja ];

# Force this pyqt_sip_dir variable to point to the sip dir in PyQt5
#
# TODO: Correct PyQt5 to provide the expected directory and fix
# build to use PYQT5_SIP_DIR consistently.
postPatch = ''
substituteInPlace cmake/FindPyQt5.py \
--replace 'sip_dir = cfg.default_sip_dir' 'sip_dir = "${python3Packages.pyqt5}/${python3Packages.python.sitePackages}/PyQt5/bindings"'
--replace 'sip_dir = cfg.default_sip_dir' 'sip_dir = "${python3Packages.pyqt5_with_qtlocation}/${python3Packages.python.sitePackages}/PyQt5/bindings"'
'';

cmakeFlags = [
"-DCMAKE_SKIP_BUILD_RPATH=OFF"
"-DPYQT5_SIP_DIR=${python3Packages.pyqt5}/${python3Packages.python.sitePackages}/PyQt5/bindings"
"-DWITH_3D=True"
"-DWITH_PDAL=TRUE"
"-DPYQT5_SIP_DIR=${python3Packages.pyqt5_with_qtlocation}/${python3Packages.python.sitePackages}/PyQt5/bindings"
"-DQSCI_SIP_DIR=${python3Packages.qscintilla-qt5}/share/sip/PyQt5"
Copy link
Contributor

@willcohen willcohen Jan 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line has now changed on master:

nixpkgs/pkgs/applications/gis/qgis/unwrapped.nix

Line 117 in 88d7e0f

"-DQSCI_SIP_DIR=${python3Packages.qscintilla-qt5}/${python3Packages.python.sitePackages}/PyQt5/bindings"

] ++ lib.optional (!withWebKit) "-DWITH_QTWEBKIT=OFF"
++ lib.optional withGrass "-DGRASS_PREFIX7=${grass}/${grass.name}";
++ lib.optional withGrass "-DGRASS_PREFIX7=${grass}/grass78";

postFixup = ''
# unpackPhase
# grass has to be availble on the command line even though we baked in
# the path at build time using GRASS_PREFIX
wrapProgram $out/bin/qgis \
--prefix PATH : ${lib.makeBinPath (lib.optional withGrass grass)}
'';
Comment on lines +131 to +137
Copy link
Member

@SuperSandro2000 SuperSandro2000 Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
postFixup = ''
# unpackPhase
# grass has to be availble on the command line even though we baked in
# the path at build time using GRASS_PREFIX
wrapProgram $out/bin/qgis \
--prefix PATH : ${lib.makeBinPath (lib.optional withGrass grass)}
'';
postFixup = lib.optionalString withGrass ''
# grass has to be availble on the command line even though we baked in
# the path at build time using GRASS_PREFIX
wrapProgram $out/bin/qgis \
--prefix PATH : ${lib.makeBinPath [ grass ]}
'';


meta = {
description = "A Free and Open Source Geographic Information System";
Expand Down
4 changes: 4 additions & 0 deletions pkgs/development/python-modules/pyqt/5.x.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
, withMultimedia ? false
, withWebKit ? false
, withWebSockets ? false
, withLocation ? false
}:

let
Expand Down Expand Up @@ -59,6 +60,7 @@ in buildPythonPackage rec {
++ lib.optional withMultimedia qtmultimedia
++ lib.optional withWebKit qtwebkit
++ lib.optional withWebSockets qtwebsockets
++ lib.optional withLocation qtlocation
;

buildInputs = with libsForQt5; [
Expand All @@ -71,6 +73,7 @@ in buildPythonPackage rec {
++ lib.optional withConnectivity qtconnectivity
++ lib.optional withWebKit qtwebkit
++ lib.optional withWebSockets qtwebsockets
++ lib.optional withLocation qtlocation
;

propagatedBuildInputs = [
Expand Down Expand Up @@ -107,6 +110,7 @@ in buildPythonPackage rec {
++ lib.optional withWebKit "PyQt5.QtWebKit"
++ lib.optional withMultimedia "PyQt5.QtMultimedia"
++ lib.optional withConnectivity "PyQt5.QtConnectivity"
++ lib.optional withLocation "PyQt5.QtPositioning"
;

meta = with lib; {
Expand Down
8 changes: 5 additions & 3 deletions pkgs/top-level/all-packages.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27778,9 +27778,11 @@ with pkgs;

qemu-utils = callPackage ../applications/virtualization/qemu/utils.nix {};

qgis-unwrapped = libsForQt5.callPackage ../applications/gis/qgis/unwrapped.nix {
withGrass = false;
};
qgis-ltr-unwrapped = libsForQt5.callPackage ../applications/gis/qgis/unwrapped-ltr.nix { };

qgis-ltr = callPackage ../applications/gis/qgis/ltr.nix { };

qgis-unwrapped = libsForQt5.callPackage ../applications/gis/qgis/unwrapped.nix { };

qgis = callPackage ../applications/gis/qgis { };

Expand Down
Loading