Best way to add sys dependencies of Python libraries?

[YPares]

I'm making a devenv setup to provide python and spaCy (which depends on zlib and libstdc++). I ended up with:

# devenv.nix
{ pkgs, ... }:
{
  packages = [ pkgs.zlib pkgs.stdenv ];

  languages.python.enable = true;
  languages.python.version = "3.11";
  languages.python.venv.enable = true;
  languages.python.venv.requirements = ./requirements.txt;

  enterShell = "export LD_LIBRARY_PATH=${pkgs.zlib}/lib:${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH";
}

# requirements.txt
spacy==3.7

Is that the recommended way to do so? (Ideally I'd like nix itself to provide spaCy, so it and its dependencies are locked along with the rest of the config, but adding it to packages resulted in build errors)

[YPares]

Aha, I found https://github.com/cachix/devenv/blob/python-rewrite/tests/python-native-libs/devenv.nix Ok, nevermind. Though could the docs be updated to specify that case? A lot of python libs need that kind of setup, if that's the recommended way to go it should probably be made explicit.

[sandydoo]

@YPares, are you on the python-rewrite branch of devenv? If you upgrade to main, then you won't have to do the LD_LIBRARY_PATH shenanigans. See https://github.com/cachix/devenv/blob/main/tests/python-native-libs/devenv.nix.

[YPares]

@sandydoo I'm using devenv 1.0.5 (installed from nixpkgs-unstable).

EDIT: I reinstalled it directly from this repo's flake (nix profile install github:cachix/devenv/main#devenv) so I'm on the main branch, and the same problems remain.

[sandydoo]

@YPares, have you run devenv update as well?

Could you please post the error you're getting and instructions how to repro it?

I did the following and it seems to work (I'm not familiar with spacy though):

devenv.nix

{ pkgs, ... }:
{
  packages = [ pkgs.zlib pkgs.stdenv ];

  languages.python.enable = true;
  languages.python.version = "3.11";
  languages.python.venv.enable = true;
  languages.python.venv.requirements = ./requirements.txt;
}

python -m spacy download en_core_web_sm

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")
print([(w.text, w.pos_) for w in doc])

[('This', 'PRON'), ('is', 'AUX'), ('a', 'DET'), ('sentence', 'NOUN'), ('.', 'PUNCT')]

[YPares]

@sandydoo Ok, I think I get the problem. I was using:

inputs:
  nixpkgs:
    url: github:NixOS/nixpkgs/nixpkgs-unstable

as my nixpkgs flake source. With it the env.LD_LIBRARY_PATH is still needed. But when using cachix/devenv-nixpkgs/rolling instead, then it's no longer necessary, and your setup above works.

The exact error was:

>>> import spacy
Traceback (most recent call last):
  File "/home/yves/dev/devenv-spacy-test/.devenv/state/venv/lib/python3.11/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/home/yves/dev/devenv-spacy-test/.devenv/state/venv/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/home/yves/dev/devenv-spacy-test/.devenv/state/venv/lib/python3.11/site-packages/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libz.so.1: cannot open shared object file: No such file or directory