Python + Poetry + Numpy ImportError with multiple python versions

[ianepreston]

This feels very similar to #1095 but I think it's different enough to warrant a new issue, although if you'd prefer me to close this and carry on that thread I'm happy to do so.

I've been trying to build out a devenv with python and poetry that can support libraries like numpy. As with the poster in #1095 I was initially getting the Original error was: libz.so.1: cannot open shared object file: No such file or directory error.
Updating my flake.nix to include libraries = with pkgs; [zlib]; in the python section, blowing away .venv, .direnv, flake.lock, and poetry.lock seemed to resolve the issue, I was able to launch python and import numpy. However, I got greedy. I want to be able to develop against multiple versions of python so I have multiple dev shells defined for each python version I want to support in the library I'm developing. Activating one of those environments caused the original import error to return. Switching back to the previously working environment also produces the error now, and deleting all the above mentioned files and restarting no longer resolves the issue for any environments. I'm really at a loss on this one. I get it not working, I get it working, but I don't get it working then not working. I assume there's some hidden file or setting I'm not touching but I don't know what it could be.

My flake.nix looks like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs-python.url = "github:cachix/nixpkgs-python";
    nixpkgs-python.inputs = { nixpkgs.follows = "nixpkgs"; };
    devenv.url = "github:cachix/devenv";
  };

  nixConfig = {
    extra-trusted-public-keys =
      "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=";
    extra-substituters = "https://devenv.cachix.org";
  };

  outputs = { self, nixpkgs, devenv, ... }@inputs:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { system = system; };
      # A list of shell names and their Python versions
      pythonVersions = {
        python39 = "3.9";
        python310 = "3.10";
        python311 = "3.11";
        default = "3.10";
      };
      # A function to make a shell with a python version
      makePythonShell = shellName: pythonVersion:
        devenv.lib.mkShell {
          inherit inputs pkgs;
          modules = [
            ({ pkgs, config, ... }: {
              languages.python = {
                version = pythonVersion;
                libraries = with pkgs; [ zlib ];
                enable = true;
                venv.enable = true;
                poetry = {
                  enable = true;
                  activate.enable = true;
                  package = pkgs.poetry;
                  install = {
                    enable = true;
                    # compile = true;
                    installRootPackage = true;
                  };
                };
              };
            })
          ];
        };
    in {
      # mapAttrs runs the given function (makePythonShell) against every value
      # in the attribute set (pythonVersions) and returns a new set
      devShells.x86_64-linux = builtins.mapAttrs makePythonShell pythonVersions;
    };
}

pyproject.toml is very simple, I have a couple of basic dependencies that will work to ensure my environment is being activated and then a requirement for pandas:

[tool.poetry]
name = "nixpytesting"
version = "0.1.0"
description = "Figuring out nix and python"
authors = ["Ian Preston"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.32.3"
pandas = "^2.2.2"


[tool.poetry.group.dev.dependencies]
ruff = "^0.4.8"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Any advice for additional troubleshooting I can try would be appreciated.

[alecandido]

Not sure about your lock file, and which version of NumPy you're using, but I noticed that a "vanilla" flake (similar to https://devenv.sh/guides/using-with-flakes/, with the obvious configs for Python, as in your example) is succeeding to run NumPy 1.x.y, but failing with NumPy 2.0.0.

It is also failing with other compiled packages (e.g. Pandas or QuTiP).

I'm now testing https://github.com/cachix/devenv-nixpkgs

[alecandido]

@ianepreston I believe the main discussion to be still related to #773, so the problem should be solved there (I believe @Atry was mainly facing it).

However, a highly discouraged workaround that you can find on the NixOS Discourse is to
manually set LD_LIBRARY_PATH, to contain the path to the libraries you're looking for.

This is partially what Devenv itself is doing

devenv/src/modules/languages/python.nix

Lines 13 to 27 in f810f8d

	package = pkgs.callPackage "${pkgs.path}/pkgs/development/interpreters/python/wrapper.nix" {
	python = cfg.package;
	requiredPythonModules = cfg.package.pkgs.requiredPythonModules;
	makeWrapperArgs = [
	"--prefix"
	"LD_LIBRARY_PATH"
	":"
	libraries
	] ++ lib.optionals pkgs.stdenv.isDarwin [
	"--prefix"
	"DYLD_LIBRARY_PATH"
	":"
	libraries
	];
	};

and it should be the purpose of the libraries option. But in this case, it would be restricted to the Python executable,
https://github.com/NixOS/nixpkgs/blob/89c49874fb15f4124bf71ca5f42a04f2ee5825fd/pkgs/development/interpreters/python/wrapper.nix#L45
https://nixos.org/manual/nixpkgs/stable/#fun-makeWrapper
while setting manually is exposed to all executables in your shell (which is quite more aggressive).

Not sure why the libraries variable is not working for me (it's not even finding libstdc++.so.6 on NixOS), but setting LD_LIBRARY_PATH is working.

I'm pretty sure it's an ugly solution, with many drawbacks, but it may solve your issues while waiting for a proper one.

(in case you want to know which is the exact path to the library you want to use, assuming it's already available in your /nix/store/, I'd advise to use nix-locate from the nix-index tool, e.g. as nix-locate --top-level libz.so.1)

[saucoide]

Same thing here, either zlib.so.1 or libstdc++.so.6

libz.so.1: cannot open shared object file: No such file or directory
Ilibstdc++.so.6: cannot open shared object file: No such file or directory

If I don't enable python with devenv's language option, but have it as a package python312Packages.python:

• fail finding zlib
• Add zlib to packages -> fails with libstdc++
• set LD_LIBRARY_PATH -> back to failing with zlib

i'm going to give up on python via nix 😞

pd: this is on linux - on macos it seems to work 🤷‍♂️

edit:
in the end i have something like this and seems to work ok, hopefully setting LD_LIBRARY_PATH in the env wont break other things :/ , using ptyohn from nixpkgs as pkgs directly and not nixpkgs-python, or enabling it with languages.python, it also worked without setting LD_LIBRARY_PATH but only if i used cachix/devenv-nixpkgs/rolling like mentioned here #1228 (comment) , it's all a bit messy at the moment

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    systems.url = "github:nix-systems/default";
    devenv.url = "github:cachix/devenv";
  };
  outputs = {self, nixpkgs, devenv, systems, ...} @ inputs:
    let
      forEachSystem = nixpkgs.lib.genAttrs (import systems);
    in
      {
        packages = forEachSystem (system: {
          devenv-up = self.devShells.${system}.default.config.procfileScript;
        });

        devShells = forEachSystem (system:
          let
            pkgs = nixpkgs.legacyPackages.${system};
          in {
            default = devenv.lib.mkShell {
              inherit inputs pkgs;
              modules = [
                ({pkgs, config, ...}: {

                  # https://devenv.sh/reference/options/
                  env = {
                    VIRTUAL_ENV = config.env.DEVENV_ROOT + "/.venv";
                    PYTHONPATH = config.env.DEVENV_ROOT;
                    MYPYPATH = config.env.DEVENV_ROOT;
                    # so C extensions can find dyamically linked libraries
                    LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc pkgs.zlib];
                  };

                   packages = [
                     pkgs.python312Packages.python
                   ];

                  ...

                })
              ];
            };
          }
        );
      };
}

[ianepreston]

I didn't end up ditching nix but I did go over to poetry2nix and am actually pretty happy with my setup now: https://blog.ianpreston.ca/posts/2024-06-30-nix.html#back-to-poetry-with-poetry2nix

[linozen]

I'm facing the same issue with the numpy (and any package that depends on it) which unfortunately makes devenv not really usable for Python work as of right now.

[pinkwah]

I'm hitting this as well. The issue is that Poetry is using an unwrapped interpreter, thus LD_LIBRARY_PATH is not set, thus libstdc++.so.6 is not found. This is not the case for when one uses venv with languages.python.venv.requirements.

In creating a minimal reproducible example, having pyproject.toml and co. next to devenv.nix works, but having the poetry stuff in a subdirectory and setting languages.python.directory doesn't. Looking into it some more. 👀