python.buildEnv: fix venv creation

[qbisi]

Original pr: #297628
Related issue:

1. Revert "xonsh: set dontWrapPythonPrograms" #301449
2. python311Packages.django-countries: 7.5.1 -> 7.6.1 #302315
3. Build failure: pypy3 #301498

#326094 (comment)

The main reason this original PR got reverted, was the assumption that you can undo every wrapper by deleting those lines which sometimes deleted legit python code like for pretalx.

In this pr, we wrap only the python executable with --inherit-argv0 --resolve-argv0. We do dont modify other executable in python-env so there will be no hacky wrapping/unwrapping things of thouse python script. To make the wrapped python script aware of the symlink joined python-environment, we first pass argv[0] to wrapped python script via env variable NIX_PYTHONSCRIPT (for some reason Python doesn't support exec -a) then we derive python-env's sitePackages location from argv[0] and then add it to sys.path.

Things done

1. change the way we build python-env, only wrap python executable with --inherit-argv0 --resolve-argv0, do not modifiy other executable under bin.
2. fix pypy installPhase to be consistent with pypy*_prebuilt / compatiable when creating python-env with --inherit-argv0 .
3. those wrapped python scripts do work with the original python interpreter and necessary python deps in sys.path. However, the script might behave different with optional dependencies in sys.path. (For e.g. pip list will check available python module in sys.path). The python-env's sitePackages can be added to sys.path utilizing argv[0].

cc @MagicRB @ruro

Packages built

[x] pypy3
[ ] xonsh

Thing checked

# build nix python-env
> nix-build -I nixpkgs=$HOME/nixpkgs -E '{pkgs ? import <nixpkgs> {}}: pkgs.python3.withPackages (ps: [ps.pip ps.pytest])
/nix/store/m1fpamwdhw70izn9pfn7df460y5m56f5-python3-3.13.6-env

# check sys.{prefix,base_prefix}
> result/bin/python -c 'import sys; print(sys.base_prefix); print(sys.prefix)'
/home/qbisi/nixpkgs/result
/home/qbisi/nixpkgs/result

# check python script pip do work with python-env
> result/bin/pip list
Package   Version
--------- -------
iniconfig 2.1.0
packaging 25.0
pip       25.0.1
pluggy    1.6.0
Pygments  2.19.2
pytest    8.4.1

# build python-venv from existing nix python-env
> result/bin/python3 -m venv --system-site-packages result-venv

# check sys.{prefix,base_prefix}
> result-venv/bin/python -c 'import sys; print(sys.base_prefix); print(sys.prefix)'
/home/qbisi/nixpkgs/result
/home/qbisi/nixpkgs/result-venv

# check existing python packages via pip list
> result-venv/bin/pip list
result-venv/bin/pip list                                                            
Package   Version
--------- -------
iniconfig 2.1.0
packaging 25.0
pip       25.2
pluggy    1.6.0
Pygments  2.19.2
pytest    8.4.1

# install python packages from pypi
> source result-venv/bin/activate
> pip install numpy
> pip list 
Package   Version
--------- -------
iniconfig 2.1.0
numpy     2.3.3
packaging 25.0
pip       25.2
pluggy    1.6.0
Pygments  2.19.2
pytest    8.4.1

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[ruro]

Also, see #358823.

IMHO, "properly" fixing this issue is actually surprisingly hard, but I believe in you. You can do it! 😜

Otherwise, I might have to find some time to try fixing it myself)))

[qbisi]

I am not sure fix wrap-python-hook do work, i have not found any example python script that will behave different with optional sitePackages added.
Maybe pip ? I will have a try, compiling.

[qbisi]

Well, i miss the deleted line “Python doesn't support exec -a.”

[qbisi]

Some packages (namely xonsh) really need the python-env's wrapper behavior (i.e. adding python-env's sitePackages to sys.path) independent of dontWrapPythonProgram and sys.argv[0].
So we need to keep sitecustomize.py and NIX_PYTHONPATH for wrapping those pythonProgram in python-env unconditionaly.

[ruro]

Can you clarify, why xonsh "needs" this? It seems to me that both xonsh-related issues that I see mentioned here should work fine with as long as the prefixes, sys.argv[0], sys.executable, etc all point to python-env instead of the original python.

After all, xonsh runs just fine on non-nix systems (whether it's installed in an FHS location or in a venv or whatever). And those systems don't have any sitecustomize magic. So why does xonsh need special handling in Nixpkgs/NixOS?

[qbisi]

xonsh set dontWrapPythonProgram=true; since wrap-python-hook will append path when wrapping xonsh, which xonsh does not want serving as a shell. By the meantime, xonsh need to be aware of it's python dependencies. This is done by bundle xonsh into the python-env, which set NIX_PYTHONPATH to python-env's sitePackages. In fact, we can't run nixpkgs#{python3Packages.xonsh}/bin/xonsh barely.

If we add sitePackages "../${python.sitePackges}" relative to sys.argv[0] , xonsh will not work if we do not copy/ln sitePackages directory, see

nixpkgs/pkgs/by-name/xo/xonsh/package.nix

Lines 10 to 27 in ab0f360

	pythonEnv = python3.withPackages (ps: [ ps.xonsh ] ++ extraPackages ps);
	xonsh = python3.pkgs.xonsh;
	in
	runCommand "xonsh-${xonsh.version}"
	{
	inherit (xonsh)
	pname
	version
	meta
	passthru
	;
	}
	''
	mkdir -p $out/bin
	for bin in ${lib.getBin xonsh}/bin/*; do
	ln -s ${pythonEnv}/bin/$(basename "$bin") $out/bin/
	done
	''

If we replace python3Packages.xonsh}/bin/xonsh shebang with the actual python-env.interpreter, then xonsh will works fine. IMO, this is effectively the same as using NIX_PYTHONPATH.

[qbisi]

I guess there exisits binary (not a python script start with #!/nix/store/xxxxx-python/bin/python) that will benefit from NIX_PYTHONPATH when bundled into a python environment. So simply replace shengbang or adding sitePackges to sys.path in python scripts does not cover all the cases.

[qbisi]

I am switching to keep NIX_PYTHONPATH while replacing NIX_PYTHONPREFIX, NIX_PYTHONEXECUTABLE with --inherit-argv0 --resolve-argv0.

I will open another pr later.

[qbisi]

Can you clarify, why xonsh "needs" this? It seems to me that both xonsh-related issues that I see mentioned here should work fine with as long as the prefixes, sys.argv[0], sys.executable, etc all point to python-env instead of the original python.

After all, xonsh runs just fine on non-nix systems (whether it's installed in an FHS location or in a venv or whatever). And those systems don't have any sitecustomize magic. So why does xonsh need special handling in Nixpkgs/NixOS?

My approch of using argv[0] does not work for xonsh, existing approch or replacing xonsh's shenbang with python-env's one will work fine.
I quit this approch mainly for the concern of possible binaries executable that really need the python-env's interpreter while we can't patch them. I think we do need the NIX_PYTHONPATH.

[ruro]

I quit this approch mainly for the concern of possible binaries executable that really need the python-env's interpreter while we can't patch them. I think we do need the NIX_PYTHONPATH.

Hmmm. I wonder, are binary executables that 1) ship as part of a python package and 2) call into python themselves even a thing? I can't think of a single example right now. I think that for python packages, in 90% of cases, bin only contains python scripts and among the remaining 9.9% are binaries/shell scripts that just run without calling into python (and they are instead themselves invoked from python or are just supplementary helper programs, e.g. for generating shell completions).

But if you think that binaries that call into python are common, then it's true that they might end up with a hard-coded path to the non-env version of the python binary. Although, it would be really helpful if you could come up with an example of such a binary (for testing).

[qbisi]

I guess netgen (Python3Packages.netgen-mesher) will be an example.
All those python packages converted from toPythonModule will likely have binary with hardcoded python interpreter.

[ruro]

@qbisi. Ah, I see. It seems that the netgen binary doesn't just call python as a binary, it links against libpythonX.Y.so. I totally forgot about that use case. So yeah, it looks like we can't get rid of sitecustomize.py, because we need the "original" python to be able to "pick up" which -env it was started from instead of just relying on sys.argv[0] of *-env/bin/python. Oh, well.

[qbisi]

migrating to #442540