python: fix venv creation

[qbisi]

This pr should fix the problem of not having sitePackages provided by nix python-env when building virtual python environment from it. See https://discourse.nixos.org/t/how-to-build-python-virtualenv-with-packages-provided-by-python3-withpackages/24766.

It is a continuation of #297628 by @cwp. The original pr got reverted for the wrapping/unwrapping behavior.

#326094 (comment)
The main reason this original PR got reverted, was the assumption that you can undo every wrapper by deleting those lines which sometimes deleted legit python code like for pretalx.

And it also cause a regression building pypy3.

Related issue

1. Revert "xonsh: set dontWrapPythonPrograms" #301449
2. python311Packages.django-countries: 7.5.1 -> 7.6.1 #302315
3. Build failure: pypy3 #301498

Diff compared to #297628

1. We still keep sitecustome.py for wrapping possible binary executable linked to python libraries (e.g. netgen in pyton3Packages.netgen-mesher). Avoid hacky wrapping/unwrapping python scripts.

2. We apply "--inherit-argv0 --resolve-argv0" only when wrapping python executable. The wrapping option "--inherit-argv0 --resolve-argv0" might be danguouse and cause different behavior, e.g. the unfixed {pypy2,pypy3}.withPackages won't start with inherited argv0 as the main executable under bin symlink to subdirectory and it will fail to find its library, see Build failure: pypy3 #301498 and pypy: fix installPhase #442224. # (as far as i have tested, we dont need --resolve-argv0)

3. with the third commit now both python-env python-venv resolve sys.base_prefix to the real path, so we can create a python-venv indepependant of how we call the python-env's python. And make sure the python-venv's python exectuable symlinks directly to nix store There was an attempt trying to resolve argv0's dirname to its' realpath.
   Before

   > nix-build -I nixpkgs=$HOME/nixpkgs -E '{pkgs ? import <nixpkgs> {}}: pkgs.python3.withPackages (ps: [])' -o result
   > result/bin/python3 -c 'import sys; print(sys.base_prefix); print(sys.prefix)'
   /home/qbisi/nixpkgs/result
   /home/qbisi/nixpkgs/result
   
   > result/bin/python3 -m venv --system-site-packages result-venv 
   > result-venv/bin/python3 -c 'import sys; print(sys.base_prefix); print(sys.prefix)'
   /home/qbisi/nixpkgs/result
   /home/qbisi/nixpkgs/result-venv
   > ls -al result-venv/bin/python3
   /home/qbisi/nixpkgs/result/bin/python3
   

   After

   > nix-build -I nixpkgs=$HOME/nixpkgs -E '{pkgs ? import <nixpkgs> {}}: pkgs.python3.withPackages (ps: [])' -o result
   > result/bin/python3 -c 'import sys; print(sys.base_prefix); print(sys.prefix)'
   /nix/store/82hqqjwsg9xgm0mxmbj495hk6sfwd0iv-python3-3.13.7-env
   /nix/store/82hqqjwsg9xgm0mxmbj495hk6sfwd0iv-python3-3.13.7-env
   
   > result/bin/python3 -m venv --system-site-packages result-venv 
   > result-venv/bin/python3 -c 'import sys; print(sys.base_prefix); print(sys.prefix)'
   /nix/store/82hqqjwsg9xgm0mxmbj495hk6sfwd0iv-python3-3.13.7-env
   /home/qbisi/nixpkgs/result-venv
   > ls -al result-venv/bin/python3
   /nix/store/82hqqjwsg9xgm0mxmbj495hk6sfwd0iv-python3-3.13.7-env/bin/python3
   

Packages build and tested (on x86_64-linux and aarch64-darwin)

• pypy2/3 (withPackages)
• python3 (withPackages)
• xonsh (with extraPackages pip)

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[gabyx]

@oneingan: this PR is a crucial fix to make the (wrapped) python interpreters (which are the base for everything which comes next) behave more
correctly, e.g the creation of virtual environments etc.

[imincik]

Great work ! Thanks @qbisi .

[gabyx]

Jeah really great work @qbisi!

[mweinelt]

This broke python313.tests.nixenv and I'm not sure if that's expected.

======================================================================
FAIL: test_base_prefix (test_python.TestCasePython.test_base_prefix)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/tests/test_python.py", line 42, in test_base_prefix
    self.assertNotEqual(sys.prefix, sys.base_prefix)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '/nix/store/51gis2wxs4zkn8hwdksd9bvn0khsrhrj-python3-3.14.0-env' == '/nix/store/51gis2wxs4zkn8hwdksd9bvn0khsrhrj-python3-3.14.0-env'

[qbisi]

This broke python313.tests.nixenv and I'm not sure if that's expected.

Thats expected behavior. now python3.withPackages (ps : []) should have base_prefix=prefix = ${python3.withPackages (ps : [])}

[ruro]

Hey. So if I understand correctly, this was merged into staging two weeks ago, but it seems that it still hasn't reached master yet (nor even staging-next). Does anyone know if this is normal, and when can we expect this to become available?

[qbisi]

This broke python313.tests.nixenv and I'm not sure if that's expected.

The test fix pr: #457133

Does anyone know if this is normal, and when can we expect this to become available?

Just wait for the last staging-next circle before 25.11 release, likely will begin in a week or two.

[adisbladis]

Does this PR relate in any way to the pyproject-nix framework by @adisbladis?

No:
All the relevant Nixpkgs APIs remain the same & when pyproject.nix is used with those nothing that it touches is changed.
When it's used with pypyroject.nix's own builders (such as in uv2nix) the nixpkgs Python infra is completely ignored.