Custom distutils/sysconfig INSTALL_SCHEME leads to different paths in virtualenv, how to avoid this?

[hroncok]

Issue

With the proposal in https://bugs.python.org/issue43976 we are testing an approach in Fedora, that changes the default install scheme to use /usr/local paths. Essentially, we set:

if (not (hasattr(sys, 'real_prefix') or
    sys.prefix != sys.base_prefix) and
    'RPM_BUILD_ROOT' not in os.environ):
        _INSTALL_SCHEMES['posix_prefix'] = {
            'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
            'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
            'purelib': '{base}/local/lib/python{py_version_short}/site-packages',
            'platlib': '{platbase}/local/{platlibdir}/python{py_version_short}/site-packages',
            'include':
                '{installed_base}/include/python{py_version_short}{abiflags}',
            'platinclude':
                '{installed_platbase}/include/python{py_version_short}{abiflags}',
            'scripts': '{base}/local/bin',
            'data': '{base}/local',
        }

However, we have realized this breaks standard assumptions about virtual environments created by virtualenv.
The activation scripts and pythons are in <venv_name>/local/bin, not <venv_name>/bin.

This is a bummer for us and I've been trying to wrap my head around this, but I don't know how to properly solve this on our end or on virtualenv's end.

Is there something I can add to the conditional to recognize this is virtualenv asking for the paths?

I realize this is not a virtualenv issue, but rather our own, but I opened it here anyway because you are best equipped to help me out with this 🙏

Environment

• OS: Fedora Linux 36 with python3 >= 3.10.0-2

cc @FFY00 @encukou @frenzymadness

[gaborbernat]

I'll have to have a think about this and come back to you 🤔 how does venv solve this or are you patching venv too?

[FFY00]

The main problem here is that you overwrite the existing posix_prefix scheme, which virtualenv uses to construct the layout of the virtual environment. Things like this are exactly the reason https://bugs.python.org/issue43976 does not allow distributors to change the existing schemes, but rather only adding new ones -- we need to be able to rely on the paths of a certain scheme to stay the same regardless of the environment.
We should have static schemes, and select between them dynamically at runtime as needed.

The second problem is that virtualenv does not pick a specific scheme, but instead uses the default one, which may change at runtime. This is now expected by the upstream as of Python 3.10, with the addition of https://docs.python.org/3/library/sysconfig.html#sysconfig._get_preferred_schemes.

---

My proposed solution:

Fedora should add a new scheme, instead of overwriting posix_prefix, and change sysconfig._get_preferred_schemes to select it when desired.

virtualenv should ask for a specific scheme, as the default one can change. I would recommend selecting posix_home for the time being, and asking the Python upstream for a new scheme for virtual environments, which IMO should have the exact same paths as posix_home currently does.

[hroncok]

how does venv solve this or are you patching venv too?

No idea how it solves this, but it naturally works. We don't patch it. Maybe it sets real_prefix before asking for the paths?

Fedora should add a new scheme, instead of overwriting posix_prefix, and change sysconfig._get_preferred_schemes to select it when desired.

The problem with this approach is that not everything respects _get_preferred_schemes. IIRC even distutils doesn't. That's why we cannot do that (just yet).

virtualenv should ask for a specific scheme...

If it does, we can patch our virtualenv for now to use the scheme we want... I've tried that, but the problem is, virtualenv prefers distutils results when it is available.

...asking the Python upstream for a new scheme for virtual environments...

Having a specific scheme for this would make this easier indeed.

[hroncok]

venv's EnvBuilder.ensure_directories seem to hardcode the subdirectories of the environment:

https://github.com/python/cpython/blob/v3.10.0/Lib/venv/__init__.py#L122-L131

[gaborbernat]

Hi, sadly we cannot prefer sysconfig over distutils as that wouldn't be backwards compatible. I'd be open for a proposal that allows upstream patching what scheme virtualenv is using, but this must be a new API.

[hroncok]

A hacky workaround is to hotpatch virtualenv/discovery/py_info.py PythonInfo.install_path():

        # A hack for https://github.com/pypa/virtualenv/issues/2208
        if result.startswith(u"local/"):
            return result[6:]
        return result

[FFY00]

The problem with this approach is that not everything respects _get_preferred_schemes. IIRC even distutils doesn't. That's why we cannot do that (just yet).

Ack. You can add a similar patch to distutils tough.
If you are patching sysconfig, distutils should also be patched to mirror the behavior.

Hi, sadly we cannot prefer sysconfig over distutils as that wouldn't be backwards compatible.

Shouldn't distutils produce the same paths on non-patched Python installations?

[hroncok]

You can add a similar patch to distutils tough.
If you are patching sysconfig, distutils should also be patched to mirror the behavior.

The entire point of this endeavor is that we cannot longer patch distutils, as setuptools wants to bundle their own and users tend to update setuptools from pip, which would erase our patches :/

[FFY00]

And it is in the process of being updated to honor sysconfig :)

[hroncok]

I wonder whether we could somehow detect that it is virtualenv who's asking for the scheme. Or if virtualenv could explicitly say that (e.g. by first checking for a specially named prefix and falling back to the present behavior). Was that what you had in mind with:

I'd be open for a proposal that allows upstream patching what scheme virtualenv is using, but this must be a new API.

?

[gaborbernat]

The later 👍

[hroncok]

This is my totally untested draft of that:

diff --git a/src/virtualenv/discovery/py_info.py b/src/virtualenv/discovery/py_info.py
index 0de6128..027507a 100644
--- a/src/virtualenv/discovery/py_info.py
+++ b/src/virtualenv/discovery/py_info.py
@@ -73,7 +73,16 @@ class PythonInfo(object):
         self.file_system_encoding = u(sys.getfilesystemencoding())
         self.stdout_encoding = u(getattr(sys.stdout, "encoding", None))
 
-        self.sysconfig_paths = {u(i): u(sysconfig.get_path(i, expand=False)) for i in sysconfig.get_path_names()}
+        # https://github.com/pypa/virtualenv/issues/2208
+        if "posix_venv" in sysconfig.get_scheme_names():
+            self.sysconfig_scheme = "posix_venv"
+            self.sysconfig_paths = {u(i): u(sysconfig.get_path(i, expand=False, scheme="posix_venv")) for i in sysconfig.get_path_names()}
+            self.distutils_install = {}  # we cannot use distutils at all if "posix_venv" exists, distutils don't know it
+        else:
+            self.sysconfig_scheme = None
+            self.sysconfig_paths = {u(i): u(sysconfig.get_path(i, expand=False)) for i in sysconfig.get_path_names()}
+            self.distutils_install = {u(k): u(v) for k, v in self._distutils_install().items()}
+
         # https://bugs.python.org/issue22199
         makefile = getattr(sysconfig, "get_makefile_filename", getattr(sysconfig, "_get_makefile_filename", None))
         self.sysconfig = {
@@ -95,7 +104,6 @@ class PythonInfo(object):
         if self.implementation == "PyPy" and sys.version_info.major == 2:
             self.sysconfig_vars[u"implementation_lower"] = u"python"
 
-        self.distutils_install = {u(k): u(v) for k, v in self._distutils_install().items()}
         confs = {k: (self.system_prefix if v.startswith(self.prefix) else v) for k, v in self.sysconfig_vars.items()}
         self.system_stdlib = self.sysconfig_path("stdlib", confs)
         self.system_stdlib_platform = self.sysconfig_path("platstdlib", confs)
@@ -119,7 +127,7 @@ class PythonInfo(object):
 
     def install_path(self, key):
         result = self.distutils_install.get(key)
-        if result is None:  # use sysconfig if distutils is unavailable
+        if result is None:  # use sysconfig if sysconfig_scheme is set or distutils is unavailable
             # set prefixes to empty => result is relative from cwd
             prefixes = self.prefix, self.exec_prefix, self.base_prefix, self.base_exec_prefix
             config_var = {k: "" if v in prefixes else v for k, v in self.sysconfig_vars.items()}

Does that seem reasonable?

Open-ended question: Should I check for nt_venv as well? I don't think Windows re-distributors need to set custom install schemes.

[gaborbernat]

Open-ended question: Should I check for nt_venv as well? I don't think Windows re-distributors need to set custom install schemes.

If the proposal from above is platform agnostic this should be too.

[hroncok]

I've tested this approach and it works. For platform agnostic, I guess the scheme should just be called "venv", right?

[gaborbernat]

Name it venv if the venv module will also use it, otherwise virtualenv 😊

[hroncok]

the venv module might use it in the future. the name shall say "this is an installation scheme for virtual environments regardless of the tool that creates them"

[hroncok]

A draft PR in #2209