Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

virtualenv with --system-site-packages can't build packages with setuptools >= 50.0 #1934

Closed
godlygeek opened this issue Sep 1, 2020 · 2 comments
Closed

virtualenv with --system-site-packages can't build packages with setuptools >= 50.0 #1934

godlygeek opened this issue Sep 1, 2020 · 2 comments
Labels
bug

Comments

@godlygeek
Copy link

I still haven't figured out exactly what's happening here, but I have enough info that it seems worth filing the issue so someone more knowledgeable can dig in.

Issue

Creating a --system-site-packages virtualenv, activating it, and then attempting an isolated build of anything that uses setuptools.build_meta as its build backend fails.

Environment

Provide at least:

  • OS: Linux (reproduced on RHEL 7.8 and Ubuntu 18.04, on Python 3.6 and 3.8)
  • pip list of the host python where virtualenv is installed:
appdirs (1.4.4)
asn1crypto (0.24.0)
certifi (2018.1.18)
chardet (3.0.4)
cryptography (2.1.4)
decorator (4.1.2)
distlib (0.3.1)
filelock (3.0.12)
httplib2 (0.9.2)
idna (2.6)
ipykernel (4.8.2)
ipython (5.5.0)
ipython-genutils (0.2.0)
jupyter-client (5.2.2)
jupyter-console (5.2.0)
jupyter-core (4.4.0)
keyring (10.6.0)
keyrings.alt (3.0)
netifaces (0.10.4)
pexpect (4.2.1)
pickleshare (0.7.4)
pip (9.0.1)
prompt-toolkit (1.0.15)
pycrypto (2.6.1)
Pygments (2.2.0)
pygobject (3.26.1)
python-apt (1.6.5+ubuntu0.2)
python-dateutil (2.6.1)
pyxdg (0.25)
PyYAML (3.12)
pyzmq (16.0.2)
qtconsole (4.3.1)
requests (2.18.4)
requests-unixsocket (0.1.5)
SecretStorage (2.3.1)
setuptools (39.0.1)
simplegeneric (0.8.1)
six (1.15.0)
systemd-python (234)
tornado (4.5.3)
traitlets (4.3.2)
unattended-upgrades (0.1)
urllib3 (1.22)
virtualenv (20.0.31)
wcwidth (0.1.7)
wheel (0.30.0)

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

~>python3.8 -m virtualenv --system-site-packages -vvv --with-traceback some_venv
434 setup logging to NOTSET [DEBUG report:43]
514 find interpreter for spec PythonSpec(path=/usr/bin/python3.8) [INFO builtin:44]
515 proposed PythonInfo(spec=CPython3.8.5.final.0-64, exe=/usr/bin/python3.8, platform=linux, version='3.8.5 (default, Jul 20 2020, 19:48:14) \n[GCC 7.5.0]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
517 accepted PythonInfo(spec=CPython3.8.5.final.0-64, exe=/usr/bin/python3.8, platform=linux, version='3.8.5 (default, Jul 20 2020, 19:48:14) \n[GCC 7.5.0]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
541 filesystem is case-sensitive [DEBUG info:28]
851 create virtual environment via CPython3Posix(dest=/home/matt/some_venv, clear=False, global=True) [INFO session:52]
853 create folder /home/matt/some_venv/bin [DEBUG _sync:25]
855 create folder /home/matt/some_venv/lib/python3.8/site-packages [DEBUG _sync:25]
857 write /home/matt/some_venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
858     home = /usr [DEBUG pyenv_cfg:38]
859     implementation = CPython [DEBUG pyenv_cfg:38]
862     version_info = 3.8.5.final.0 [DEBUG pyenv_cfg:38]
862     virtualenv = 20.0.31 [DEBUG pyenv_cfg:38]
863     include-system-site-packages = true [DEBUG pyenv_cfg:38]
863     base-prefix = /usr [DEBUG pyenv_cfg:38]
864     base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
864     base-executable = /usr/bin/python3.8 [DEBUG pyenv_cfg:38]
866 symlink /usr/bin/python3.8 to /home/matt/some_venv/bin/python [DEBUG _sync:44]
872 create virtualenv import hook file /home/matt/some_venv/lib/python3.8/site-packages/_virtualenv.pth [DEBUG api:94]
874 create /home/matt/some_venv/lib/python3.8/site-packages/_virtualenv.py [DEBUG api:97]
877 ============================== target debug ============================== [DEBUG session:54]
879 debug via /home/matt/some_venv/bin/python /home/matt/.local/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:209]
878 {
  "sys": {
    "executable": "/home/matt/some_venv/bin/python",
    "_base_executable": "/home/matt/some_venv/bin/python",
    "prefix": "/home/matt/some_venv",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/home/matt/some_venv",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python38.zip",
      "/usr/lib/python3.8",
      "/usr/lib/python3.8/lib-dynload",
      "/home/matt/some_venv/lib/python3.8/site-packages",
      "/home/matt/.local/lib/python3.8/site-packages",
      "/usr/local/lib/python3.8/dist-packages",
      "/usr/lib/python3/dist-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.8.5 (default, Jul 20 2020, 19:48:14) \n[GCC 7.5.0]",
  "makefile_filename": "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib/python3.8/os.py'>",
  "site": "<module 'site' from '/usr/lib/python3.8/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib/python3.8/datetime.py'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from '/usr/lib/python3.8/json/__init__.py'>"
} [DEBUG session:55]
1092 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matt/.local/share/virtualenv) [INFO session:59]
1111 got embed update of distribution pip from /home/matt/.local/share/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]
1115 got embed update of distribution setuptools from /home/matt/.local/share/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]
1139 got embed update of distribution setuptools from /home/matt/.local/share/virtualenv/wheel/3.8/embed/1/setuptools.json [DEBUG via_disk_folder:133]
1144 got embed update of distribution wheel from /home/matt/.local/share/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]
1155 got embed update of distribution pip from /home/matt/.local/share/virtualenv/wheel/3.8/embed/1/pip.json [DEBUG via_disk_folder:133]
1159 got embed update of distribution wheel from /home/matt/.local/share/virtualenv/wheel/3.8/embed/1/wheel.json [DEBUG via_disk_folder:133]
1164 install setuptools from wheel /home/matt/.local/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-49.6.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1169 Attempting to acquire lock 134402757171520 on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any.lock [DEBUG filelock:270]
1169 install pip from wheel /home/matt/.local/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.2.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1170 Lock 134402757171520 acquired on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any.lock [INFO filelock:274]
1171 Attempting to acquire lock 134402757171664 on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.2.2-py2.py3-none-any.lock [DEBUG filelock:270]
1174 Attempting to release lock 134402757171520 on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any.lock [DEBUG filelock:315]
1174 Lock 134402757171520 released on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any.lock [INFO filelock:318]
1175 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any/pkg_resources to /home/matt/some_venv/lib/python3.8/site-packages/pkg_resources [DEBUG _sync:52]
1176 Lock 134402757171664 acquired on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.2.2-py2.py3-none-any.lock [INFO filelock:274]
1177 Attempting to release lock 134402757171664 on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.2.2-py2.py3-none-any.lock [DEBUG filelock:315]
1177 Lock 134402757171664 released on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.2.2-py2.py3-none-any.lock [INFO filelock:318]
1178 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.2.2-py2.py3-none-any/pip-20.2.2.dist-info to /home/matt/some_venv/lib/python3.8/site-packages/pip-20.2.2.dist-info [DEBUG _sync:52]
1184 install wheel from wheel /home/matt/.local/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.35.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:53]
1185 Attempting to acquire lock 134402757171808 on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.35.1-py2.py3-none-any.lock [DEBUG filelock:270]
1195 Lock 134402757171808 acquired on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.35.1-py2.py3-none-any.lock [INFO filelock:274]
1198 Attempting to release lock 134402757171808 on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.35.1-py2.py3-none-any.lock [DEBUG filelock:315]
1199 Lock 134402757171808 released on /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.35.1-py2.py3-none-any.lock [INFO filelock:318]
1202 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.35.1-py2.py3-none-any/wheel-0.35.1.dist-info to /home/matt/some_venv/lib/python3.8/site-packages/wheel-0.35.1.dist-info [DEBUG _sync:52]
1266 copy /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.2.2-py2.py3-none-any/pip-20.2.2.virtualenv to /home/matt/some_venv/lib/python3.8/site-packages/pip-20.2.2.virtualenv [DEBUG _sync:52]
1272 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-20.2.2-py2.py3-none-any/pip to /home/matt/some_venv/lib/python3.8/site-packages/pip [DEBUG _sync:52]
1279 copy /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.35.1-py2.py3-none-any/wheel-0.35.1.virtualenv to /home/matt/some_venv/lib/python3.8/site-packages/wheel-0.35.1.virtualenv [DEBUG _sync:52]
1307 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.35.1-py2.py3-none-any/wheel to /home/matt/some_venv/lib/python3.8/site-packages/wheel [DEBUG _sync:52]
1518 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any/setuptools-49.6.0.dist-info to /home/matt/some_venv/lib/python3.8/site-packages/setuptools-49.6.0.dist-info [DEBUG _sync:52]
1592 generated console scripts wheel wheel3.8 wheel-3.8 wheel3 [DEBUG base:48]
1611 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any/_distutils_hack to /home/matt/some_venv/lib/python3.8/site-packages/_distutils_hack [DEBUG _sync:52]
1642 copy directory /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any/setuptools to /home/matt/some_venv/lib/python3.8/site-packages/setuptools [DEBUG _sync:52]
2503 copy /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any/easy_install.py to /home/matt/some_venv/lib/python3.8/site-packages/easy_install.py [DEBUG _sync:52]
2509 copy /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any/setuptools-49.6.0.virtualenv to /home/matt/some_venv/lib/python3.8/site-packages/setuptools-49.6.0.virtualenv [DEBUG _sync:52]
2516 copy /home/matt/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-49.6.0-py3-none-any/distutils-precedence.pth to /home/matt/some_venv/lib/python3.8/site-packages/distutils-precedence.pth [DEBUG _sync:52]
2530 generated console scripts easy_install-3.8 easy_install easy_install3 easy_install3.8 [DEBUG base:48]
3325 generated console scripts pip3 pip-3.8 pip3.8 pip [DEBUG base:48]
3325 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:64]
3350 write /home/matt/some_venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
3350    home = /usr [DEBUG pyenv_cfg:38]
3350    implementation = CPython [DEBUG pyenv_cfg:38]
3350    version_info = 3.8.5.final.0 [DEBUG pyenv_cfg:38]
3351    virtualenv = 20.0.31 [DEBUG pyenv_cfg:38]
3351    include-system-site-packages = true [DEBUG pyenv_cfg:38]
3351    base-prefix = /usr [DEBUG pyenv_cfg:38]
3351    base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
3351    base-executable = /usr/bin/python3.8 [DEBUG pyenv_cfg:38]
3391 created virtual environment CPython3.8.5.final.0-64 in 2963ms
  creator CPython3Posix(dest=/home/matt/some_venv, clear=False, global=True)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matt/.local/share/virtualenv)
    added seed packages: pip==20.2.2, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:17]

Now, taking this created virtualenv and trying to pip install python-dateutil into it, building from source instead of using wheels, gives:

(some_venv) ~>pip install --no-binary :all: --ignore-installed python-dateutil
Collecting python-dateutil
  Using cached python-dateutil-2.8.1.tar.gz (331 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 2:
   command: /home/matt/some_venv/bin/python /home/matt/some_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-a9unamqs/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm
       cwd: None
  Complete output (60 lines):
  Ignoring setuptools: markers 'python_version == "3.3"' don't match your environment
  Collecting setuptools
    Using cached setuptools-50.0.2.zip (2.2 MB)
  Collecting wheel
    Using cached wheel-0.35.1.tar.gz (59 kB)
  Collecting setuptools_scm
    Using cached setuptools_scm-4.1.2.tar.gz (48 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
  ERROR: Exception:
  Traceback (most recent call last):
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
      status = self.run(options, args)
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 324, in run
      requirement_set = resolver.resolve(
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
      abstract_dist = _get_prepared_distribution(
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution
      abstract_dist.prepare_distribution_metadata(finder, build_isolation)
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
      self._setup_isolation(finder)
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
      reqs = backend.get_requires_for_build_wheel()
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 160, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
      raise BackendUnavailable(data.get('traceback', ''))
  pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
    File "/home/matt/some_venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
      obj = import_module(mod_path)
    File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 783, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 5, in <module>
      import distutils.core
    File "/tmp/pip-build-env-y44b9ynh/overlay/lib/python3.8/site-packages/_distutils_hack/__init__.py", line 83, in create_module
      return importlib.import_module('setuptools._distutils')
    File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ModuleNotFoundError: No module named 'setuptools._distutils'

  ----------------------------------------
ERROR: Command errored out with exit status 2: /home/matt/some_venv/bin/python /home/matt/some_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-a9unamqs/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm Check the logs for full command output.

This does not reproduce with a virtualenv created with python -m venv --system-site-packages, but does with one created with python -m virtualenv --system-site-packages. It reproduces with both new and old versions of virtualenv (I tested with virtualenv 16.4.0 and 20.0.31). It only reproduces if the virtualenv was created with --system-site-packages.

As far as I've been able to glean, setuptools._distutils is a new module that is present in the latest upstream setuptools, but is not present in the version of setuptools that is installed in the system site packages directory. I think that, when the pep517 isolated build is performed, the _distutils_hack that was installed by setuptools in the isolated build environment attempts to import setuptools._distutils, and finds the setuptools from the system site packages, which doesn't have that submodule, instead of the one from the isolated build environment, which does.

I haven't yet pieced together whether this is a virtualenv bug, or a pip bug, or a setuptools bug - I'm starting here because I've been able to reproduce this with virtualenv but not with venv in a seemingly similar configuration.
@godlygeek godlygeek added the bug label Sep 1, 2020
@godlygeek godlygeek mentioned this issue Sep 1, 2020
Closed
@godlygeek
Copy link
Author

godlygeek commented Sep 1, 2020 •
edited
Loading

Scratch this - I've managed to reproduce this with a -m venv virtualenv as well (for some reason it wasn't reproducing with venv on RHEL but did on Ubuntu), so this no longer appears to be anything specific to virtualenv.

@godlygeek
Copy link
Author

Closing the loop on this, I believe this is a pip bug; see pypa/pip#6264 (comment).

@pypa pypa locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@godlygeek