install-poetry.py: ModuleNotFoundError: No module named 'virtualenv'

[shniubobo]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• [N/A] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04 WSL

• Poetry version: N/A

• Link of a Gist with the contents of your pyproject.toml file: N/A

• install-poetry.py version: 9e17159

I originally posted this at #4195 (comment), but later realized that my situation was different from that issue, so I decided to copy that comment to a seperate issue.

I searched the issues and found python-poetry/install.python-poetry.org#61 which was related, but that post didn't fully cover the issue. ensurepip is not available for the system python in Ubuntu, and the correct way to install system pip can vary among ditros. Additionally, the wheel of virtualenv does not contain its dependencies, and thus poetry has to provide a version of virtualenv with all of its dependencies vendorized.

Below is my original comment, with a little editing. If my solution would be accepted, I would be willing to make a PR to implement it.

Issue

python3 install-poetry.py fails with ModuleNotFoundError: No module named 'virtualenv'.

At first it was because of pip not being installed by default for the system python, but after sudo apt install python3-pip the problem persisted.

Then I tried installing virtualenv manually to see what was going on:

$ mktemp -d
/tmp/tmp.YuayGzcNwi
$ python3 -m pip install virtualenv -t /tmp/tmp.YuayGzcNwi
[unrelated logs omitted]
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 360, in run
    prefix=options.prefix_path,
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/lib/python3/dist-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/lib/python3.6/distutils/command/install.py", line 274, in finalize_options
    raise DistutilsOptionError("can't combine user with prefix, "
distutils.errors.DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base

But if I use a newer version of pip, virtualenv can be successfully installed:

$ mktemp -d
/tmp/tmp.4j1OAffTGx
$ python3.9 -m pip install virtualenv -t /tmp/tmp.4j1OAffTGx
[unrelated logs omitted]
Installing collected packages: six, platformdirs, filelock, distlib, backports.entry-points-selectable, virtualenv
Successfully installed backports.entry-points-selectable-1.1.0 distlib-0.3.2 filelock-3.0.12 platformdirs-2.2.0 six-1.16.0 virtualenv-20.7.0

$ python3 -m pip -V  # system python with system pip
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
$ python3.9 -m pip -V
pip 21.2.3 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

And with this version of pip, poetry can be successfully installed:

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3.9 -
[unrelated logs omitted]
Poetry (1.1.7) is installed now. Great!

You can test that everything is set up by executing:

`poetry --version`
$ poetry --version
Poetry version 1.1.7

---

So there are three problems with install-poetry.py:

1. It assumes that pip is installed.
2. It assumes that an up-to-date pip is installed.
3. The pip install error message is suppressed.

One possible solution to the first and second problems: provide a version of virtualenv with all of its dependency vendorized, and let the script download this version of virtualenv directly instead of using pip.

[christopherpickering]

Same on alpine:latest.. It assumes virtualenv is already installed.

[christopherpickering]

It seems like the problem is caused by pip not coming installed with python. Poetry assumes that pip is installed. If you tweak the install script to print out the error:

 with temporary_directory() as tmp_dir:
            x = subprocess.Popen(
                [sys.executable, "-m", "pip", "install", "virtualenv", "-t", tmp_dir],
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
            )
            out,err = x.communicate()
            print(out,err, x.returncode)
            sys.path.insert(0, tmp_dir)

            import virtualenv

            virtualenv.cli_run([str(env_path), "--clear"])

you'll get something like

b'' b'/usr/bin/python3: No module named pip\n' 1

[christopherpickering]

The subprocess should probably check if pip is installed first.. maybe something like this:

curl -sSL https://bootstrap.pypa.io/get-pip.py | $(which python3) -

but its still broken for python3 on linux.

[christopherpickering]

@finswimmer for reproducing:

docker run -i -t alpine:latest 
apk add curl python3
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | $(which python3) -

I'm not sure how you get the stdout from your subprocess call? For me it is always swallowed.

[Skyross]

I've faced with the same issue with this sequence of actions:

• installed poetry using pip (because of security policy, we did not download installers via curl)
• running poetry install --no-dev uninstalls virtualenv
• any poetry run ... causing this error

[abn]

@Skyross are you executing poetry install --no-dev in the same environment as you installed poetry in? In that case this is expected behaviour.

The new version of poetry does not rely on virtualenv from the environment anymore.You can also try one of the following options.

Option 1 - Use a virtual environment

mkdir ~/.poetry
python -m venv ~/.poetry/venv
~/.poetry/venv/bin/python -m pip install poetry
ln -sf ~/.poetry/venv/bin/poetry ~/bin/poetry  # or to any bin directory in your path

Option 2 - Use pipx*

pip install pipx
pipx install poetry

---

@christopherpickering the isue you are facing is due to missing build time dependencies for cffi as this is in the dependency tree for Poetry and is not available as a wheel compatible with your environment.

poetry-install-error.log

Collecting poetry==1.1.11
  Using cached poetry-1.1.11-py2.py3-none-any.whl (175 kB)
Collecting shellingham<2.0,>=1.1
  Using cached shellingham-1.4.0-py2.py3-none-any.whl (9.4 kB)
Collecting packaging<21.0,>=20.4
  Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting tomlkit<1.0.0,>=0.7.0
  Using cached tomlkit-0.7.2-py2.py3-none-any.whl (32 kB)
Collecting clikit<0.7.0,>=0.6.2
  Using cached clikit-0.6.2-py2.py3-none-any.whl (91 kB)
Collecting cleo<0.9.0,>=0.8.1
  Using cached cleo-0.8.1-py2.py3-none-any.whl (21 kB)
Collecting cachy<0.4.0,>=0.3.0
  Using cached cachy-0.3.0-py2.py3-none-any.whl (20 kB)
Collecting requests-toolbelt<0.10.0,>=0.9.1
  Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting html5lib<2.0,>=1.0
  Using cached html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting virtualenv<21.0.0,>=20.0.26
  Using cached virtualenv-20.10.0-py2.py3-none-any.whl (5.6 MB)
Collecting requests<3.0,>=2.18
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting poetry-core<1.1.0,>=1.0.7
  Using cached poetry_core-1.0.7-py2.py3-none-any.whl (424 kB)
Collecting crashtest<0.4.0,>=0.3.0
  Using cached crashtest-0.3.1-py3-none-any.whl (7.0 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.4
  Using cached CacheControl-0.12.10-py2.py3-none-any.whl (20 kB)
Collecting keyring<22.0.0,>=21.2.0
  Using cached keyring-21.8.0-py3-none-any.whl (32 kB)
Collecting pexpect<5.0.0,>=4.7.0
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting pkginfo<2.0,>=1.4
  Using cached pkginfo-1.7.1-py2.py3-none-any.whl (25 kB)
Collecting msgpack>=0.5.2
  Using cached msgpack-1.0.2.tar.gz (123 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting lockfile>=0.9
  Using cached lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting pastel<0.3.0,>=0.2.0
  Using cached pastel-0.2.1-py2.py3-none-any.whl (6.0 kB)
Collecting pylev<2.0,>=1.3
  Using cached pylev-1.4.0-py2.py3-none-any.whl (6.1 kB)
Collecting six>=1.9
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting webencodings
  Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting SecretStorage>=3.2
  Using cached SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting jeepney>=0.4.2
  Using cached jeepney-0.7.1-py3-none-any.whl (54 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-3.0.6-py3-none-any.whl (97 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.7-py3-none-any.whl (38 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting filelock<4,>=3.2
  Using cached filelock-3.4.0-py3-none-any.whl (9.8 kB)
Collecting platformdirs<3,>=2
  Using cached platformdirs-2.4.0-py3-none-any.whl (14 kB)
Collecting backports.entry-points-selectable>=1.0.4
  Using cached backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl (6.2 kB)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.3-py2.py3-none-any.whl (496 kB)
Collecting cryptography>=2.0
  Using cached cryptography-35.0.0-cp36-abi3-musllinux_1_1_x86_64.whl (3.7 MB)
Collecting cffi>=1.12
  Using cached cffi-1.15.0.tar.gz (484 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Using legacy 'setup.py install' for msgpack, since package 'wheel' is not installed.
Using legacy 'setup.py install' for cffi, since package 'wheel' is not installed.
Installing collected packages: pycparser, urllib3, idna, charset-normalizer, cffi, certifi, requests, pylev, pastel, msgpack, jeepney, cryptography, crashtest, webencodings, six, SecretStorage, pyparsing, ptyprocess, platformdirs, lockfile, filelock, distlib, clikit, cachecontrol, backports.entry-points-selectable, virtualenv, tomlkit, shellingham, requests-toolbelt, poetry-core, pkginfo, pexpect, packaging, keyring, html5lib, cleo, cachy, poetry
    Running setup.py install for cffi: started
    Running setup.py install for cffi: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /root/.local/share/pypoetry/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-t4n7vec2/cffi_4067e5159d1b434a84a62a1d1bd97e3a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-t4n7vec2/cffi_4067e5159d1b434a84a62a1d1bd97e3a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg install --record /tmp/pip-record-iguvj3s8/install-record.txt --single-version-externally-managed --compile --install-headers /root/.local/share/pypoetry/venv/include/site/python3.9/cffi
         cwd: /tmp/pip-install-t4n7vec2/cffi_4067e5159d1b434a84a62a1d1bd97e3a/
    Complete output (45 lines):
    
        No working compiler found, or bogus compiler options passed to
        the compiler from Python's standard "distutils" module.  See
        the error messages above.  Likely, the problem is not related
        to CFFI but generic to the setup.py of any Python package that
        tries to compile C code.  (Hints: on OS/X 10.8, for errors about
        -mno-fused-madd see http://stackoverflow.com/questions/22313407/
        Otherwise, see https://wiki.python.org/moin/CompLangPython or
        the IRC channel #python on irc.libera.chat.)
    
        Trying to continue anyway.  If you are trying to install CFFI from
        a build done in a different context, you can ignore this warning.
    
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/cffi
    copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-x86_64-3.9
    creating build/temp.linux-x86_64-3.9/c
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fomit-frame-pointer -g -fno-semantic-interposition -fomit-frame-pointer -g -fno-semantic-interposition -fomit-frame-pointer -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/root/.local/share/pypoetry/venv/include -I/usr/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
    error: command 'gcc' failed: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: /root/.local/share/pypoetry/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-t4n7vec2/cffi_4067e5159d1b434a84a62a1d1bd97e3a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-t4n7vec2/cffi_4067e5159d1b434a84a62a1d1bd97e3a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg install --record /tmp/pip-record-iguvj3s8/install-record.txt --single-version-externally-managed --compile --install-headers /root/.local/share/pypoetry/venv/include/site/python3.9/cffi Check the logs for full command output.

Traceback:

  File "<stdin>", line 872, in main
  File "<stdin>", line 503, in run
  File "<stdin>", line 525, in install
  File "<stdin>", line 632, in install_poetry
  File "<stdin>", line 340, in pip
  File "<stdin>", line 337, in python
  File "<stdin>", line 330, in run

You can avoid this by installing the required build dependencies.

podman run --rm -i --entrypoint sh docker.io/alpine:latest <<EOF
set -xe
apk add --quiet --no-cache curl python3 python3-dev python3-dev openssl-dev libffi-dev build-base
curl -sSL https://install.python-poetry.org/ | python3 -
/root/.local/bin/poetry --version
EOF

+ apk add --quiet --no-cache curl python3 python3-dev python3-dev openssl-dev libffi-dev build-base
+ curl -sSL https://install.python-poetry.org/
+ python3 -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/root/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.1.11)
Installing Poetry (1.1.11): Creating environment
Installing Poetry (1.1.11): Installing Poetry
Installing Poetry (1.1.11): Creating script
Installing Poetry (1.1.11): Done

Poetry (1.1.11) is installed now. Great!

To get started you need Poetry's bin directory (/root/.local/bin) in your `PATH`
environment variable.

Add `export PATH="/root/.local/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/root/.local/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

+ /root/.local/bin/poetry --version
Poetry version 1.1.11

---

We no longer expect pip to be present in the environment. We however, require that if the venv module is present (unlike ubuntu distro) with your python installation, for it to add pip to an virtual environment it creates, If the venv module is not present, we now fallback to the virtualenv zipapp instead. (#4099)

---

Given all of the above. I am going to close this. For future issues about the installer, use https://github.com/python-poetry/install.python-poetry.org/issues.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.