pip fails on windows due to interaction with pyinvoke on windows

[stefanoborini]

Environment

• pip version: 20.2.3
• Python version: 3.7.4
• OS: Windows 10

invoke 1.4.1

Description

pip3 fails with

ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\pip-uninstall-dqrsmia8\\_yaml.cp37-win_amd64.pyd'
Consider using the `--user` option or check the permissions.

When invoked in a particular situation involving invoke. Note that the script is a minimalistic version that reproduces the error with as little noise as possible.

This is really strange and I believe might be a bug of invoke or an interaction of some sort. Note that I am not using any feature of invoke. I am just importing it.

The following script

import invoke
import subprocess

DEFAULT_PYPI_MIRROR_URL = "secret"
DEFAULT_PYPI_URL = "secret"

command = f"pip3 install --force-reinstall --upgrade --index-url {DEFAULT_PYPI_MIRROR_URL} --extra-index-url {DEFAULT_PYPI_URL} pre-commit"
print(command)
subprocess.check_output(command.split(' '))

Leads to the error

However, the pip3 install command executes successfully for the following:

1. the command is invoked directly from the powershell prompt, or
2. the import invoke in the above script is commented out or removed.

I also checked the permissions of the (temporary) file that triggers the error, and there are no issues. It can be deleted without any issue.

Expected behavior

I would expect it to work :)

How to Reproduce

See description.

Output

See descripton.

[pfmoore]

It looks like you have something running that is using pyyaml. Windows doesn't allow you to modify a loaded DLL, so pip can't reinstall pyyaml while the module is being used.

I don't think invoke is relevant here, unless your actual invoke script is importing pyyaml (the test script you provide isn't, but that script doesn't fail for me).

[stefanoborini]

@pfmoore I have absolutely nothing using pyyaml, What you see is what I run. Also note that it does not always fail on pyyaml, and why would I get a failure on a temporary file?

[pfmoore]

Hmm, maybe it's a virus checker opening the dll to check it? We unpack the wheel into a temporary location, then clean up when we're finished. If your virus checker aggressively opens the file to scan it and in doing so, locks the file so that pip can't clean up, that would result in the error you're seeing.

If that's the case, this would be a duplicate of #7280

[stefanoborini]

@pfmoore I tried to reproduce it by creating a completely new virtual env. I cannot reproduce it in that one. I have no idea what's going on to be honest, but I had this happening on two different windows machines. let me investigate a bit more.

[stefanoborini]

@pfmoore can now reproduce it again. Steps I used:

PS> python -m venv bar
PS> cd bar
PS bar> . .\Scripts\Activate.ps1
(bar) PS bar> cp ..\foo\x.py .
(bar) PS bar> cat .\x.py 
import subprocess

DEFAULT_PYPI_MIRROR_URL = "secret"
DEFAULT_PYPI_URL = "secret"

command = f"pip3 install --force-reinstall --upgrade --index-url {DEFAULT_PYPI_MIRROR_URL} --extra-index-url {DEFAULT_PYPI_URL} pre-commit"
print(command)
subprocess.check_output(command.split(' ')) 
(bar) PS bar> python -m pip install --upgrade pip --index-url secret 
Looking in indexes: secret
Collecting pip
Using cached secret/api/pypi/pypi-remote/packages/packages/4e/5f/528232275f6509b1fff703c9280e58951a81abe24640905de621c9f81839/pip-20.2.3-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 19.0.3
Uninstalling pip-19.0.3:
Successfully uninstalled pip-19.0.3
Successfully installed pip-20.2.3
(bar) PS bar> python .\x.py
pip3 install --force-reinstall --upgrade --index-url secret --extra-index-url secret pre-commit
(bar) PS bar> python .\x.py
pip3 install --force-reinstall --upgrade --index-url secret --extra-index-url secret pre-commit
Note how it works twice without a problem
(bar) PS bar> python -m pip install --upgrade invoke --index-url secret
Looking in indexes: secret 
Collecting invoke
Using cached secret/api/pypi/pypi-remote/packages/packages/2c/16/f00efa99ae9f255142a230ce6819c37ae9dd29a7144477c1161cc72d01ed/invoke-1.4.1-py3-none-any.whl (210 kB)
Installing collected packages: invoke
Successfully installed invoke-1.4.1
(bar) PS bar> python .\x.py
pip3 install --force-reinstall --upgrade --index-url secret --extra-index-url secret pre-commit
Still working
(bar) PS bar> vim .\x.py
Here I add a single import invoke at the top of the file
(bar) PS bar> python .\x.py
pip3 install --force-reinstall --upgrade --index-url secret --extra-index-url secret pre-commit
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\xxx\AppData\\Local\\Temp\\pip-uninstall-tdmgbrms\\_yaml.cp37-win_amd64.pyd'
Consider using the `--user` option or check the permissions.

To me it does not look like antivirus. If it were antivirus, it would happen even when invoked from the command line. And it does seem to relate to the import of invoke.

[stefanoborini]

@pfmoore I am going to cross post this to pyinvoke, because I see they are doing some strange stuff with pyyaml...

[pfmoore]

Yep, basically it looks like they import yaml (via a really funky route). So if you have pyyaml installed, importing invoke will lock it, so you can't upgrade it from a process that imports invoke.

Good catch - I'd checked invoke's dependencies, but it doesn't declare pyyaml as a required dependency so I didn't spot it.

Edit: in case it's not clear, pre-commit does depend on pyyaml, so that's what triggers the reinstall of pyyaml.

[stefanoborini]

@pfmoore I am rather sure at this point you are right and this is not a pip issue. Can be closed IMHO. However, it's really strange behavior and I'd love to understand why it accesses a pyd file that is temporary. Maybe some windows strangeness?

Anyway. Closing. Thanks for the help. Please do not delete so that it can be referenced by pyinvoke bug tracker.