pip breaks after installation of some packages

[arch-user-france1]

Description

pip dies

Expected behavior

pip does not die

pip version

unsure, cannot run pip

Python version

python 3.5.10

OS

Arch

How to Reproduce

not sure

Output

pip                                     ✔   
Traceback (most recent call last):
  File "/usr/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3.10/site-packages/pip/_internal/cli/cmdoptions.py", line 29, in <module>
    from pip._internal.models.target_python import TargetPython
  File "/usr/lib/python3.10/site-packages/pip/_internal/models/target_python.py", line 6, in <module>
    from pip._internal.utils.compatibility_tags import get_supported, version_info_to_nodot
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/compatibility_tags.py", line 7, in <module>
    from pip._vendor.packaging.tags import (
ImportError: cannot import name 'PythonVersion' from 'pip._vendor.packaging.tags' (/home/france1/.local/lib/python3.10/site-packages/packaging/tags.py)

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).

[arch-user-france1]

I have installed the following packages:

absl-py==0.13.0
aiohttp==3.7.4.post0
antlr4-python3-runtime==4.8
async-timeout==3.0.1
attrs==21.2.0
backcall==0.2.0
cachetools==4.2.2
certifi==2021.5.30
chardet==4.0.0
decorator==5.0.9
einops==0.3.0
fsspec==2021.6.1
ftfy==6.0.3
future==0.18.2
google-auth==1.32.0
google-auth-oauthlib==0.4.4
grpcio==1.38.1
idna==2.10
imageio==2.9.0
imageio-ffmpeg==0.4.4
ipython==7.25.0
ipython-genutils==0.2.0
jedi==0.18.0
kornia==0.5.4
Markdown==3.3.4
matplotlib-inline==0.1.2
multidict==5.1.0
numpy==1.21.0
oauthlib==3.1.1
omegaconf==2.1.0
packaging==20.9
parso==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.2.0
prompt-toolkit==3.0.19
protobuf==3.17.3
ptyprocess==0.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyDeprecate==0.3.0
Pygments==2.9.0
pyparsing==2.4.7
pytorch-lightning==1.3.7.post0
PyYAML==5.4.1
regex==2021.4.4
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.7.2
six==1.16.0
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
#torch==1.9.0+cu111
torch
torchaudio
torchmetrics
#torchvision==0.10.0+cu111
torchvision
tqdm==4.61.1
traitlets==5.0.5
typing-extensions==3.10.0.0
urllib3==1.26.6
wcwidth==0.2.5
Werkzeug==2.0.1
yarl==1.6.3```

[arch-user-france1]

Note: upgrading pip does not work because pip does not work, those 'fixes' did not work I found online

[pfmoore]

The line in your output saying from 'pip._vendor.packaging.tags' (/home/france1/.local/lib/python3.10/site-packages/packaging/tags.py) suggests that this is related to some form of unbundling done by your OS vendor (Arch), as it appears pip is not getting its vendored copy of packaging.

You should raise this with your OS vendor in the first instance, as they are responsible for any issues caused by their patches.

[pfmoore]

(Your local installation of packaging is too old, pip needs a newer version which is why we vendor it).

[arch-user-france1]

I still don't really understand.
What do you mean by "Your local installation of packaging is too old, pip needs a newer version which is why we vendor it"?
You also should note that it happened after installing some pip packages and not updating.

What I forgot to tell is that anaconda environments are still working (pip's fine there) but I dislike them and they probably use their own pip.

[pfmoore]

Sorry. The key point here is that you need to raise this with Arch Linux, not with us, as they have modified pip for their distribution, and we don't support those modifications.

[pradyunsg]

This is an issue with Arch Linux. /cc @felixonmars @FFY00

The Arch Linux distributed version of pip is modified in a way that makes it possible to break it (as the user has done here) by installing other packages in the system environment. Please do report this to Arch Linux as a bug, pointing them to this issue and this Twitter thread: https://twitter.com/pradyunsg/status/1556960080565780481

Beyond that, I suggest using virtual environments whenever possible, and avoiding using the Arch-Linux provided pip/Python, until they are able to fix this.

[FFY00]

cc @dvzrv

[FFY00]

A possible way to solve this without affecting the users too much would be for arch to create a custom scheme for arch-provided packages and have pip only look at that. However, this is currently not possible due because the proposed mechanism that would allow us to implement this haven't been merged in CPython and arch has a policy against patching. Maybe a exception can be made here, but I am not sure, the CPython upstream did already acknowledge these kind of needs with https://docs.python.org/3/library/sysconfig.html#sysconfig._get_preferred_schemes, and the discussion around the proposed mechanisms isn't either if they are a bad thing/solution, but rather than if they are enough for other distributions, specifically Debian. If it helps, we can coordinate with the Python upstream in this.

That said, I am not the python or python-pip maintainer, so this is not my call to make. I only really see two options to solve this, either 1) remove the devendoring in python-pip, as the vendoring is a specifically designed mechanism with a purpose other than just simplify bootstrapping, or 2) add an addition install scheme and have pip only pip up that one, as I described above.

I understand this all is very complex, please let me know if there is anything you'd like me to clarify.

[pradyunsg]

Personally, I think (1) is a better solution for the short term at least. (2) may be an option, although you can likely achieve if it's done using sys.meta_path within just pip -- but you'll need to manage those somehow.

[FFY00]

Ideally, 2) would be implemented via a interpreter option that we can put in the shebang.

[dvzrv]

@FFY00 thanks for the thorough reply. I'd be interested in option 2, although I have to admit I don't fully understand how that would look like implemented.

For the time being it might indeed make most sense to not devendor pip anymore. :S

[pradyunsg]

Ideally, 2) would be implemented via a interpreter option that we can put in the shebang.

Another way to achieve this would be to follow what pip's vendoring documentation says. From https://pip.pypa.io/en/stable/development/vendoring-policy/#debundling:

Generate wheels for each of pip’s dependencies (and any of their dependencies) using your patched copies of these libraries. These must be placed somewhere on the filesystem that pip can access (pip/_vendor is the default assumption).

[arch-user-france1]

Shall I do anything?

[dvzrv]

@arch-user-france1 I think with python-pip 22.2.2-2 this issue should be gone for you.

@pradyunsg I think this issue can be closed.

[pradyunsg]

Thanks @dvzrv! ❤️

[arch-user-france1]

@arch-user-france1 I think with python-pip 22.2.2-2 this issue should be gone for you.

@pradyunsg I think this issue can be closed.

I'm currently waiting until the Manjaro repository gets updated. Damn, I hate Manjaro so much.