v5.0.0 causes pip resolver to take longer than usual (looks into old versions)

[atodorov]

Expected behaviour

pip install would install necessary packages and that would take couple of minutes.

Actual behaviour

pip starts resolving dependencies and in the process starts backtracking into older versions which just takes so long that it never completes (in practice GitHub Actions just times out the job).

What are the steps to reproduce this issue?

From kiwitcms/enterprise#110

1. pip install -r devel.txt
2. Search the logs for "pip is looking at multiple versions of"
3. After 2hrs GitHub kills the test job

Any logs, error output, etc?

https://github.com/kiwitcms/enterprise/pull/110/checks?check_run_id=3293801944

$ cat devel.txt 
-r requirements.txt

flake8
kiwitcms
pylint-django
textdistance
twine
readme_renderer[md]
wheel

$ cat requirements.txt 
django-python3-ldap==0.12.0
django-ses==2.2.1
dj-database-url==0.5.0
kiwitcms-github-app==1.3.0
kiwitcms-tenants==1.6.0
raven==6.10.0
social-auth-app-django==5.0.0
social-auth-kerberos==0.2.4
python3-saml==1.11.0

Any other comments?

Does not seem to happen without upgrading to v 5.0.0. The master branch in the referenced repository has v4.0.0 and pip install seems to work fine, see https://github.com/kiwitcms/enterprise/runs/3291432887. Which hints to me that social-app-django or social-core has changed some of the requirements.

Note: I have submitted this feedback with pip at https://pip.pypa.io/surveys/backtracking and waiting for their reply. pip version is the latest one -> 21.2.3

[nijel]

social-auth-core has new deps (defusedxml>=0.5.0rc1 and python3-openid>=3.0.10), and probably some of that conflict with the ones you already have...

[atodorov]

I'm back at this issue, trying to figure out what happens here.

Something I tried locally on my computer was checking the versions inside an existing venv and then trying to upgrade which all worked fine:

$ pip freeze | grep defused
defusedxml==0.7.1

$ pip freeze | grep openid
python3-openid==3.2.0

$ pip freeze | grep social
social-auth-app-django==4.0.0
social-auth-core==4.1.0
social-auth-kerberos==0.2.4

$ pip --version
pip 21.2.4 from /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages/pip (python 3.8)

$ pip install -U social-auth-app-django==5.0.0
Collecting social-auth-app-django==5.0.0
  Using cached social_auth_app_django-5.0.0-py3-none-any.whl (24 kB)
Requirement already satisfied: social-auth-core>=4.1.0 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-app-django==5.0.0) (4.1.0)
Requirement already satisfied: defusedxml>=0.5.0rc1 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (0.7.1)
Requirement already satisfied: oauthlib>=1.0.3 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (3.1.1)
Requirement already satisfied: requests>=2.9.1 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (2.26.0)
Requirement already satisfied: cryptography>=1.4 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (3.4.7)
Requirement already satisfied: PyJWT>=2.0.0 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (2.1.0)
Requirement already satisfied: python3-openid>=3.0.10 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (3.2.0)
Requirement already satisfied: requests-oauthlib>=0.6.1 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (1.3.0)
Requirement already satisfied: cffi>=1.12 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from cryptography>=1.4->social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (1.14.6)
Requirement already satisfied: pycparser in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from cffi>=1.12->cryptography>=1.4->social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (2.20)
Requirement already satisfied: idna<4,>=2.5 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from requests>=2.9.1->social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (3.2)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from requests>=2.9.1->social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from requests>=2.9.1->social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (1.26.6)
Requirement already satisfied: certifi>=2017.4.17 in /home/senko/.virtualenvs/enterprise-temp/lib/python3.8/site-packages (from requests>=2.9.1->social-auth-core>=4.1.0->social-auth-app-django==5.0.0) (2021.5.30)
Installing collected packages: social-auth-app-django
  Attempting uninstall: social-auth-app-django
    Found existing installation: social-auth-app-django 4.0.0
    Uninstalling social-auth-app-django-4.0.0:
      Successfully uninstalled social-auth-app-django-4.0.0
Successfully installed social-auth-app-django-5.0.0

Then trying to install all dependencies from requirements.txt inside a freshly created venv worked without a hitch too.

Installing from devel.txt seems to trigger the problem but only when installing everything in bulk, not when installing individual packages one by one. At this point I don't think that's a problem with python-social-auth packages but need to figure out why pip is acting strange.

[digismack]

I've run into the same issue and our short-term fix was to pin pip to 20.2.2.

[nijel]

According to pypa/pip#10373 this was fixed.