Unexplained sytest poetry build failure weirdness

[DMRobertson]

https://github.com/matrix-org/sytest/runs/7250780423?check_suite_focus=true#step:6:929

I wonder if this is fixed by bumping to poetry 1.1.14 (see matrix-org/synapse#13285 and python-poetry/poetry#5973)

[squahtx]

The docker build is failing with:

Step 6/13 : RUN ${PYTHON_VERSION} -m pip install --no-cache-dir poetry==1.1.12
 ---> Running in 71c95faf2e53
...
Requirement already satisfied: cryptography>=2.0 in /usr/lib/python3/dist-packages (from SecretStorage>=3.2; sys_platform == "linux"->keyring<22.0.0,>=21.2.0; python_version >= "3.6" and python_version < "4.0"->poetry==1.1.12) (2.6.1)
Installing collected packages: pkginfo, crashtest, pylev, pastel, clikit, cleo, ptyprocess, pexpect, idna, urllib3, certifi, charset-normalizer, requests, cachy, zipp, importlib-metadata, poetry-core, msgpack, lockfile, cachecontrol, web
encodings, html5lib, shellingham, filelock, distlib, platformdirs, virtualenv, tomlkit, jeepney, SecretStorage, keyring, requests-toolbelt, pyparsing, packaging, poetry
  Running setup.py install for msgpack: started
    Running setup.py install for msgpack: finished with status 'done'
  Found existing installation: SecretStorage 2.3.1
    Not uninstalling secretstorage at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'SecretStorage'. No files were found to uninstall.
  Found existing installation: keyring 17.1.1
    Not uninstalling keyring at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'keyring'. No files were found to uninstall.
Successfully installed SecretStorage-3.3.2 cachecontrol-0.12.11 cachy-0.3.0 certifi-2022.6.15 charset-normalizer-2.1.0 cleo-0.8.1 clikit-0.6.2 crashtest-0.3.1 distlib-0.3.5 filelock-3.7.1 html5lib-1.1 idna-3.3 importlib-metadata-1.7.0 je
epney-0.8.0 keyring-21.8.0 lockfile-0.12.2 msgpack packaging-20.9 pastel-0.2.1 pexpect-4.8.0 pkginfo-1.8.3 platformdirs-2.5.2 poetry-1.1.12 poetry-core-1.0.8 ptyprocess-0.7.0 pylev-1.4.0 pyparsing-3.0.9 requests-2.28.1 requests-toolbelt-
0.9.1 shellingham-1.4.0 tomlkit-0.11.1 urllib3-1.26.10 virtualenv-20.15.1 webencodings-0.5.1 zipp-3.8.1                                                                                                                                      Removing intermediate container 71c95faf2e53
 ---> 78de8bd1a51b
...
Step 10/13 : RUN wget -q https://github.com/matrix-org/synapse/archive/develop.tar.gz         -O /synapse.tar.gz &&         mkdir /synapse &&         tar -C /synapse --strip-components=1 -xf synapse.tar.gz &&         ln -s -T /venv /synapse/.venv &&         cd /synapse &&         poetry install -q --no-root --extras all &&         rm -rf `poetry config cache-dir` &&         rm -rf /synapse &&         rm /synapse.tar.gz
 ---> Running in 338dba064dcc
Traceback (most recent call last):
  File "/usr/local/bin/poetry", line 6, in <module>
    from poetry.console import main
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/application.py", line 7, in <module>
    from .commands.about import AboutCommand
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/commands/__init__.py", line 4, in <module>
    from .check import CheckCommand
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/commands/check.py", line 2, in <module>
    from poetry.factory import Factory
  File "/usr/local/lib/python3.7/dist-packages/poetry/factory.py", line 18, in <module>
    from .repositories.pypi_repository import PyPiRepository
  File "/usr/local/lib/python3.7/dist-packages/poetry/repositories/pypi_repository.py", line 11, in <module>
    from cachecontrol import CacheControl
  File "/usr/local/lib/python3.7/dist-packages/cachecontrol/__init__.py", line 13, in <module>
    from .wrapper import CacheControl
  File "/usr/local/lib/python3.7/dist-packages/cachecontrol/wrapper.py", line 5, in <module>
    from .adapter import CacheControlAdapter
  File "/usr/local/lib/python3.7/dist-packages/cachecontrol/adapter.py", line 11, in <module>
    from .controller import CacheController, PERMANENT_REDIRECT_STATUSES
  File "/usr/local/lib/python3.7/dist-packages/cachecontrol/controller.py", line 17, in <module>
    from .serialize import Serializer
  File "/usr/local/lib/python3.7/dist-packages/cachecontrol/serialize.py", line 10, in <module>
    import msgpack
ModuleNotFoundError: No module named 'msgpack'

msgpack is definitely installed in step 6. For whatever reason, msgpack alone gets installed in /usr/lib/python3.7/site-packages/, while everything else ends up in /usr/local/lib/python3.7/dist-packages/. Its version is also not reported in the Successfully installed line.

[squahtx]

If python3 -m pip install --no-cache-dir poetry==1.1.12 is run a second time, pip thinks msgpack is missing and re-does the install:

Step 7/14 : RUN ${PYTHON_VERSION} -m pip install --no-cache-dir poetry==1.1.12
 ---> Running in 54ac7610e129
Requirement already satisfied: poetry==1.1.12 in /usr/local/lib/python3.7/dist-packages (1.1.12)
...
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging<21.0,>=20.4->poetry==1.1.12) (3.0.9)
Collecting msgpack>=0.5.2 (from cachecontrol[filecache]<0.13.0,>=0.12.9; python_version >= "3.6" and python_version < "4.0"->poetry==1.1.12)
  Downloading https://files.pythonhosted.org/packages/22/44/0829b19ac243211d1d2bd759999aa92196c546518b0be91de9cacc98122a/msgpack-1.0.4.tar.gz (128kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Requirement already satisfied: lockfile>=0.9; extra == "filecache" in /usr/local/lib/python3.7/dist-packages (from cachecontrol[filecache]<0.13.0,>=0.12.9; python_version >= "3.6" and python_version < "4.0"->poetry==1.1.12) (0.12.2)
...
Requirement already satisfied: cryptography>=2.0 in /usr/lib/python3/dist-packages (from SecretStorage>=3.2; sys_platform == "linux"->keyring<22.0.0,>=21.2.0; python_version >= "3.6" and python_version < "4.0"->poetry==1.1.12) (2.6.1)
Installing collected packages: msgpack
  Running setup.py install for msgpack: started
    Running setup.py install for msgpack: finished with status 'done'
Successfully installed msgpack
Removing intermediate container 54ac7610e129
 ---> 6405a53b357d

and we get the same error as before afterwards.

[squahtx]

Omitting the --no-cache-dir option fixes things, somehow.

Step 7/14 : RUN ${PYTHON_VERSION} -m pip install poetry==1.1.12
 ---> Running in d46e74d95375
Requirement already satisfied: poetry==1.1.12 in /usr/local/lib/python3.7/dist-packages (1.1.12)
...
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging<21.0,>=20.4->poetry==1.1.12) (3.0.9)
Collecting msgpack>=0.5.2 (from cachecontrol[filecache]<0.13.0,>=0.12.9; python_version >= "3.6" and python_version < "4.0"->poetry==1.1.12)
  Downloading https://files.pythonhosted.org/packages/22/44/0829b19ac243211d1d2bd759999aa92196c546518b0be91de9cacc98122a/msgpack-1.0.4.tar.gz (128kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Requirement already satisfied: lockfile>=0.9; extra == "filecache" in /usr/local/lib/python3.7/dist-packages (from cachecontrol[filecache]<0.13.0,>=0.12.9; python_version >= "3.6" and python_version < "4.0"->poetry==1.1.12) (0.12.2)
...
Requirement already satisfied: cryptography>=2.0 in /usr/lib/python3/dist-packages (from SecretStorage>=3.2; sys_platform == "linux"->keyring<22.0.0,>=21.2.0; python_version >= "3.6" and python_version < "4.0"->poetry==1.1.12) (2.6.1)
Building wheels for collected packages: msgpack
  Running setup.py bdist_wheel for msgpack: started
  Running setup.py bdist_wheel for msgpack: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/11/d4/bf/9213122db9816a32382b176cdb92ce62cb18fecf3a2a2d0307
Successfully built msgpack
Installing collected packages: msgpack
Successfully installed msgpack-1.0.4
Removing intermediate container d46e74d95375
 ---> 70994d694c71

[squahtx]

msgpack 1.0.4 introduced build dependencies on Cython and setuptools, whereas previously, msgpack 1.0.3 would use whatever version of setuptools was available.

When pip installing msgpack 1.0.3, the default version of setuptools is used, which is 40.8.0 in the container.
When pip installing msgpack 1.0.4, pip uses the latest version of setuptools, 63.2.0.

pip installing msgpack 1.0.3 with setuptools 63.2.0 installs msgpack to the wrong location too, so it's not a regression in msgpack 1.0.4.

I tried installing with a bunch of different setuptools versions and it seems that between setuptools 47.3.2 and 48.0.0, something changed in setuptools that causes msgpack to be installed in /usr/lib/python3.7/site-packages/ instead of /usr/local/lib/python3.7/dist-packages/ when the --no-cache-dir option is specified.

[squahtx]

Filed as pypa/setuptools#3457