Python builds using Poetry break with 20201129.1

[shadycuz]

Description
Builds in github actions have broken and the only change I can find is that the version has changed from Version: 20201115.1 to 20201129.1

Area for Triage:

Python
Question, Bug, or Feature?:

Bug
Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

I have not tried on 16.04
Expected behavior
Since the build did not break locally and python, poetry and nox all have their version pinned, I expected builds to continue to work.

Actual behavior
Builds fail with a pip and or poetry issue.

nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp2k2e6e06
nox > pip install --constraint=/tmp/tmp2k2e6e06 flake8 flake8-black flake8-bugbear flake8-import-order
nox > Command pip install --constraint=/tmp/tmp2k2e6e06 flake8 flake8-black flake8-bugbear flake8-import-order failed with exit code 1:
Collecting flake8
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    flake8 from https://files.pythonhosted.org/packages/d4/ca/3971802ee6251da1abead1a22831d7f4743781e2f743bd266bdd2f46c19b/flake8-3.8.4-py2.py3-none-any.whl#sha256=749dbbd6bfd0cf1318af27bf97a14e28e5ff548ef8e5b1566ccfb25a11e7c839

Repro steps

I have an issue open with poetry. My PR is here and from my PR you can of course get the details for the passing and failing builds using the Test or Coverage workflows. I have also created a Diff of a working and failing Test builds up to the point where the one fails.

The only thing I find strange about the diff is that in the build that works, it doesn't install some of the packages. For example, if you search for Collecting flake8 you would expect to find it since poetry and nox should make sure it's installed from scratch every build. Even though it doesn't seem to install flake8, flake8 runs just fine. If you go to the failing build and search you see that installing those packages is what is failing the build.

nox > Command pip install --constraint=/tmp/tmp2k2e6e06 flake8 flake8-black flake8-bugbear flake8-import-order failed with exit code 1:
Collecting flake8
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    flake8 from https://files.pythonhosted.org/packages/d4/ca/3971802ee6251da1abead1a22831d7f4743781e2f743bd266bdd2f46c19b/flake8-3.8.4-py2.py3-none-any.whl#sha256=749dbbd6bfd0cf1318af27bf97a14e28e5ff548ef8e5b1566ccfb25a11e7c839

Edit:

Also of note from the dff that the failing builds on 20201129.1 complain about pip not being latest:

WARNING: You are using pip version 20.2.4; however, version 20.3.1 is available.
You should consider upgrading via the '/opt/hostedtoolcache/Python/3.8.6/x64/bin/python -m pip install --upgrade pip' command.

This warning about pip not being fully updated is not seen on the passing builds from the earlier version. Edit: I just checked and updating pip did not fix the builds. Successfully installed pip-20.3.1

Edit: Here is the diff between the working image and the newer one that is not working. ubuntu18/20201115.1...ubuntu18/20201129.1

If you ctrl-f and search for unbuntu1804 you can quickly locate the readme and json files for this image version. According to the readme, the python or pip version should not have changed at all. The only python related change I could see in the readme was a change to PyPY version and it was only a patch bump.

The other change from the json shows that this script is not being run https://github.com/actions/virtual-environments/blob/ubuntu18/20201115.1/images/linux/scripts/installers/test-toolcache.sh on the newer image.

The only thing strange I found while reviewing the diffs is that the image version in the readme does not match the tag.

It should be version 20201115.1 not 20201108.1, I'm guessing this is a bug in the diff viewer? Which makes me question my findings so far, or maybe it's using a cached version of the tag and this tag was forced push to fix the version info? Now sure.

[dsame]

@shadycuz while i am investigating the problem, can you please provide a reference to the build that actually fails and provide a minimal repro with a bit less than 14 files PR?

[shadycuz]

@dsame I have opened a new PR and the only change is the README.

All builds for workflow "Tests" >= 19 are failing because of version 20201129.1.

All previous builds passed but on version 20201115.1. The last build to pass was for a merge into master.

The only change between the last working build and the latest failed build is the readme and the version of the image.

Please let me know if I can provide any other info or assistance.

Thanks

[dsame]

@shadycuz thanks a lot, not the problem seems to be clear

[shadycuz]

@dsame

Someone has tracked it down to an issue with newer pip versions. I tried forcing pip to a pre 20.3 version but that did not work. What did work for pinning virtualenv to a different version.

See this working build. Also here is the pip issue.

[shadycuz]

I feel good enough about closing this now. The issue is with the change to how pip handles the constraints flag. pip maintainers have confirmed its not a bug but a design choice based on the new pip resolver.

Workarounds:

• Stop passing in packages with pinned hashes using the pip --contstraints flag.
• If you are using pip directly then pin it to a version before 20.3
  python -m pip install --upgrade pip==20.2.4
• If you are using something else like virtualenv that depends on pip, make sure you pin it.
  python -m pip install --upgrade virtualenv==20.0.26
  Or use a env var VIRTUALENV_PIP=20.2.4