setuptools <49.0.0 does not handle Python 3.10 correctly in `python_version` environment markers #1248

sileht · 2021-10-11T10:26:40Z

The newly added python 3.10 doesn't work with pip 20 and setuptools 47

old pip/setuptools version thinks python 3.10 is python 3.1:

Python 3.10 requires pip>=21 and setuptools>=52 (https://github.com/python/cpython/blob/3.10/Lib/ensurepip/__init__.py)

The text was updated successfully, but these errors were encountered:

edmorley · 2021-10-11T10:42:36Z

@sileht Hi! Thank you for opening an issue :-)

The Python buildpack doesn't use ensurepip to install the versions of Pip or setuptools in the build for a few reasons, and instead installs the version of Pip/setuptools defined here:

heroku-buildpack-python/bin/steps/python

Lines 153 to 167 in ec2fd78

    
           PIP_VERSION='20.2.4' 
        
           # Must use setuptools <47.2.0 until we fix: 
        
           # https://github.com/heroku/heroku-buildpack-python/issues/1006 
        
           SETUPTOOLS_VERSION='47.1.1' 
        
           WHEEL_VERSION='0.36.2' 
        
           if [[ "${PYTHON_VERSION}" == ${PY34}* ]]; then 
        
             # Python 3.4 support was dropped in pip 19.2+, setuptools 44+ and wheel 0.34+. 
        
             PIP_VERSION='19.1.1' 
        
             SETUPTOOLS_VERSION='43.0.0' 
        
             WHEEL_VERSION='0.33.6' 
        
           elif [[ "${PYTHON_VERSION}" == ${PY27}* || "${PYTHON_VERSION}" == ${PYPY27}* ]]; then 
        
             # Python 2.7 support was dropped in setuptools 45+. 
        
             SETUPTOOLS_VERSION='44.1.1' 
        
           fi

We're currently using Pip 20.2.4 since it's the last release before the new resolver became the default, and there are still several compatibility issues that make upgrading to newer pip painful (more in #1109). Thankfully the upcoming Pip 21.3 release (due within days, though we will want to wait for any dust to settle) resolves the most significant of those, so we should be good to update soon.

For setuptools, we're currently using 47.1.1, since 47.2.0 and newer are affected by #1006. The long term fix for that is changing the way the build system works, so that the directory in which builds occur is the same as that which is used at runtime. However doing that breaks some other buildpacks, so is not a quick change (and that work has had to be put on hold).

Re Python 3.10 specifically, do you have some examples of things that are failing currently using the above pip/setuptools versions?

Our CI is currently passing, and the pip issue linked in the issue description was fixed in pip 19.3, so shouldn't affect Python 3.10 usage on Heroku.

Many thanks :-)

sileht · 2021-10-11T11:13:11Z

Yeah, I'm aware the buildpack doesn't use ensurepip, I just put the link as a reference for the recommended version.

Here is an example of a backtrace, you have some others in bug reports I posted in the description.

Traceback (most recent call last):
  File "/app/.heroku/python/bin/mergifyio-import-check", line 33, in <module>
    sys.exit(load_entry_point('mergifyio', 'console_scripts', 'mergifyio-import-check')())
  File "/app/mergifyio/import_check.py", line 6, in check
    from mergifyio.web import app as application  # noqa
  File "/app/mergifyio/web.py", line 18, in <module>
    from mergifyio.blueprints import badges
  File "/app/mergifyio/blueprints/badges.py", line 5, in <module>
    import pkg_resources
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3262, in <module>
    def _initialize_master_working_set():
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3245, in _call_aside
    f(*args, **kwargs)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3274, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pep562; python_version < "3.7"' distribution was not found and is required by ddtrace

sileht · 2021-10-11T11:15:38Z

Our CI is currently passing, and the pip issue linked in the issue description was fixed in pip 19.3, so shouldn't affect Python 3.10 usage on Heroku.

Does your CI install any libraries that use python environment markers in requirements? if not that may explain why you didn't trigger the issue.

sileht · 2021-10-18T07:34:05Z

Hi @edmorley, maybe you can run the tests on the pull request I send, it should trigger the issue.

edmorley · 2021-10-18T13:23:05Z

@sileht Hi! Sorry for the delay and thank you for the PR to try and reproduce.

I have triggered CI on that PR, however it doesn't show the issue - the pep562 package is correctly ignored:

remote: -----> Building on the Heroku-20 stack
remote: -----> Using buildpack: https://github.com/heroku/heroku-buildpack-python.git#pr/1249
remote: -----> Python app detected
remote: -----> Using Python version specified in runtime.txt
remote: -----> Installing python-3.10.0
remote: -----> Installing pip 20.2.4, setuptools 47.1.1 and wheel 0.36.2
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Ignoring pep562: markers 'python_version < "3.7"' don't match your environment
remote:        Collecting urllib3
remote:          Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
remote:        Installing collected packages: urllib3
remote:        Successfully installed urllib3-1.26.7
remote: -----> Discovering process types
remote:        Procfile declares types -> (none)

See:
https://app.circleci.com/pipelines/github/heroku/heroku-buildpack-python/218/workflows/a7eaaeb1-9fb1-420b-bb50-bbb6b1e2f97a/jobs/933

Looking at the traceback in the earlier comment, I see it's originating via pkg_resources, which is part of setuptools - which may suggest an issue with setuptools rather than pip, and so why the requirements.txt file based testcase wasn't affected.

Could you open a support ticket (and give app access permission, and link to the ticket from here) so I can take a closer look at your app to try and work out a reduced testcase that demonstrates the issue?

edmorley · 2021-10-18T20:48:34Z

I was able to reproduce when using:

Python 3.10
A requirements.txt file containing an editable local package specifier (eg -e ./local_package/)
The local package containing a setup.py / setup.cfg with an install_requires of pep562; python_version < "3.7" and an entrypoint set
A bin/post_compile that invokes said entrypoint

Example failure log:

$ make compile FIXTURE=spec/fixtures/requirements_editable/
Running compile using: STACK=heroku-20 FIXTURE=spec/fixtures/requirements_editable/

-----> Using Python version specified in runtime.txt
-----> Installing python-3.10.0
-----> Installing pip 20.2.4, setuptools 47.1.1 and wheel 0.36.2
-----> Installing SQLite3
-----> Installing requirements with pip
       Obtaining file:///build/local_package (from -r /build/requirements.txt (line 1))
       Obtaining gunicorn from git+https://github.com/benoitc/[email protected]#egg=gunicorn (from -r /build/requirements.txt (line 2))
         Cloning https://github.com/benoitc/gunicorn (to revision 20.1.0) to /app/.heroku/src/gunicorn
       Installing collected packages: gunicorn, local-package
         Running setup.py develop for gunicorn
         Running setup.py develop for local-package
       Successfully installed gunicorn local-package
-----> Running post-compile hook
Running entrypoint for the local package: Traceback (most recent call last):
  File "/app/.heroku/python/bin/local_package", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3262, in <module>
    def _initialize_master_working_set():
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3245, in _call_aside
    f(*args, **kwargs)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3274, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pep562' distribution was not found and is required by local-package

Upgrading setuptools to 49.0.0 or newer resolved the issue - I believe this is the change that fixed it:
pypa/setuptools@a877dab

I'll prioritise finding a shorter term fix for #1006 (compared to the ideal fix which is further out), so we can finally update setuptools.

@sileht In the meantime, could you try adding setuptools>=49.0.0 to the top of your requirements.txt to see if that work around the issue?

sileht · 2021-10-18T20:58:23Z

I setup a tiny project that reproduces the issue, too, but it looks like you just beat me 😛

https://github.com/sileht/heroku-py3.10-bug

Anyways, upgrading setuptools, workaround the issue.

edmorley · 2021-10-18T21:00:31Z

Thank you! I'll look into this more tomorrow :-)

Currently the build system performs builds in a different directory (`/tmp/build_<hash>`) to which the app will be run at runtime (`/app`). This means that any hardcoded paths in the slug must be rewritten by the buildpack, so the app still works after being moved. One such case of hardcoded paths, are the `.pth` and `.egg-link` files that are created in the `site-packages` directory by Pip/setuptools for editable package installs (aka develop mode). The most common way someone might use editable mode is via `-e <package specifier>` entries in their `requirements.txt` file. Until now, the Python buildpack rewrote paths inside these files during the compile itself, which meant the build-time paths were no longer present when subsequent buildpacks ran. This happened to work due to an interaction of legacy setuptools behaviour and a buildpack bug, but stops working in setuptools 47.2.0 or later - as described in #1006. Longer term we would like to stop building in one location and running the app from another, so that the path rewriting isn't required at all. However that change breaks some other buildpacks so requires a long-term transition plan. In the meantime, this change moves path rewriting to a `.profile.d/` script, so that it occurs at runtime, and so after all other buildpacks have run. Additional test coverage of editable packages was added previously in #1251 and #1253, and has confirmed that this new `profile.d/` script approach will prevent the issues in #1006 when setuptools is updated in a future PR. There is one subtle implication of moving this path rewriting, in that subsequent cached builds will no longer see the existing package as being already installed, so won't uninstall if before reinstalling it (as seen in the test log output change). However this is not believed to have any significant impact. Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.)

…#1252) Currently the build system performs builds in a different directory (`/tmp/build_<hash>`) to which the app will be run at runtime (`/app`). This means that any hardcoded paths in the slug must be rewritten by the buildpack, so the app still works after being moved. One such case of hardcoded paths, are the `.pth` and `.egg-link` files that are created in the `site-packages` directory by Pip/setuptools for editable package installs (aka develop mode). The most common way someone might use editable mode is via `-e <package specifier>` entries in their `requirements.txt` file. Until now, the Python buildpack rewrote paths inside these files during the compile itself, which meant the build-time paths were no longer present when subsequent buildpacks ran. This happened to work due to an interaction of legacy setuptools behaviour and a buildpack bug, but stops working in setuptools 47.2.0 or later - as described in #1006. Longer term we would like to stop building in one location and running the app from another, so that the path rewriting isn't required at all. However that change breaks some other buildpacks so requires a long-term transition plan. In the meantime, this change moves path rewriting to a `.profile.d/` script, so that it occurs at runtime, and so after all other buildpacks have run. Additional test coverage of editable packages was added previously in #1251 and #1253, and has confirmed that this new `profile.d/` script approach will prevent the issues in #1006 when setuptools is updated in a future PR. There is one subtle implication of moving this path rewriting, in that subsequent cached builds will no longer see the existing package as being already installed, so won't uninstall if before reinstalling it (as seen in the test log output change). However this is not believed to have any significant impact. Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.) GUS-W-7828034.

Updates: - setuptools from 47.1.1 to: - 50.3.2 for Python 3.5 - 57.5.0 for Python 3.6+ - wheel from 0.36.2 to 0.37.0. Of note, the newer setuptools is fully compatible with Python 3.10, thereby fixing #1248. Updating to newer setuptools was blocked on #1006, but that's now been fixed by #1252. The setuptools version hasn't been updated all the way to the latest (58.2.0), since v58 dropped support for 2to3, which caused breakage in a few packages, so I would rather hold off as long as possible (and there are no fixes that we need since then). Release notes: https://setuptools.pypa.io/en/latest/history.html#v57-5-0 https://wheel.readthedocs.io/en/stable/news.html Full changelogs: pypa/setuptools@v47.1.1...v57.5.0 pypa/wheel@0.36.2...0.37.0 Fixes #1248. GUS-W-10052807.

edmorley · 2021-10-19T17:11:52Z

Thank you for the report! This is now fixed by #1254, which is available on main (by specifying a GitHub buildpack URL) and will be released to the heroku/python buildpack registry alias tomorrow via #1255.

sileht · 2021-10-19T17:13:21Z

Awesome! Thanks @edmorley

sileht mentioned this issue Oct 11, 2021

ci(tests): add environments marker to python3.10 test #1249

Closed

edmorley self-assigned this Oct 18, 2021

edmorley mentioned this issue Oct 19, 2021

Perform editable package .pth and .egg-link path rewriting at runtime #1252

Merged

edmorley changed the title ~~python 3.10 installation is broken~~ Current version of setuptools does not handle Python 3.10 correctly in python_version environment markers Oct 19, 2021

edmorley added the bug label Oct 19, 2021

edmorley mentioned this issue Oct 19, 2021

Update setuptools and wheel #1254

Merged

edmorley linked a pull request Oct 19, 2021 that will close this issue

Update setuptools and wheel #1254

Merged

edmorley changed the title ~~Current version of setuptools does not handle Python 3.10 correctly in python_version environment markers~~ setuptools <49.0.0 does not handle Python 3.10 correctly in python_version environment markers Oct 19, 2021

edmorley closed this as completed in #1254 Oct 19, 2021

abrichr mentioned this issue Sep 7, 2022

Trouble installing 1.2.0a0 from pip chriskiehl/Gooey#809

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setuptools <49.0.0 does not handle Python 3.10 correctly in `python_version` environment markers #1248

setuptools <49.0.0 does not handle Python 3.10 correctly in `python_version` environment markers #1248

sileht commented Oct 11, 2021

edmorley commented Oct 11, 2021

sileht commented Oct 11, 2021

sileht commented Oct 11, 2021

sileht commented Oct 18, 2021

edmorley commented Oct 18, 2021 •

edited

Loading

edmorley commented Oct 18, 2021

sileht commented Oct 18, 2021

edmorley commented Oct 18, 2021

edmorley commented Oct 19, 2021

sileht commented Oct 19, 2021

setuptools <49.0.0 does not handle Python 3.10 correctly in python_version environment markers #1248

setuptools <49.0.0 does not handle Python 3.10 correctly in python_version environment markers #1248

Comments

sileht commented Oct 11, 2021

edmorley commented Oct 11, 2021

sileht commented Oct 11, 2021

sileht commented Oct 11, 2021

sileht commented Oct 18, 2021

edmorley commented Oct 18, 2021 • edited Loading

edmorley commented Oct 18, 2021

sileht commented Oct 18, 2021

edmorley commented Oct 18, 2021

edmorley commented Oct 19, 2021

sileht commented Oct 19, 2021

setuptools <49.0.0 does not handle Python 3.10 correctly in `python_version` environment markers #1248

setuptools <49.0.0 does not handle Python 3.10 correctly in `python_version` environment markers #1248

edmorley commented Oct 18, 2021 •

edited

Loading