celery 5.2.3 fails to install due to requiring newer setuptools

[tadeoos]

Hey folks,

This is in the exact same spirit as issue #949

Here's what we're seeing when deploying:

remote: -----> Using Python version specified in runtime.txt
remote: -----> No change in requirements detected, installing from cache
remote: -----> Using cached install of python-3.8.12
remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
remote: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
remote: celery 5.2.3 requires setuptools<59.7.0,>=59.1.1, but you have setuptools 57.5.0 which is incompatible.

Can we bump the setuptools version to be >=59.1.1 or is there other reccomended way to handle this?

[edmorley]

Hi! Thank you for reporting this :-)

So I'd love to be able to update to the latest (or even, any newer) setuptools, however setuptools v58.0.0 included a breaking change that affects some packages (see the number of ping-backs in the issue history from other projects having to apply fixes):
https://setuptools.pypa.io/en/latest/history.html#v58-0-0
pypa/setuptools#2086
pypa/setuptools#2731

Whilst the number of affected packages is only a minority in the ecosystem, at cloud-scale that's still potentially hundreds/thousands of apps.

We'll obviously need to update setuptools at some point, however only once the tipping point of cost (gradually decreasing number of affected packages + people updating their app's deps to the fixed versions) is outweighed by the benefits (an increasing number of packages that only work on newer setuptools).

In the meantime, the setuptools version can be overridden if needed, by adding a setuptools version to requirements.txt explicitly (I would recommend at the top).

However looking at this Celery error specifically, it seems that they don't actually rely on any newer setuptools features, and that the version range made more restrictive than perhaps they intended (it only needed to set an upper version bound, but set a very restrictive lower bound too). (The setuptools version change was made in Celery v5.2.3 in celery/kombu#1466, in order to work around issues with setuptools 59.8.0+ pypa/setuptools#2936).

I was going to file an issue asking Celery to widen the version range, however in the last few days they've already adjusted it, in:
celery/kombu#1481
celery/celery#7218

See also:
celery/celery#7214
celery/celery#7202

It looks like that fix will be in the next Celery release (presumably v5.2.4).

With that release, Celery will work with the setuptools version used by this buildpack again, and I haven't heard of any other packages needing newer setuptools - so we're still best staying put on the existing setuptools version for now :-)