Simplify vendoring and declare dependencies optionally #4457
Conversation
jaraco
force-pushed
the
debt/2825-devendor
branch
from
4e4c01e
to
d2704c3
Compare
…(only) to simply install the dependencies to the _vendor folder.
Workaround for pypa/pip#12770.
…ng the vendor dir to sys.path.
…ndencies to the _vendor folder.
…ng the vendor dir to sys.path.
jaraco
force-pushed
the
debt/2825-devendor
branch
from
d2704c3
to
b4b6bf7
Compare
|
It's hard to see in GitHub, due to the changes to the vendored directories, but here's a stat summary excluding Net 363 lines removed. |
There was a problem hiding this comment.
Thank you very much for working on this Jason.
This is a though one. In principle I have no problem with this and I think it would be fine. But it is hard to foresee the consequences of it.
The complicated part is that setuptools has no way of telling the installer/builder that the dependencies are "optional in a way", and that if they fail to install the dependencies (e.g. due to cycles) it is fine because we also have them bundled.
For example: if the user passes some flags to pip that forces pip to no use wheels for the installation, will this also mean that the build will fail with this new approach to vendoring?
I believe this is related to pypa/packaging-problems#342. Ideally we should have a standard solution for that problem.
It would be interesting to hear @pfmoore and @pradyunsg thoughts on this matter.
My main thought is that Unfortunately, I don't have any specific examples of people who should be consulted on this question. Maybe opening a discussion on the packaging Discourse would be worthwhile? |
|
Thank you much, @pfmoore. This brings me to the question: if setuptools can build itself with no extra dependency listed on |
|
I'm not sure I follow your question. To build setuptools, the However, if you want to install some project X that uses setuptools, the installer will set up a build environment that contains setuptools (building and then installing it as described above) and it will then install all of the dependencies listed in So I guess my answer is yes, this looks potentially dangerous to me. |
|
I notice that hatchling is able to declare dependencies. Is that because none of the dependencies use hatchling as their build backend (trove-classifiers and pluggy use setuptools, tomli and packaging and pathspec use flit)? Does that mean that hatchling would break the world if pathspec or trove-classifiers were to switch to hatchling? I see hatchling does something clever - it relies on its own ouroboros backend for getting build dependencies as part of I see that flit addresses this concern by implementing the backend with no dependencies (flit-core). Maybe setuptools should do something similar, and have a separate backend with no dependencies that can build setuptools.
Let me see if I can reproduce the danger. I'll get a Docker container that's resource-constrained and try to get the system to fall over building this branch from source. Perhaps I should simply remove the declared dependencies for now, or move them to optional dependencies. That would separate this concern about cyclic (build) dependencies from simplifying the vendoring. |
I tried to reproduce the danger, but have thusfar been unsuccessful. I have this dockerfile: FROM jaraco/multipy-tox
RUN git clone https://github.com/pypa/setuptools
WORKDIR setuptools
RUN git checkout debt/2825-devendor
ENV PIP_NO_CACHE=1
ENV PIP_NO_BINARY=:all:
# disable pip-run
ENV PYTHONPATH=
CMD py -3.12 -m pip install .It appears to be succeeding in installing setuptools and its dependencies (though it fails on What I realized, though, is that when building a dependency and thus getting "setuptools" from source, it's getting it from PyPI, which doesn't yet have the declared dependencies. To simulate the danger, I'll need to upload an sdist of this PR somewhere and use that index.
I took a moment to brush up on pypa/packaging-problems#342, which attempts to capture the issue. I believe it's an outstanding issue and it's only been avoided by the following implicit constraint:
I think I've got that right. A build backend can have dependencies if all of the dependencies rely on a different build backend (hatchling, for now). Or a build backend can have no dependencies (flit_core). I wonder if we should document this constraint somewhere. Is there a place where we could document this constraint? |
jaraco
force-pushed
the
debt/2825-devendor
branch
from
3626cc9
to
bccb86b
Compare
Now that the vendoring doesn't depend on importing the setuptools package, it's safe for pkg_resources to import _from_ there without actually importing setuptools.
This is now necessary since `setuptools >= 71` started preferring externally present stdlib deps over the vendored ones. Refs: * pypa/setuptools#4457 * pypa/setuptools#4483 * pypa/setuptools#2825
| @@ -0,0 +1,166 @@ | |||
| GNU LESSER GENERAL PUBLIC LICENSE | |||
There was a problem hiding this comment.
Sorry to ghost post on you, but why did autocommand get added here? The rest look they were modifications, but this seems like a new _vendor package entirely. I ask because as it is LGPL, we have some licensing concerns. Thanks!
There was a problem hiding this comment.
I am not a maintainer, but I can answer your question of "why": autocommand is a dependency of jaraco.text. Used there: https://github.com/search?q=repo%3Ajaraco%2Fjaraco.text%20autocommand&type=code AFAIK setuptools doesn't need it, it's just pulled in with dependencies. If you really need to I'm sure you can devendor it.
| @@ -74,6 +74,8 @@ | |||
|
|
|||
| import _imp | |||
|
|
|||
| sys.path.extend(((vendor_path := os.path.join(os.path.dirname(os.path.dirname(__file__)), 'setuptools', '_vendor')) not in sys.path) * [vendor_path]) # fmt: skip | |||
There was a problem hiding this comment.
This hack breaks when this package is within a zip file... pypa/virtualenv#1727 cc @jaraco
There was a problem hiding this comment.
Sorry to hear that.
It's not fair to characterize this as a hack. It's the vendoring approach, which is required by the ecosystem to satisfy PEP 517's "no circular build dependencies" constraint. Setuptools is following best-available standards for supplying sophisticated behavior. There's a lot of work going on in pypa/packaging-problems#342 to solve the issue more holistically so that build backends can have and declare proper dependencies.
When I first authored this change, I considered the zipimport case and thought that this approach might work for the zipimport case but I never verified.
Since the release of this change, the vendoring support is there as a fallback for environments that cannot include the dependencies naturally. The best approach would be for the app/environment builder to ensure that setuptools[core] is installed (such that vendoring is unnecessary).
I'd be open to expanding the vendoring support to support better the zipimport case. I'll open an issue about it.
|
This causes #3072 to become an issue again? |
Summary of changes
Although this PR doesn't fully remove vendored packages, it re-writes the logic to make the vendoring layer very thin and optional, removing lots of cruft that's accumulated around it.
In particular, it adds the vendored packages to
sys.pathinstead of proxying them through a submodule. It adds them to the back ofsys.pathto allow naturally-installed packages to take precedence (vendored packages are only a fallback). The dependencies are no longer pinned to a specific version except that the packages in_vendorare for a specific version.Benefits include:
externmodule redirection and associated double-accounting.importlib_metadatahack can be removed (as only one copy ofimportlib_metadatawill ever be imported)._vendordirectory (and supplying the dependencies).In order to avoid cyclic dependencies, the declared dependencies are for now declared in an optional
coreextra so they now appear in the metadata.Ref #2825
Pull Request Checklist
newsfragments/.(See documentation for details)