Presence of a pyproject.toml file causing unexpected behavior #9738
Comments
omry
added
S: needs triage
|
If you add a |
pradyunsg
added
C: build logic
|
I see. Is this behavior documented in the pip docs? I actually tried to look for build time dependencies and failed to find anything helpful. |
|
I tried this: But I couldn't figure out how to specify a local dependency. Based on this SO question, it was not supported 8 months ago. As a fallback, is there a way to get pip to ignore the |
|
It's mostly documented in PEPs 517 and 518, as those are the relevant standards and pip implements standard defined behaviour. There's no standard-compliant way to specify a build dependency as a local file relative to the source directory, no. I'm assuming that putting your towncrier configuration in a file other than But assuming not, then there are some short-term alternatives. I describe these as "short term", but there's no immediate plan for pip to remove them - it's just that ultimately, pip (and the packaging ecosystem in general) will only support PEP 517/518, and at that point you'll need to have found a standard way of implementing your workflow. But for now, you can probably use |
|
Thanks for your answers @pfmoore. Black is also using pyproject.toml for configuration. It does support specifying a path in the command line but it's not very friendly to people running black manually. Using one of the two plugins is reasonable workaround for now. It looks like pip is using the presence of pyprojects.toml as the gate for the pep517 logic. My use case is as follows:
|
No, there isn't. This was an intentional design choice to make downstream packages to start adopting the newer packaging practices, such as build isolation and PEP 517, when they try to use the newer standards for the same. There have been requests for adding such an escape hatch for a while now, and the answer to those is usually to figure out how to make that package work within the model for package builds that we want in the future. As it stands, you can't have the nice things provided by PEP 518 and PEP 517 (pyproject.toml) without spending some effort to make sure your package is compatible with how the newer build process works (build isolation, PEP 517 backends etc). We haven't so far seen a compelling reason to allow using the niceties of the new standards without moving away from practices that we're pushing people away from.
You can use https://www.python.org/dev/peps/pep-0517/#in-tree-build-backends, if what you want to do is have code used only for performing builds. Note that you'd need to wrap an existing PEP 517 build backend for this and I don't think the setuptools backend would be easy to wrap. |
No there isn't, sorry.
The wording from PEP 517 is as follows:
What pip is doing if the Using
No. Doing so would need changes to the standards, and I'm not aware anyone has proposed any such changes. I don't think it's a use case that is sufficiently widely used to have come to the attention of anyone willing to propose a standard (which, just to be clear, means "anyone" - there's no barrier on proposing standards, other than the requirement to be willing to put in the work).
Not that I know of, sorry. Maybe someone else can offer suggestions. I think most people would distribute the build helper - but that's just a guess and as you say may not be appropriate for you. |
Wait, using the |
See discussion in #8437 and especially this comment #8437 (comment) |
Sorry, I was simplifying (or maybe trying too hard to be precise while simplifying). It's the "going through PEP 517" which is the incompatibility, as that triggers build isolation, but yes, the inability to import the local helper is because of build isolation. I wanted to emphasise that Off-topic for this issue, but there are other incompatibilities - for example the legacy backend supports |
|
I just went through all the comments in #8437 (took me some time). opting out of build isolationI am going to ignore the requests to enable opting out of build isolation for a moment (through the content of pyproject.toml). This would eliminate the problem for me in the short term. I don't want to add additional pressure to do that though. I read the discussion and I will find a way to work with your decisions there. Using in-tree build logicWhat I am trying to do is to run in-tree logic as part of the build process. Currently this is possible by adding logic to setup.py. Specifically I have logic to determine the version of my package and I also generate some code with a parser generator (I have some free functions and some build Commands right now). Even excluding the question of how to share some functionality between different setup.py files in my repo, there is a bigger underlying question: A bad solutionIt looks like the escape hatch is basically "make your own build backend", something like: [build-system]
requires = ["setuptools", "wheel"]
backend-path = ["build_helper"]
build-backend = "build_helper"def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
# my own wheel build here
def build_sdist(sdist_directory, config_settings=None):
# my own sdist build hereI don't think we want each project to make their own broken build system though. A good (?) solutionI think support for local build dependencies will offer the cleanest solution for my problem. There is probably a pretty large set of projects with complex setup.py that is using in-tree logic, and if I understand things correctly - all those will have a hard time transitioning into the bright new future. |
|
@omry Thanks for your thoughtful comments. This is an area where opinions can get very polarised because everyone has their own views as to what the "key use case" for pip should be. I appreciate you taking the time to describe your situation objectively and clearly.
That's a notoriously hard statement to quantify. All I can reasonably say here is that we have seen a number of people complaining about build isolation, but the reasons are varied - and I don't recall any other case of someone with build helper code like you describe. There may be a lot of people struggling with this, but they are suspiciously quiet if so (unfortunately, we have direct experience of what "a pip feature that inconveniences a lot of people" looks like 🙁) The PEP 517 viewFrom the perspective of PEP 517, the key is to clearly separate the responsibilities of the installer front end and the build back end, and to ensure that the user (both end user and project developer) does not have to worry about implementation details. Build isolation is an important part of that, as it protects the installer/build system from worrying about what the user might have installed. But conversely, it does make it essential that projects have a way to accurately describe their expectations for the build environment. Another important part of the equation is that projects are self-contained. To that end, referencing arbitrary locations relative to the source tree is not allowed, because it could result in builds failing because a sibling directory was missing, or something of that nature. However, there is a good use case for projects wanting to include build-specific code within the project. The in-tree hook mechanism was designed for that scenario (although the use case that inspired it was bootstrapping build backends, which is slightly different than your use case). Using in-tree hooksYou quite reasonably expressed reservations about using the in-tree backend mechanism. However, I think the problem here is a lack of good documentation or good examples, rather than that being a bad solution. The issue is that we designed the in-tree hook mechanism based on a specific use case, that of a build back end wanting to build itself. But the mechanism is general, and can be used for any in-tree build code, it's just that no-one has really used it like that, so there aren't many good examples. Like most things, someone has to go first, and write up their experiences for everyone else 🙂 To address your specific concerns:
Absolutely. But there's no reason why projects can't extend an existing build system: # Not changing this one, so just import it and re-export it unchanged
from setuptools.build_meta import build_sdist
# Wrapping this, so we rename it for our own use
from setuptools.build_meta import build_wheel as orig_build_wheel
def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
# Add helper code to build environment - might even just be sys.path.append(location)
orig_build_wheel(wheel_directory, config_settings, metadata_directory)That still requires projects to implement some bits themselves, but it's not too bad.
The above is the sort of thing you can do. A solution using a "normal" backendIf the use case is sufficiently important, I can imagine backend wrappers being developed and published on PyPI, so you might have:
[build-system]
# wrap-setuptools will add setuptools and wheel to the build requirements,
# so no need to list them explicitly
requires = ["wrap_setuptools"]
build-backend = "wrap_setuptools:add_local_helper"
[tool.wrap_setuptools]
helpers = ["./build_helper"]The hypothetical I think What does this mean for pip?I'll freely admit that the above is mostly just theorising, based on what I know of the PEP 517 design, and how I personally imagine the build backend ecosystem developing. There's plenty of work to do to make this a reality. But for me, there are crucial advantages:
I'm not saying that pip will never entertain doing anything that helps in situations like this, but I think such features would need to be weighed against the fact that backend wrapper solutions like this could be developed outside of pip. In particular, it's important to remember that pip features are not "better" just because you¹ pay the cost of writing a backend wrapper but the pip developers pay the cost of a pip feature. I could easily imagine a wrapper backend being cheaper in overall terms than a pip feature. ¹ "you" generally, not you @omry 🙂 |
|
@pfmoore , I definitely understand (and fear) the pain of having a large community with very strict expectations.
I am going to throw a few darts at big projects, and then I am going to look at them to see if I suspect they will have issues. Here are the setup.py files of each I could along with my untested judgement if they will have issues or not to transfer to a pure data driven build system:
Tally: Granted, those are all big projects and it's certainly possible that they can be ported without a major effort. I am going to respond to the rest later. I just wanted to put this 1 hour survey in its own comment. |
|
I got a slightly simpler solution than the "Using in-tree hooks" one you proposed here. Turns out any functions exported by the 'build backend' are also available in setup.py. I tried to do something similar to your improved suggestion ("A solution using a "normal" backend") to see if I can make it work and I was not able to.
I did not find a way to determine the original build dir from within the build backend file. Your point about the benefit of an isolated build environment are clear. If I have: Installing plugins/plugin1, pip will not see the complete project source tree. However, for some project it makes sense to consider the whole checkout tree, even if the installation is of a sub directory. |
That is in violation of PEP 517, which views the directory with Let me ask this. If you built a sdist for this project, what would it contain? Assume for now that you do
You don't need to, that file is (by definition) at the root of the source tree, so it's in your CWD (build hooks are called with the source directory as CWD).
I suspect that you're meaning "files outside the source tree" when you say "rest" (if we accept that the source tree is rooted in the directory with
No one is suggesting a "pure data driven build system". Setuptools is, and probably always will be, a procedural build system. What we're trying to do is to make more of the build metadata and configuration that is common to all build backends, available without needing to run the project build process. But the build step itself can do whatever it likes. The root cause of this particular issue is that one piece of data that front ends want to know is "how do I set up a fresh environment which I can use to run the build steps for this project?" Your examples have fundamentally all been of the form "you need to install package X, but package X isn't publicly available". Hmm, putting the question this way makes me think. In your very original example: why can't import sys
from pathlib import Path
sys.path.append(Path(__file__).parent / "build_helper")There's no need to actually install build-helper in this case - it's present in the source tree (and presumably shipped in the sdist) so just use it directly. I assume there's probably some other constraint that you didn't mention, but I'm not clear what that is. |
Yeah, I agree. I am expecting the sdist to contain the plugin only. When I
I meant passing config to the build hook from pyproject.toml, something like what you described: [tool.wrap_setuptools]
helpers = ["./build_helper"]
The example in my repro was something that just reproduced my the change in behavior when pyproject.toml was in the repo, it was not emulating the actual problem I had. Playing it out, I would need: The plugin pyproject file would need access to the build hook and it's not in the source tree so I think this is not going to work. Zooming out a bit: I am trying to apply the same thing (in spirit) to the build itself. |
|
Here is a more realistic emulation of the real problem. Turns out this doesn't work even without pyproject.toml (and even with --no-build-isolation). While creating it I realized I was probably misinterpreting what you meant by build isolation.
It appears that the temp location is used even with the flag --no-build-isolation. |
|
OK. I think the problem here is fundamentally that pip (and indeed, Python packaging in general) works with projects, which are a single directory tree with a
But that's exactly what I said wasn't supported - the
So I think you're just going to have to accept that the structure you're using isn't supported (by PEP 517, pip or in general by the packaging ecosystem). Setuptools lets you do this, because setuptools basically lets you do anything you want, but the direction of the packaging ecosystem is generally away from that sort of unstructured referencing of data outside of the project. There's no reason you can't keep the project structure and workflow that you're using at the moment, but you just need to realise that newer versions of tools are likely to stop supporting this (much like no-one now supports building Python extensions using plain distutils, or using raw makefiles like we used to in pre-distutils days). Whether the cost of using older tools, or applying increasingly hacky workarounds, is worse than the cost of changing your project structure, is something only you can decide, TBH. |
Yes. As @sbidoul spotted, this is not related to build isolation, but rather to building the project in a copy of the source tree. I'm confused as to why you originally only hit this when you have a My musings above still apply, though - what you're doing isn't really the model that the existing standards are based on, so you should expect to hit some rough spots like this. |
|
@sbidoul, @pfmoore, I hear you. |
I already had pyproject.toml files (for towncrier and black) in place. |
There is currently no plan to enable that on a per-project basis. The goal is to make in-tree builds the default and only way, at some point in the future, following a deprecation period for the current behaviour. |
Could be a good addition to support configuring feature flags in general, but that's a whole new discussion. |
|
I am closing this, thanks for the discussion and help. Here is a short summary (happy to update if I missed anything important):
|
pip version
21.0.1
Python version
3.8.0
OS
Ubuntu 18.04
Additional information
No response
Description
Somehow, the presence of a pyproject.toml file is causing pip install from a directory to fail importing of installed modules.
Expected behavior
The presence of pyproject.toml does not impact the behavior of pip install.
How to Reproduce
This is a bit complicated to explain minimal repro here should be enough.
Structure:
Content of files in repro repository.
Output
Installing build_helper:
Successfully installing primary project:
Creating an empty pyproject.yaml, after which installation fails:
Code of Conduct
The text was updated successfully, but these errors were encountered: