-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poetry install fails for nested local dependencies and develop = false #3368
Comments
Is it really related to |
I think you're right. I've gone and created an example repo showing the problem: https://github.com/TrevorPace/poetry-localdep-issue The issue occurs actually when even generating the poetry.lock file. I'll update the description of the issue accordingly. |
OK. Then it looks to me like it is simply a use case that is not supported (yet). Looks like your best bet is to add all path dependencies, even the indirect ones, in the
Maybe related: |
I'm not sure about that, because when using "develop = true" it's actually correctly computing the correct paths for pkg2 when creating the .pth files in the virtual environment. There must be actual code looking into the the pyproject.toml of pkg1, resolving the dependencies and installing them into the virtual environment (or pkg2 would be neither installed nor correctly resolved). It seems like when it comes to actually creating the wheel or whatever intermediary it is passing to pip it isn't correctly updating the dependency. Which is why it works fine for pkg1 (because it has no other local dependencies to resolve). It also works fine for the main application, because it doesn't actually create that intermediary (just the app1.egg-info/). I'll take a look at the poetry source to see what's up. |
So, after a bit of a deep dive in the code. I believe what is happening is:
In the case of development dependencies installed locally though this problem is avoided because a custom Builder is used in the PipInstaller which avoids the invocation with pip (and thus the isolated wheel generation step). |
Slightly off-topic, but I just want to add as well, that this whole Couldn't we just always do a local install with our custom Builder, except when providing a build-flag to poetry during install: This way for development we do nothing, and for creating Docker files we would just add the flag and disable creation of virtual environments to have everything installed into the filesystem. If we still wanted to decide to install packages or not we could use the optional flag accordingly. |
Thanks for sharing your debugging experience. I did not follow exactly, but I am sure it will be helpful to the maintainers when they get time to visit this ticket. Personal opinion: I am not fan of path dependencies at all. I see how it could be helpful in many cases, but I feel like actually it can work in very limited cases only. So as a general rule I would really recommend anyone to avoid them as much as possible. This to me looks like the typical kind of features that are added for some very specific and straightforward use cases in mind, but for the creator of the feature it is hard to tell when it is going to break, so no explicit limitations are indicated in the documentation. But still there is a point where it is going to break (same story with editable installations). In my mind, these features are here to help during development only, for example. But users somehow push it a bit too far out of the feature's comfort zone, and it shows its limitations and breaks. In particular I would not expect path dependencies to work on 2nd degree dependencies, to me it seems like expecting too much out of this. Basically I would limit path to direct dependencies. And in particular to scenarios such as this:
|
I think this is actually the issue, I described in another ticket: #2877. In our setup we are also using "develop = false" (which I don't recall why we use it, but there was a reason - I will check again). I think path dependencies are very valuable, especially in monorepo setups (where relative paths are usually fixed). Moving the dependencies to the first level in such a setup defeats the purpose of dependency resolution and adds quite some manual overhead (we have ~ 20 poetry packages with approximately 3 to 4 levels of dependencies within the repository, this is already quite hard to keep up). The current state is problematic as it is inconsistent: Relative path dependencies can be specified and there is no explicit warning/error and no documentation that they would not work transitively. Transitive dependency resolution is quite a strong guarantee I would expect from a solver & installation tool. Looking at the error that is shown right now it seems like there is a state that should be invalid, if the feature is not supported. But as a user of the tool, after reading the documentation, it is not clear to me why this should not be the case. I understand that from an internal standpoint it might be a use case that is harder to support, but I think that should not be a reason to dismiss the use case (unless there is a way to reach the same result, but without the problem with explicit mentioning of transitive dependencies). Right now this is stopping the wider poetry adoption in my organisation and looking at the bug tracker this seems to be something that was already asked multiple times. I would be happy to help out getting this to work with some advice on the implementation. Or help to make sure other people do not run into the same problem by enhancing the error message and providing documentation on the use case and workarounds. |
@malte-klemm-by I'm not a maintainer, but I will try to look at this again, see if I can pin point more accurately what is going wrong here. Digression: I believe the potentially surprising and/or unreliable behaviour becomes quite clear with more knowledge of Python's packaging story in general (not just poetry). In my mind it is clear that things like path dependencies are not first class citizens at all. And from my point of view I also do not know why they should be. If I could I would probably get rid of them entirely. I guess I see how path dependencies could be somewhat useful for things like "mono-repo", although I do not have much knowledge about those (seen from afar I do not see the advantages of mono-repos). Anyway, I do not understand why mono-repos want to bypass the distribution phase. Mono-repo or not, my impression is that it should still be possible to build the individual projects and upload them to an index (PyPI or private repository), and thus get rid of path dependencies. Also I must admit I do not really understand why people bother with building individual projects in a mono-repo. This all seems quite counter-productive, a weird mix of things, that result in doing many things that are against best practices. Why do you need dependency resolution in a mono-repo? Aren't all dependencies supposed to be in the repo, even the external ones? All these things do not make sense to me. Why go for a radically different approach (mono-repo) and then complain that tools do not work with this approach? I always assumed that mono-repos used radically different tools. I am curious about these things. There is a bit of work in progress that could potentially help with these kinds of project structure, maybe it is of interest to you: |
To address your comments on the mono-repo: There are significant advantages for a mono-repo when it comes to closed-source development of microservice-based projects. When it comes to development you are able to literally change files in the dependent libraries on the fly without having to first build/push to a private repo. By having all of your components in one repo it promotes quicker code sharing as well removes the need for any sort of "manual" versioning of the various components. As a result, you simply deal with one repo and merging the branches from it. A refactor, or split of services would not require removing or adding other repos, but instead be completely isolated to the branch. When it comes to deployment you can basically use git describe tags to get a unique version identifier for all the artifacts in that specific commit. Meaning that you guarantee all of the individual services are functionally aligned, there is no thinking about "What version does service X have to be with service Y?". Working with private repos is pretty painful when it comes to making CI/CD pieplines as well as docker images. When it comes to python/poetry, ultimately what we are looking for is a way to do development locally (IE install into a virtual environment for a given service) in a way that local paths are used, but for an install (into something like a Docker container) we want to be creating actual packages for each component in the dependency tree and installing them into site-packages. NPM for example doesn't allow local development via linked resources, but instead when you install a local dependency it copies it (and all other dependencies) into your source tree. That makes it easy for deployment, but for development you need to rerun the install step after making any changes to the local dependency services or libraries. My general opinion is that the whole use of the "develop" flag within the actual pyproject.toml file is counter-productive. If anything we should have a flag passed when running "poetry install" like "--local-dev" or similar. This way without the flag all packages would be copied into the virtual environment correctly with all local path dependencies resolved to just package names (similar to npm install or what we need for docker file packaging), but with the flag the package links (egg-link I think) would be installed into the virtual environment instead. Ultimately, what we are trying to do is stack a hierarchy of dependencies and either let them be there for development, or flatten them into standalone packages based on what we actually need. |
@sinoroc: Thanks for the pointers, I looked into them earlier, but I wasn't sure they are fully overlapping. @TrevorPace: We have exactly the same use case and use the mono repo (or rather mini mono repo, so splitting an application into smaller packages but in one repository) for the same advantage. To not further fuel the discussion: I think the value of such a setup depends heavily on your application/library and even more your infrastructure for CI/CD, deployment and even your organisation.
For CI/CD we build the packages locally and install the packages using pip with a requirements.txt built from poetry. So we don't need the round trip via a (private) pypi just for tests. But for development we would like to have the nested path dependencies working (right now we track all transitive dependencies manually) so that we can do changes in multiple packages and have the changes instantly running or test them. I think what you proposed would be a perfect fit for our use case as well. |
Alright, I'll try to forget the mono-repo bit (I am still not sure what it implies or not). So your issue is as described in #2877 (comment), isn't it? Somewhere poetry seems to lose track of what the relative paths are relative to. As a side note: maybe a flat hierarchy in the project's directory structure would work better (i.e. all apps and libs in the same directory instead of an apps directory and a libs directory). The rest I don't understand. Particularly these bits:
Poetry can't just magically guess what dependencies should be installed as editable. It has to be specified for each dependency explicitly. I can not really imagine how a global flag would work. Or do you have only local path dependencies and strictly no other 3rd party dependencies (from PyPI for example)? |
Resurrecting this discussion as this bug is still current, and I suspect it may be related to #4051, and @sinoroc expresses some preference here which I think needs to be repudiated:
... etc. Whether you like path dependencies or not is totally irrelevant to the bug in question. Poetry is supposed to be able to resolve dependency hierarchies recursively, and it should do so correctly for each kind of dependency it is designed to support. In this instance, Poetry should be able to handle these kinds of dependencies - it's central to the design goals of the tool. Breaking when You don't understand path dependencies or don't think they're "nice", or whatever. That doesn't mean that this isn't a bug. Spending paragraphs of italicized text explaining that you don't understand something is distracting. You explain that you don't even understand the "monorepo" use case - if you don't understand the use case, and don't understand the failure, then don't make the issue difficult to approach by building walls of text on top of each other. This bug affects Poetry 1.1.6. |
Just as a note, while the workaround worked until |
Just to add a different perspective on the path dependency issue.. I think a large use case, is those using a mono repo for a large system.. at least that is our use case and while I feel like the debate about mono vs separate repos is a war in and of itself. it doesn't seem to make sense to me for a general purpose tool like poetry to be opinionated about such things. |
I will also put out the following horrible work around we have in our make files.. this basically 'fixes' the lock file to have the correct directory references... you have to do that for every second order dependency ... obviously libs/constants is particular to our repo.. |
Using |
I am using Poetry (version 1.3.2) and the issue definitely appears with |
OP provided a repro for the issue they were reporting, at https://github.com/TrevorPace/poetry-localdep-issue. And that repro works just fine with poetry 1.3.2
This issue should be closed: the problem that it reported has been fixed. If others of you in this thread have different problems, please raise new issues. |
Resolved via #4245 |
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
-vvv
option).Issue
I am attempting to install a poetry based application with a monorepo-like hierarchy. For simplicity sake the structure looks something like:
The apps/app1/pyproject.toml is completely empty other than a reference to one of the libraries:
pkg1 = { path = "../../lib/pkg1", develop = false }
pkg1 has then a similar local dependency on pkg2:
pkg2 = { path = "../pkg2", develop = false }
An example repo of this is here: https://github.com/TrevorPace/poetry-localdep-issue
When attempting to run
poetry install
inapps/app1
to create the initial poetry.lock file I get the following issues:So, it appears that we are actually able to correctly traverse the package tree when determining what to build, but when pkg1 actually is being built it's generating something with a local reference in it still. My initial thought is that it is somehow related to #3148.
The text was updated successfully, but these errors were encountered: