Bug: poetry install fails with transitive path dependencies via git

[jpugliesi]

• Poetry version: 1.7.1
• Python version: 3.10.12
• OS version and name: macOS 13.4
• pyproject.toml: https://github.com/jpugliesi/poetry-git-local-dep-bug

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Poetry fails to correctly install when it has a transitive path dependency, via a git subdirectory dependency.

For example, see https://github.com/jpugliesi/poetry-git-local-dep-bug where the example package's pyproject.toml contains a git dependency namespace-cli:

[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["John Pugliesi <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"
namespace-cli = {git = "https://github.com/adriangb/python-monorepo.git", subdirectory = "poetry/workspaces/cli"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The namespace-cli package then has a relative path dependency on another project in the same repo as namespace-cli: (namespace-lib):

[tool.poetry]
name = "namespace-cli"
version = "0.0.0"
description = ""
authors = []
packages = [{include = "namespace", from = "src"}]

[tool.poetry.dependencies]
python = "^3.10"
rich = "^12.6.0"
namespace-lib = { path = "../lib"} # relative path dependency

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Installing the original example package will produce an error:

git clone https://github.com/jpugliesi/poetry-git-local-dep-bug && cd poetry-git-local-dep-bug
pipx install poetry==1.7.1
poetry install

Error:

➜  poetry install
Configuration file exists at /Users/x/Library/Preferences/pypoetry, reusing this directory.

Consider moving TOML configuration files to /Users/x/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.
The currently activated Python version 3.9.16 is not supported by the project (~3.10).
Trying to find and use a compatible version.
Using python3.10 (3.10.12)
Creating virtualenv example in /private/tmp/poetry-git-local-dep-bug/.venv
Installing dependencies from lock file
Path /private/tmp/lib for namespace-lib does not exist
Path /private/tmp/poetry-git-local-dep-bug/.venv/src/python-monorepo/poetry/workspaces/lib for namespace-lib does not exist

Path /private/tmp/lib for namespace-lib does not exist

It seems poetry is resolving the path of the transitive namespace-lib relative to the root directory of the poetry-managed virtualenv, whereas it should be resolving the path relative to the subdirectory of the git dependency.

[dimbleby]

layered path dependencies are pretty much always a bad idea - if you search this issue tracker you will find plenty of examples.

In this case you can see that what poetry has put into its lockfile for namespace-lib is this:

  [package.source]
  type = "directory"
  url = ".venv/src/python-monorepo/poetry/workspaces/lib"

which probably was true at time of locking. But if it's not true at time of install - well, too bad.

Similarly look at the dependency in the lockfile from namespace-cli on namespace-lib:

  [package.dependencies]
  namespace-lib = {path = "../lib"}

well of course, what else should it say? but in the new context this just doesn't mean anything useful.

(This sort of thing is the reason that you cannot upload wheels containing path dependencies to pypi: if you use relative paths then the metadata becomes meaningless, if you use absolute paths then it only works on your machine)

This is doomed. Find some other way!

[JonathanRayner]

layered path dependencies are pretty much always a bad idea - if you search this issue tracker you will find plenty of examples.

Perhaps the point is this is less of a bug and more of a feature request. It's pretty common to work in monorepos with path dependencies, then find you need to install something via git in some other repo.

It's true that someone could find another way, but it doesn't seem that unreasonable to me to feature request that when adding a git dependency poetry could detect that there are path dependencies and resolve them (or raise a helpful error warning that this is impossible).

In my opinion, the expected behavior when adding a git dependency that itself contains path dependencies would be to resolve them as multiple git dependencies with different subdirs from the same repo (which would require solving #6958 ). Maybe there are flaws to this idea, I didn't think too carefully about it yet.

[tbreslein]

I just found this because I ran into this issue. Funnily enough though, running poetry add <URL_WITH_SUBDIR> works fine.
So in the example this should work:

poetry add "https://github.com/adriangb/python-monorepo.git#subdirectory=poetry/workspaces/cli"

This does add the entry to the pyproject.toml, but if I were to do a fresh poetry install in this project, it runs into the issue with transitive dependencies again. So there apparently is a way to make it work, and it is anything but "doomed" 😄

[norbusan]

Same same here, this is a more than common problem I have seen and is a PITB to work around.

The option that works is instead of using path component to use github again, but that brings one into a Goedelian hell of self-referential updates, because poetry.lock needs to refer to the same git commit hash it tries to change. Add a chain of dependency and the ONLY option is hand-editing poetry.lock file.