Path dependencies with path dependencies

[leahein]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Linux mint 19.3
• Poetry version: 1.0.10
• pyproject.toml file: https://github.com/leahein/poetry-path-deps-within-path-deps

Issue

Path dependencies that rely on path dependencies don't seem to install correctly.

Example file structure:

my-app/
  pyproject.toml
  lib/
    my-lib-1/
      pyproject.toml
      lib/
        my-lib-2/
          pyproject.toml

Steps to reproduce:

> cd my-app/lib/my-lib-1
> poetry add lib/my-lib-2
# Installs successfully
> cd ../..
> poetry add lib/my-lib-1
> [EnvCommandError]
Command ['/home/leah/Development/scripts/my-app/.venv/bin/pip', 'install', '--no-deps', '-U', '-e', '/home/leah/Development/scripts/my-app/lib/my-lib-1/lib/my-lib-1/lib/my-lib-2'] errored with the following return code 1, and output: 
/home/leah/Development/scripts/my-app/lib/my-lib-1/lib/my-lib-1/lib/my-lib-2 should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

The path for my-lib-2 is lib/my-lib-1/lib/my-lib-1/lib/my-lib-2, when it should be /lib/my-lib-1/lib/my-lib-2.

For the full output:

[EnvCommandError]
Command ['/home/leah/Development/scripts/my-app/.venv/bin/pip', 'install', '--no-deps', '-U', '-e', '/home/leah/Development/scripts/my-app/lib/my-lib-1/lib/my-lib-1/lib/my-lib-2'] errored with the following return code 1, and output: 
/home/leah/Development/scripts/my-app/lib/my-lib-1/lib/my-lib-1/lib/my-lib-2 should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
You are using pip version 19.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


Traceback (most recent call last):
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/_vendor/py3.7/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/_vendor/py3.7/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/console/commands/add.py", line 156, in handle
    status = installer.run()
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/installation/installer.py", line 286, in _do_install
    self._execute(op)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/installation/installer.py", line 302, in _execute
    getattr(self, '_execute_{}'.format(method))(operation)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/installation/installer.py", line 327, in _execute_install
    self._installer.install(operation.package)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/installation/pip_installer.py", line 31, in install
    self.install_directory(package)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/installation/pip_installer.py", line 230, in install_directory
    return self.run(*args)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/installation/pip_installer.py", line 127, in run
    return self._env.run_pip(*args, **kwargs)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/utils/env.py", line 884, in run_pip
    return self._run(cmd, **kwargs)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/utils/env.py", line 1112, in _run
    return super(VirtualEnv, self)._run(cmd, **kwargs)
  File "/home/leah/.asdf/installs/poetry/1.0.10/lib/poetry/utils/env.py", line 916, in _run

[abn]

@leahein appreciate you raising this. The root cause here is that there is some inconsistencies in internal assumptions made regarding the locked source path.

From the root project:

[package.source]
reference = ""
type = "directory"
url = "lib/my-lib-1/lib/my-lib-2"

In this case when the package is attempted to be installed, the path constructed becomes <root project>/<first level dependency>/<second level dependency path>. Which unfortunaly is derrived from the lock file of the root project Since when operations are performed we do not hold a notion of what the parent dependency was, we cannot detect that this is relative to another project dependency's path. The specific issue here occurs at:

poetry/poetry/installation/executor.py

Lines 474 to 477 in 17b165b

	if package.root_dir:
	req = os.path.join(str(package.root_dir), package.source_url)
	else:
	req = os.path.realpath(package.source_url)

All that said, it should also be acknowledged that this might not be a great idea to support either, We cannot expect relative paths to the portable. This means a project locked on one user's machine might not work on another. The case you have described above is probably an exception, but something we would like to handle a bit better as being discussed at #2270.

[ghost]

I observe the same problem, but also for non-nested projects. I have a mono repository setup where dependencies between packages in the repository are defined with relative paths. If I understand your answer correctly, this has nothing to do in particular with nesting packages, but with the way the path is constructed.

In my setup, I have a structure like this:

C/
 pyproject.toml
A/
 pyproject.toml
foo/
 B/
  pyproject.toml
 ...

The dependency chain is now A -> B -> C and the relative paths are ../foo/B in A/pyproject.toml and ../../C in foo/B/pyproject.toml. I debugged this a bit and also came to the code you quoted above. There, I found that it fails to install C and the source_url for C is the same as the lock file in A (the package I try to install), i.e. ../C so relative to A whereas the root_dir of C is foo/B which together leads to an non existing path foo/B/../C.

I was able to get it working by mutating all packages in the solver here:

poetry/poetry/puzzle/solver.py

Lines 229 to 233 in dfa5c0f

	# Return the packages in their original order with associated depths
	final_packages = packages
	depths = [results[package] for package in packages]
	return final_packages, depths

And setting

for pkg in packages:
    pkg.root_dir = self.package.root_dir

thereby setting the root path of every package to the root directory of the package to be installed (which makes sense to me as all the relative root paths seem also to be relative to the package installed). But I am not sure what this could break (and especially the solver is a bit to complex to easily make a change).

To me this seems independent of the problem of managing lock files and relative paths on multiple machines (i.e. the above mentioned setup should work, but it also does not solve the problem with multiple machines).

If this is a different bug, I can open a new one if you want.

[finswimmer]

This seems to be fixed in the meantime.