poetry sets incorrect package source type when multiple site-package directories are present

[abn]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Issue

TLDR; installed package detection breaks when multiple site-specific directories exist.

In certain scenarios, there exists multiple site-specific package directories. The most
common of this is due to the distinction between purelib and platlib paths
(see sysconfig#Installation Paths).

Another example is the python:2.7 image.

docker run --rm -it python:2.7 python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python2.7/site-packages', '/usr/local/lib/site-python']

Different distros have their own view of site-specific file directories. And I suspect, different python installers do too.

The issue can be reproduced using the snippet provided at the end. Here, in an environment with virtualenv >= 20.0.0, tox creates the environment such that lib64 directory is not linked to lib.

Note that;

• we use a Red Hat distro (Fedora) as this is one where the platlib directory differs from the purelib directory.
• if the virtual environment was created using the venv module or virtualenv < 20.0.0, lib64 will be a symlink to lib.
• the change in behaviour for virtualenv is by design (lib64 directory is no longer symlinked to lib? pypa/virtualenv#1751)

Once executed you will notice that the first tox run executes as expected.

<snipped>
Package operations: 13 installs, 0 updates, 0 removals, 1 skipped
<snipped>

However, the second run will update all packages. This is not expected.

<snipped>
Package operations: 0 installs, 13 updates, 0 removals, 1 skipped
<snipped>

The cause for this is the differing source types for the installed package and required package.

poetry/poetry/puzzle/solver.py

Lines 95 to 96 in 4715f1d

	elif package.source_type != pkg.source_type:
	operations.append(Update(pkg, package))

The root cause for this is because the package is being detected as a non-standard package since
only the first seen site-package directory is being checked.

poetry/poetry/repositories/installed_repository.py

Lines 45 to 52 in 4715f1d

	is_standard_package = True
	try:
	path.relative_to(env.site_packages)
	except ValueError:
	is_standard_package = False
	if is_standard_package:
	continue

The checking of multiple directories were briefly talked about in a comment in #1786. However, I do not think that was resolved there.

Possible Solutions

In both these cases, Env.site_packages will need to be an iterable.

1. Iterate over all paths in sys.path filtering for site-packages. This will be the simplest solution, however will not account for cases like above where the directory is /usr/local/lib/site-python.
2. Use site.getsitepackages(). This will not work for environments created using virtualenv < 20.0.0 as a bundled version for site.py was being used. Since in these environments lib64 is already being symlinked, we could simply fallback to searching for site-packages assuming that the virtual environments will not be creating dist-packages. We should also bump minimum version for poetry's virtualenv dependency as well.

Not sure if there is a solution without any trade-off.

Reproducer

docker run --rm -i fedora:31 <<SCRIPT_EOF
set -x

dnf -qy install -y pip

pip install --quiet --upgrade poetry tox virtualenv>=20.0.0

install -d poetry-issue-reproducer/poetry_issue_reproducer
pushd poetry-issue-reproducer

touch poetry_issue_reproducer/__init__.py

cat > pyproject.toml <<EOF
[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"

[tool.poetry]
name = "poetry-issue-reproducer"
packages = [
    { include = "poetry_issue_reproducer" }
]
version = "1.0.0-alpha.0"
description = "A reproducer for a poetry issue"
authors = [
    "Issue Reproducer <[email protected]>"
]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]
tox = "^3.13"

[tool.tox]
legacy_tox_ini = """
[tox]
isolated_build = True
envlist = py

[testenv]
whitelist_externals =
    poetry
commands =
    poetry install -vvv

"""
EOF

# first run works as expected
tox

# second run updates packages, this is not expected
tox
SCRIPT_EOF

[kasteph]

#2281 looks related.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.