Poetry wants to reinstall every system package to the venv when using virtualenvs.options.system-site-packages

[awoimbee]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Poetry version: 1.2.0b3

Issue

When I set virtualenvs.options.system-site-packages to true, my program running inside the poetry venv has access to the system deps (good). But when I run poetry install, poetry wants to re-download and re-install everything (bad).

Either I use virtualenvs.options.system-site-packages true and poetry should use the system deps (unless not possible, eg not if version mismatch). Or I use virtualenvs.options.system-site-packages false and system deps are never used.

Currently If I want to use a dependency from the system site-packages I have to remove it from pyproject.toml and poetry.lock before poetry install.

To reproduce

docker run -it --rm python:3.10 bash

pip install poetry==1.2.0b3
poetry config virtualenvs.options.system-site-packages true
cat <<EOT > pyproject.toml
[tool.poetry]
name = "gros-zozo"
version = "0.0.0"
description = "saussice"
authors = ["superman"]

[tool.poetry.dependencies]
python = "^3.8"
six = "*"

[build-system]
requires = ["poetry>=1.1"]
build-backend = "poetry.masonry.api"
EOT
poetry install --dry-run

output is:

Creating virtualenv gros-zozo-il7asoJj-py3.10 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 0 installs, 1 update, 0 removals

  • Updating six (1.16.0 /usr/local/lib/python3.10/site-packages -> 1.16.0)

Here I use --dry-run for the example, but without --dry-run poetry downloads the packages then installs then to the venv (but I already have these packages ! I don't want to re-download everything !).

see #1393 (comment)

[awoimbee]

Using poetry 1.2.0rc1 (EDIT: 1.2.0)

1. I tracked down where poetry triggers an update:

   1. There is a weird condition in poetry about package source here
   2. poetry-core, is_same_package_as calls is_same_source_as.

2. System deps are treated as source_type="directory", this seems like an issue to me as a random dir (toto = { path = "../toto", develop = true } is treated the same as a system dep).

   1. System deps are treated as source_type="directory" because of is_standard_package = env.is_path_relative_to_lib(path). As a workaround I add if self.is_venv(): return self.parent_env.is_path_relative_to_lib(path) here.
   2. But then I have an issue with poetry looking in site-packages & ubuntu using dist-packages.
      [REMOVED]
      EDIT: Fixed with SETUPTOOLS_USE_DISTUTILS=stdlib
      EDIT2: Nope, poetry inside the venv (poetry run poetry) still tries to use site-packages

EDIT:
Hacks to make it work:

# hack 1: treat everything in /usr/ as a "standard_package"
sed -i -E 's|( *).*env.is_path_relative_to_lib\(path\)|\1s_path = str(path)\n\0 or s_path.startswith("/usr/")|' /usr/local/lib/python3*/dist-packages/poetry/repositories/installed_repository.py
# hack 1.1: same idea, another way of doing it
perl -0777 -pi -e 's|( +)(def is_path_relative_to_lib\([\S\s]+?)( +return False)|\1\2\1    if self.is_venv(): return Path("/usr") in path.parents\n\3|s' /usr/local/lib/python3*/*-packages/poetry/utils/env.py

# hack 2: edit is_path_relative_to_lib to check the system lib too
# -> doesn't always work, see EDIT2 above
perl -0777 -pi -e 's|( +)(def is_path_relative_to_lib\([\S\s]+?)( +return False)|\1\2\1    if self.is_venv(): return self.parent_env.is_path_relative_to_lib(path)\n\3|s' /usr/local/lib/python3*/*-packages/poetry/utils/env.py

[ashleysommer]

Having this same problem, using the v1.2.0 release.

In my docker container, I install some system packages, then in my poetry project, I turn on system-site-packages option, and pin the package version in my poetry.toml (and in poetry.lock) to the exact same package version that is already installed.

But when running poetry install --no-root, poetry goes ahead and re-installs the same packages, but from PIP. This fails, because these packages cannot be compiled in my docker container. (and defeats the purpose, because the compiled package is already installed by the system).

[neersighted]

PTAL @radoering -- looks like more fallout from the work on dependency origins (and could be intended behavior that we need to document).

[radoering]

I wonder if system-site-packages has ever worked that way. It seems like the code in question is older than the option.

I'd say this else is too much of a generalization:

poetry/src/poetry/repositories/installed_repository.py

Lines 161 to 164 in cf3dcad

	else:
	# If not, it's a path dependency
	source_type = "directory"
	source_url = str(path.parent)

[marekro]

Yes, looks like I did not find this issue before filing #7234 - apologies. Maybe I can make up for it with another workaround:
I use the following sequence to avoid running into the bug, but keep the goodness of poetry's dependency resolution:

With a poetry.toml:

[virtualenvs.options]
system-site-packages = true

Run this:

poetry use env python3.10
poetry lock
poetry export > requirements.txt
poetry run pip3 install -r requirements.txt

It seems that pip is smart enough to "see" the already available packages in the system-wide Python site-packages, and skips the redundant build. I would be happy if the poetry installation manager would do the same.

[awoimbee]

@neersighted any comment as to why this is not planned ?
You might as well just remove virtualenvs.options.system-site-packages at this point.

[neersighted]

Looks like I got a wire crossed and closed the wrong issue (#7234 is a duplicate). That being said, system-site-packages is not Poetry's option and that should be kept in mind -- it is functionality of the virtualenv package and PEP being exposed for power users.

[marekro]

Clearly, system-site-packages is a venv option, not a native option of poetry. That said, poetry uses and supports venv. So when a (power) user wants to make venv use system-site-packages, then poetry should respect this choice - and be happy with the packages seen in the venv - regardless whether found in the venv itself or in the underlying Python’s system-site-packages. Needless to say that poetry should install a package version in venv in case the version does not match. Thanks for reopening this ticket and I hope this can be fixed soon.

[gingershaped]

Is this ever going to be fixed? It's been a thorn in my side for using Poetry on a Raspberry Pi, where I often install packages system-wide to avoid needless hours of building.

[dimbleby]

Like all bugs: this will not be fixed before someone who cares about it shows up and fixes it. Is that you?

[niniack]

Hi @abompard, I'm not quite certain that this issue is fixed by #8359. I am using the master branch on poetry and despite having a package installed in my container, I still run into the same issue where poetry reinstalls existing packages, like this:

root@a9ea362e494d:/# poetry config virtualenvs.options.system-site-packages true
root@a9ea362e494d:/# cat pyproject.toml 
[tool.poetry]
name = "myproject"
version = "0.0.0"
description = "myproject"
authors = ["me"]
license = "MIT"
readme = "README.md"

[build-system]
requires = ["poetry-core>=1.5.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = ">=3.8.1,<3.11"
toml = { version = "*"}
requests = { version = "*"}
spython = { version = "*"}
typer = {version = "*", extras = ["all"]}

[tool.poetry.group.torch]
optional = true

[tool.poetry.group.torch.dependencies]
torch = {version = ">=2.1.0", source = "torch", allow-prereleases = true}
torchvision = {version = "*", source = "torch", allow-prereleases = true}
torchaudio = {version = "*", source = "torch", allow-prereleases = true}

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu118"
root@a9ea362e494d:/# poetry install --no-root --only torch --dry-run
Installing dependencies from lock file

Package operations: 0 installs, 19 updates, 0 removals

  - Updating filelock (3.9.0 /usr/local/lib/python3.10/dist-packages -> 3.13.1)
  - Updating markupsafe (2.1.3 /usr/local/lib/python3.10/dist-packages -> 2.1.3)
  - Updating mpmath (1.3.0 /usr/local/lib/python3.10/dist-packages -> 1.3.0)
  - Updating certifi (2022.12.7 /usr/local/lib/python3.10/dist-packages -> 2023.11.17)
  - Updating charset-normalizer (2.1.1 /usr/local/lib/python3.10/dist-packages -> 3.3.2)
  - Updating fsspec (2023.4.0 /usr/local/lib/python3.10/dist-packages -> 2023.10.0)
  - Updating idna (3.4 /usr/local/lib/python3.10/dist-packages -> 3.4)
  - Updating jinja2 (3.1.2 /usr/local/lib/python3.10/dist-packages -> 3.1.2)
  - Updating sympy (1.12 /usr/local/lib/python3.10/dist-packages -> 1.12)
  - Updating triton (2.1.0 /usr/local/lib/python3.10/dist-packages -> 2.1.0)
  - Updating typing-extensions (4.4.0 /usr/local/lib/python3.10/dist-packages -> 4.4.0)
  - Updating urllib3 (1.26.13 /usr/local/lib/python3.10/dist-packages -> 2.1.0)
  - Updating networkx (3.0 /usr/local/lib/python3.10/dist-packages -> 3.1)
  - Updating numpy (1.24.1 /usr/local/lib/python3.10/dist-packages -> 1.24.4)
  - Updating pillow (9.3.0 /usr/local/lib/python3.10/dist-packages -> 10.1.0)
  - Updating torch (2.1.1+cu118 /usr/local/lib/python3.10/dist-packages -> 2.1.1+cu118)
  - Updating requests (2.28.1 /usr/local/lib/python3.10/dist-packages -> 2.31.0)
  - Updating torchaudio (2.1.1+cu118 /usr/local/lib/python3.10/dist-packages -> 2.1.1+cu118)
  - Updating torchvision (0.16.1+cu118 /usr/local/lib/python3.10/dist-packages -> 0.16.1+cu118)
root@a9ea362e494d:/# poetry --version
Poetry (version 1.8.0.dev0)

Now, I understand that my issue could stem from something different because, well, Pytorch. However, it also happens for Pytorch's dependencies, which is why I wanted to raise this here.

If I am misunderstanding something, I would greatly appreciate being pointed in the right direction. Thank you.

[awoimbee]

It works for me (Installing torch (2.0.0+cu117): Skipped for the following reason: Already installed) (tested poetry 1.7 and master 31540a802e7779ab13bbfc39c8eefd6402a59605).

# find / -name 'torch' 2>/dev/null
/usr/local/lib/python3.10/dist-packages/torch

My "source" is setup as explicit and torch is not in a dependency group. These and the fact that your group is optional might be triggering a bug ?

[niniack]

Thanks for testing! Would you mind sharing your pyproject.toml for the pytorch case?

I've boiled it down as much as I could and I still cannot get it to work:

My source is explicit and no longer part of a dep group. I also set forced it to 2.0.0, in case it was something about how pip installs that version, but no luck.

TOML file

[tool.poetry]
name = "poetry-torch"
version = "0.0.0"
description = "Testing poetry and torch"
authors = ["Me"]

[build-system]
requires = ["poetry>=1.1"]
build-backend = "poetry.masonry.api"

[tool.poetry.dependencies]
python = ">=3.8.1,<3.11"
torch = {version = "=2.0.0", source = "torch"}

[[tool.poetry.source]]
name = "torch"
priority = "explicit" 
url = "https://download.pytorch.org/whl/cu117"

Dockerfile

# Start from the NVIDIA PyTorch image
FROM nvidia/cuda:11.7.1-runtime-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive

# Install required tools and libraries, and clean up to reduce layer size
RUN apt-get update \
    && apt-get install -y software-properties-common \
    && apt-get install -y libstdc++6 libgl1 wget curl git bzip2 python3 python3-pip\
    && apt-get remove --purge -y software-properties-common \
    && apt-get autoremove -y \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Symlink python 
RUN rm -f /usr/bin/python && ln -s /usr/bin/python3 /usr/bin/python

ENV POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    POETRY_CACHE_DIR=/tmp/poetry_cache \
    POETRY_HOME=/opt/poetry \
    POETRY_VIRTUALENVS_OPTIONS_SYSTEM_SITE_PACKAGES=1

RUN --mount=type=cache,target=~/.cache/pip \
    pip3 install torch==2.0.0 --index-url https://download.pytorch.org/whl/cu117

# Install poetry with pinned version
RUN curl -sSL https://install.python-poetry.org | POETRY_VERSION=1.7.1 python3 -
ENV PATH="$PATH:$POETRY_HOME/bin"

# Set site packages to be available
# The env vars should have handled this 
# But that is currently broken for OPTIONS
RUN poetry config virtualenvs.options.system-site-packages true

COPY poetry.lock pyproject.toml ./

Testing it out:

(base) alodie@alodie:~/git/poetry-torch$ docker run -it poetry-torch /bin/bash

==========
== CUDA ==
==========

CUDA Version 11.7.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

root@43cc2d141b0d:/# poetry --version
Poetry (version 1.7.1)
root@43cc2d141b0d:/# find / -name 'torch' 2>/dev/null
/usr/local/lib/python3.10/dist-packages/torch
/usr/local/lib/python3.10/dist-packages/torch/include/torch
/usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include/torch
root@43cc2d141b0d:/# poetry install --dry-run
Creating virtualenv poetry-torch in /.venv
Installing dependencies from lock file

Package operations: 0 installs, 11 updates, 0 removals

  • Updating cmake (3.25.0 /usr/local/lib/python3.10/dist-packages -> 3.27.7)
  • Updating filelock (3.9.0 /usr/local/lib/python3.10/dist-packages -> 3.13.1)
  • Updating lit (15.0.7 /usr/local/lib/python3.10/dist-packages -> 17.0.5)
  • Updating markupsafe (2.1.3 /usr/local/lib/python3.10/dist-packages -> 2.1.3)
  • Updating mpmath (1.3.0 /usr/local/lib/python3.10/dist-packages -> 1.3.0)
  • Updating jinja2 (3.1.2 /usr/local/lib/python3.10/dist-packages -> 3.1.2)
  • Updating networkx (3.0 /usr/local/lib/python3.10/dist-packages -> 3.1)
  • Updating triton (2.0.0 /usr/local/lib/python3.10/dist-packages -> 2.0.0)
  • Updating typing-extensions (4.4.0 /usr/local/lib/python3.10/dist-packages -> 4.8.0)
  • Updating sympy (1.12 /usr/local/lib/python3.10/dist-packages -> 1.12)
  • Updating torch (2.0.0+cu117 /usr/local/lib/python3.10/dist-packages -> 2.0.0+cu117)

Installing the current project: poetry-torch (0.0.0)

[UFO-101]

I also have a simple example of this not working in a Docker build:

https://github.com/UFO-101/auto-circuit/blob/main/Dockerfile

# Based on https://github.com/AlignmentResearch/flamingo/blob/main/examples/devbox/Dockerfile
# and https://nanmu.me/en/posts/2023/quick-dockerfile-for-python-poetry-projects/
FROM pytorch/pytorch as base

ARG UID=10000
ARG GID=101
ARG USERNAME=dev

ENV PYTHONUNBUFFERED=1 \
    # pip
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    \
    # Poetry
    # https://python-poetry.org/docs/configuration/#using-environment-variables
    POETRY_VERSION=1.7.1 \
    # make poetry install to this location
    POETRY_HOME="/opt/poetry" \
    # do not ask any interactive question
    POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_PATH=/home \
    POETRY_VIRTUALENVS_IN_PROJECT=false \
    POETRY_VIRTUALENVS_OPTIONS_SYSTEM_SITE_PACKAGES=true

# prepend poetry and venv to path
ENV PATH="$POETRY_HOME/bin:$PATH"

# Update the package list, install sudo, create a non-root user, and grant password-less sudo permissions
RUN apt update && \
    apt install -y sudo && \
    addgroup --gid $GID ${USERNAME} && \
    adduser --uid $UID --gid $GID --disabled-password --gecos "" ${USERNAME} && \
    usermod -aG sudo ${USERNAME} && \
    echo "${USERNAME} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

# Install some useful packages
RUN apt update && \
    apt install -y rsync git vim

# Install Poetry
RUN apt install -y curl && \
    curl -sSL https://install.python-poetry.org | python3 -

RUN chmod 777 /home

# Set the non-root user as the default user
USER ${USERNAME}
WORKDIR /home/${USERNAME}/auto-circuit

# Copying the Python project files
COPY --chown=${USERNAME}:${USERNAME} . /home/${USERNAME}/auto-circuit

# Install dependencies
RUN poetry install --with dev

WORKDIR /home
RUN rm -rf /home/${USERNAME}

[birdcolour]

Have been running up against this problem myself, and the most useful workaround I can see for this (or perhaps it is the intended way?) is to define the dependencies that your container provides as an optional group or set of extras, and opt out of it on container builds. Since updating any of these will necessitate using a new base container version, you'll definitely want to pin these to an exact version anyway.

For example, this container provides ultralytics and all of the finicky ML backends/nvidia gubbins already installed in dist-packages, so we add it to a separate container dependency group which we don't use inside the container:

pyproject.toml

[tool.poetry]
name = "poetry-ml-containers"
version = "0.1.0"
description = "Install stuff with poetry safely in docker containers using system-site-packages"
authors = ["me"]

[tool.poetry.dependencies]
python = "^3.10"

# normal deps go here

[tool.poetry.group.container]
optional = true

[tool.poetry.group.container.dependencies]
# deps that the container provides
ultralytics = {version = "=8.1.18"}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Dockerfile

FROM ultralytics/ultralytics:latest-cpu

RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/usr/local python3 - \
 && poetry config virtualenvs.options.system-site-packages true

COPY pyproject.toml poetry.lock ./

$ poetry lock

Use the group outside the container to get all dependencies on your local system venv

$ poetry install --dry-run --with container | grep ultralytics
  • Installing ultralytics (8.1.18)

Build

$ docker build -t foo .
$ docker run -it --rm foo bash

Verify that package is already installed correctly

# pip freeze | grep ultralytics
-e git+https://github.com/ultralytics/ultralytics@2d75f72598d5ce80786be61f140673a6d1dd7e57#egg=ultralytics
# python -c "import ultralytics; print(ultralytics.__version__)"
8.1.18

Install without overwriting anything, hooray!

# poetry install --dry-run
Creating virtualenv poetry-ml-containers-6zGKWtXl-py3.11 in /root/.cache/pypoetry/virtualenvs
Installing dependencies from lock file

Installing the current project: poetry-ml-containers (0.1.0)

What we're trying to avoid (and what appears to still be buggy behaviour?)

# poetry install --dry-run --with container | grep ultralytics
  - Updating ultralytics (8.1.18 /usr/src/ultralytics -> 8.1.18)

It's a fairly simple and contrived example, so in reality you may need to pin a few more things from your container site-packages into the container extras/group, but a bit of manual solving is fairly inevitable with this kind of "overlay" dependency solution. I'll leave the horrible bash one-liner to pipe the output of pip freeze within a container into a dependency group as an exercise to the reader/plugin developer :)

[ashleysommer]

@birdcolour Thanks for sharing that, and it does help with my use-case too.
In my case, we were using poetry since before the new dependency-groups come in, so we still mark system-provided dependencies as optional dependencies in the "old" way, like:

[tool.poetry.dependencies]
foo = {version=bar, optional=True}

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.