Poetry removes it's own dependencies with install --no-dev when it manages the environment it is installed in

[gbdlin]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.
  Well... This issue is mentioned by @sontek in Poetry install removes packages that are necessary #3139 (comment)_ but the response below advises to create a new issue, so here I am.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Docker image based on python:3.9.1

• Poetry version: 1.1.6

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/gbdlin/311dda3cd34672370cd97bf07f306a61

Issue

When you specify a package in the dev-dependencies that is either a dependency of the poetry itself or it depends on one of the poetry dependencies, and you've installed poetry inside an environment that poetry is supposed to manage, poetry will uninstall this mentioned dependency after invoking poetry install --no-dev, making itself unusable in the future.

I'm aware that it is advised to install poetry outside of the environment it should manage, but this is very often seen inside docker images (such example provided in the gist linked above) to save on the resources.

Example output presenting mentioned issue (without -vvv ands):

$ pip install poetry
Requirement already satisfied: pip in /usr/local/lib/python3.9/site-packages (21.0.1)
Collecting poetry
  Downloading poetry-1.1.6-py2.py3-none-any.whl (172 kB)
[ ... 70 irrelevant output lines truncated ... ]
Collecting appdirs<2,>=1.4.3
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Installing collected packages: [ ... ] appdirs, [ ... ] poetry
Successfully installed [ ... ] appdirs-1.4.4 [ ... ] poetry-1.1.6 [ ... ]

$ poetry config virtualenvs.create false && poetry install --no-dev

Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing appdirs (1.4.4)
Removing intermediate container af920367a74d

As you can see, appdirs, which is required by the poetry, was uninstalled by poetry. Full log, with the -vvv and the attempt to run poetry again, available here

[abn]

This is expected, since you are asking poetry to manage the system site, the same that you have installed poetry into. I suspect here the case is because appdirs is in your project's dependency tree, and you have told poetry to make it so that none of the project dev dependencies are installed. Well, poetry does so, dutifully.

Honestly, creating a virtual environment for the project does not save any meaningful amount of resources and it causes more problems than it solves.

And with upcoming release of poetry you can even create virtual envs without any packages (incl. pip, setuptools).

Alternatively, you can also just do pip install /path/to/project. The caveat here is that you loose any locked versions. But hey, you avoid adding poetry or creating venvs. ;)

[gbdlin]

If this is expected, it's for sure not documented properly. Both in the documentation and when just using poetry install --help, the --no-dev option is described as Do not install the development dependencies., not Get rid of the development dependencies. There is also a 2nd command line options --remove-untracked, from which I would expect to remove some dependencies.

But even if the --no-dev's expected behavior is to remove all dev dependencies, poetry should do one of those:

• do an exception for its own dependencies if it's being run from the same virtualenv,
• don't remove anything implicitly when it's being run from the same virtualenv,
• provide additional command line option that allows skipping the removal of dev dependencies.

[AlexandreDecan]

I ran into a similar issue on Travis where poetry removes virtualenv even if --no-dev is not used (actually, it is "removed" when I execute poetry update --lock).

The issue only arises with Python 3.6 and PyPy (see https://travis-ci.com/github/AlexandreDecan/portion/builds/223762003) but I think it's a consequence of pre-commit relying on virtualenv but I don't know why pre-commit is skipped for Python 3.6.12 since my dependency on it declares pre-commit = {version = "^2.12.1", python = "^3.6.1"} (hence, ^3.6.1 should be satisfied the same way than 3.7, or 3.8, or...). Anyway, that's a different issue ;-)

I agree with @gbdlin that ideally, something should be done about this (esp. since I also did not expect poetry to remove packages unless --remove--untracked is provided).

[lainisourgod]

@gbdlin I agree that poetry's behaviour is invalid but for now you can avoid the problem by installing poetry in isolated environment: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -

[blueyed]

Duplicate of #3139? (at least that one is older, and covers removal of virtualenv also)

[gbdlin]

@blueyed

Duplicate of #3139? (at least that one is older, and covers removal of virtualenv also)

This issue is mentioned by @sontek in #3139 (comment)_ but the response below advises to create a new issue, so here I am.

@sontek maybe raise a new big with a bit more information, and maybe an example with logs we could try to reproduce with.

As a note, it is not recommended that use the same environment poetry is installed in as your project's virtual environment.

Originally posted by @abn in #3139 (comment)

[MasterNayru]

I am just hitting this issue myself now.

It seems like the behaviour that I am after is pretty much for poetry to be treated like a runtime dependency of the project, but NOT added as an actual dependency of any packages built and pushed to a PyPi repo. Is there any clean way to get that behaviour?

I get that the curl url | python install method puts poetry in its own venv and side-steps this issue, but given that poetry is distributed through pip, it seems crazy to me to not support the case where poetry and the project being managed are in the same environment. Is there any reliable way for poetry to establish that it isn't isolated in its own environment and put in some form of exception to stop uninstalling its own dependencies?

[pierresouchay]

Yes, we have been hitting this issue as well (we call it the "suicide" behaviour)

Could simply the behavior of --no-dev be changed not not uninstall dev-dependencies? (just ensure that main dependencies are installed)

[mwgamble]

I think a better approach would be to have a poetry clean command or something along those lines, that uninstalls packages that aren't required . This is preferable IMO, rather than tying it to poetry install.

[johncronan]

Could simply the behavior of --no-dev be changed not not uninstall dev-dependencies? (just ensure that main dependencies are installed)

I feel like this is the right answer. --no-dev should mean perform operations without regard to dev packages, instead of remove dev packages from the list of what needs to be in the environment.

(I only found this by accident because of something related, but I too use poetry in a Docker image with virtualenvs.create false.)

edited.. Wait, is it impossible? Maybe I'm trying to make a distinction without a difference.

[finswimmer]

This has change with poetry 1.2.0a2: Poetry wan't remove installed packages unless the --sync option is given.

However, if a dependency will be removed due to a version change e.g. on poetry update the package will be removed of course from the venv.

fin swimmer

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.