Poetry install removes packages that are necessary #3139
Comments
|
If the dependencies are managed by poetry (ie. via lock file) and you specify In your scenario, assuming you are pre-loading required dependencies, you might be better of building a wheel of your package and installing it instead. poetry build -f wheel
pip install dist/*.whl |
abn
added
the
kind/question
|
Yes, I did have some preinstalled dependencies that wanted to keep while installing the project dependencies with Poetry. Thank you @abn, I'll try with the wheel. |
|
Hi @abn, one question. I was reading the docs. Is this bevahiour already covered by the flag |
|
Yes and no. The key thing to note here is that the Poetry will not remove, iirc, packages that were never managed by poetry. So if your package |
|
I see. I would be a great idea to document this behavior in the docs. Thanks you @abn |
|
I just hit this and I think it might still be a bug: Poetry depends on:
If I add a dependency that requires them (black, flake8, pre-commit) and run: It removes things it depends on. Which I don't think is the intended result. It really should keep its own dependencies around at a minimum |
|
@sontek maybe raise a new big with a bit more information, and maybe an example with logs we could try to reproduce with. As a note, it is not recommended that use the same environment poetry is installed in as your project's virtual environment. |
|
Copying my comment from another issue:
If this is not supported and poetry thinks this is not how poetry should be used, is there another ideal/preferred way to make this workflow work? |
|
Bumping this. I have a project with a multistage docker that installs package A. In the main package, we specify dependency B. Package B requires A, but it doesn't list it as a dependency (it actually does, but I think they built it incorrectly). Just had a massive headache trying to figure out why package A is suddenly unavailable. There's quite a bit of dependency hell, but the gist of the issue was that tl;dr: The "install" command should only "install" by default. It doesn't "upgrade" by default, why should it "remove"? |
|
@fcruzel I don't think this should have been closed. The wheel is a treatment, but it's not a solution. This is a legit bug / forced feature. |
|
@muppetjones Agree. Try adding any dev dependency to your project that has an indirect dependency on any popular like like This means poetry does not provide a valid alternative to plain old Ideally, poetry should do one of the following:
|
|
Completely agree. All good options. Maybe a fourth option that's the
reverse of (2): Only install but add another flag (though it would get
confusing with the existing "remove" flag).
…On Tue, Mar 30, 2021 at 11:08 AM Owais Lone ***@***.***> wrote:
@muppetjones <https://github.com/muppetjones> Agree. Try adding any dev
dependency to your project that has an indirect dependency on any popular
like like requests. dephell is one example. This means poetry will
requests from your project's venv causing a lot more pain than poetry
install solves in the first place. This forces you to install your dev
dependencies into test projects or re-install test project's dependencies
after every single poetry install --no-dev.
This means poetry does not provide a valid alternative to plain old setup.py
develop command. Imagine setup.py develop either forcing you to install
all your dev depencies or forcefully uninstalling any common dependencies
from the target env. It wouldn't make any sense.
Ideally, poetry should do one of the following:
1. Change behavior of install command to not remove packages and only
install them.
2. Remove by default but allow a --no-remove flag to disable behavior.
3. Offer a new poetry develop command that behaves the same way as python
setup.py develop as in does not install dev deps and never removes any
deps.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3139 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPSOEY4SJMUJZYPBBXRFGDTGHZO5ANCNFSM4SILRKRA>
.
--
This email and any attachments may contain confidential information. If you
believe that you have received it in error, please delete the email and
attachments and then notify Tempus by calling *800.739.4137*.
|
|
+1 for |
|
we see this problem appearing with poetry version 1.1.6 when installing poetry in a venv with +1 for the |
|
Hello, most problem that poetry removes 'appdirs' applications needed to work with poetry itself. If you use dev dependencies which use something like 'appdirs' dependencies (and poetry use this dependencies) and run commands Look: So, if you want to install dev dependencies in the next step of the pipeline for linting or run unit tests, you will crash with an error (so poetry will not work anymore): Probably this can be treated with such a crutch in dockerfile when you put such dependencies in the system :-( |
|
Anyone managed to make it work without hacks in the Dockerfile? Or is there any plan to fix this issue? As other pointed out, if you use By the way, another way to make it work is to install poetry like this: instead of what we most of us probably have: Doing this will allow poetry and its dependencies to be isolated from yours. However, a lot of people don't like or even discourage installing stuff like that (mostly because there's no guarantee that the content of that file won't change). |
|
We are also seeing this problem (or a very similar problem) with version 1.1.6. Specifically, in our case, dependencies in Things seem to work fine with 1.1.5. |
I think this stems from the mindset that Poetry manages its own virtualenv, but when virtualenv creation is disabled, the same behavior remains and this was never fixed when adding the ability to work outside of virtualenvs. |
|
I am running poetry inside a conda virtualenv I am able to add some packages, but I am getting some failures. Then when I try to install a subset of packages, I get cases where poetry will remove packages which make it impossible to run poetry again without uninstalling and reinstalling: Example: |
|
Based on our experience and testing, looks like this might be part of the root issue: #4051 |
|
Isn't the issue here that |
|
At least when not using a virtualenv, it seems wrong that poetry would remove distributions that were already installed since they might be used by other services on the system. For production, binary packages with dependencies on system libraries should be installed by the system package manager, not from pypi. But it is handy to include those deps in a dev or extra section as a convenience to new developers on other platforms. EDIT: actually, based on my testing, poetry doesn't remove optional dependencies if the extra isn't selected, but they are already installed. So a work-around is to use extras instead of dev. |
|
This is really not what I would have expected for this behavior. There should be a way to say "I want to keep what is installed, but also install these packages". Right now making multiplatform container images (that support arm64 and amd64) with poetry is a nightmare that requires building the packages separately, and this issue makes that even worse. |
|
Saw there was a workaround for Docker builds with no virtualenv in which you install poetry via curl. At the moment that workaround is also not working (am attempting to install dependencies with: poetry install --no-interaction --no-ansi). I also tried rolling back to poetry 1.1.5 but that still had the issue. I found the issue all the way down to 1.0.0. At that point I just gave in and installed in a virtualenv on my docker image. |
|
There isn't a workaround as far as I can tell. I've just stopped using poetry because of this, since I need to support multiplatform projects and apparently that's not something this project can currently handle. |
|
Just to add my use case - I have 2 poetry packages that I want installed into a Docker image used at development time:
I'm noticing that Package A installs and then Package B ends up removing Package A's dependencies and Jupyter dependencies. For now I have to make sure the dependencies are exactly the same across both packages and include Jupyter dependencies :( This is a bad workaround and now i'm not sure how to make this work with Poetry... |
|
I have a docker image we use for testing. It has preinstalled numerous packages, such as pylint, black etc. |
|
Another use case for the searchers: poetry is excluding the coverage module required by pytest-cov in docker which results in |
|
I've got |
|
We hit the same issue when specifying With poetry 1.1.13, when running |
|
Poetry shouldn’t import the virtualenv package at all if it doesn’t need to create a virtualenv. It should be an optional dependency. This would help mitigate the main issue in this thread, though it’s obviously not the full solution. |
|
I just got bitten by this too while building inside a CI/CD pipeline in a docker environment without the venv option. I currently run a |
|
Hello everyone, since poetry 1.2.0a2 poetry will no longer remove installed packages unless you use the fin swimmer |
|
Hello, I just tested this with version 1.2.0b1 and i got an error that trace back to so it seems to me the fix correctly stopped to remove packages My use-case is a docker image that pre-install some common dependencies for later use in the ci/cd process of several similar projects. I use the following Dockerfile (alpine based) I don't have the problem if i put |
|
@Cajuteq - you may want to open a new ticket so this gets some attention. |
|
@Cajuteq I am not a 100% across your use case. But I can tell you that using a system environment is not really a practice I recommend even in containers. You could try something like this if it helps. FROM docker.io/python:3.10
RUN curl -sSL https://install.python-poetry.org | python3 - --version 1.2.0b1
ENV PATH /root/.local/bin:$PATH
WORKDIR /opt/app
COPY poetry.lock .
COPY pyproject.toml .
RUN poetry config virtualenvs.in-project true \
&& poetry install --no-interaction --no-ansi$ cat pyproject.toml
[tool.poetry]
<--snipped-->
[tool.poetry.dependencies]
python = "^3.10"
poetry = {git = "https://github.com/python-poetry/poetry.git", rev = "master"}
[tool.poetry.group.dev.dependencies]
#pytest = "^7.1.1"
<--snipped-->
$ docker build -v /tmp/venv:/opt/app/.venv:z -t local/test .
<--snipped-->
STEP 7/7: RUN poetry config virtualenvs.in-project true && poetry install --no-interaction --no-ansi
Recreating virtualenv foobar in /opt/app/.venv
The virtual environment found in /opt/app/.venv seems to be broken.
Installing dependencies from lock file
Package operations: 38 installs, 0 updates, 0 removals
<--snipped-->
COMMIT local/test
--> c0b1dd2af56
Successfully tagged localhost/local/test:latest
c0b1dd2af566219fc04dc6aafc62929f9c94d8789d47865d661c3b989773019f
$ # add new dev dependency for testing
$ cat pyproject.toml
<--snipped-->
[tool.poetry.dependencies]
python = "^3.10"
poetry = {git = "https://github.com/python-poetry/poetry.git", rev = "master"}
[tool.poetry.group.dev.dependencies]
pytest = "^7.1.1"
<--snipped-->
$ docker build -v /tmp/venv:/opt/app/.venv:z -t local/test .
<--snipped-->
STEP 7/7: RUN poetry config virtualenvs.in-project true && poetry install --no-interaction --no-ansi
Installing dependencies from lock file
Package operations: 6 installs, 0 updates, 0 removals
• Installing attrs (21.4.0)
• Installing iniconfig (1.1.1)
• Installing pluggy (1.0.0)
• Installing py (1.11.0)
• Installing tomli (2.0.1)
• Installing pytest (7.1.1)
COMMIT local/test
--> 2edf0b040c3
Successfully tagged localhost/local/test:latest
2edf0b040c3bdf427d461c0f0f640311411952af68f1d4b80b8e09317bbccb7fAlternatively, you can mount in |
There were massive issues with the Dockerfile not building due to:
```
Step 11/13 : RUN poetry config virtualenvs.create false && poetry install --no-dev
---> Running in f70d4cf0e420
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file
Package operations: 30 installs, 0 updates, 1 removal
• Removing setuptools (63.2.0)
• Installing pycparser (2.21)
• Installing cffi (1.15.1)
• Installing cryptography (37.0.4)
• Installing dnspython (2.2.1)
• Installing frozenlist (1.3.0)
• Installing idna (3.3)
• Installing multidict (6.0.2)
• Installing pycares (4.2.1)
• Installing aiodns (3.0.0)
• Installing aiosignal (1.2.0)
• Installing async-timeout (4.0.2)
• Installing attrs (21.4.0)
• Installing brotli (1.0.9)
• Installing cchardet (2.1.7)
• Installing charset-normalizer (2.1.0)
• Installing commonmark (0.9.1)
• Installing email-validator (1.2.1)
• Installing pygments (2.12.0)
• Installing pyjwt (2.4.0)
EnvCommandError
Command ['/usr/local/bin/pip3.10', 'install', '--no-deps', '/root/.cache/pypoetry/artifacts/19/4b/ee/54a2fb42a3316b07a2e3421088822915d6f931052658578ec1aad977ea/cchardet-2.1.7.tar.gz'] errored with the following return code 1, and output:
Processing /root/.cache/pypoetry/artifacts/19/4b/ee/54a2fb42a3316b07a2e3421088822915d6f931052658578ec1aad977ea/cchardet-2.1.7.tar.gz
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
[end of output]
```
It removed its own dependency, `setuptools` and fails afterwards. This
is because `setuptools` is not part of our dependencies *but it's
installed*, so the environment is 'synchronized' and any extraneous
packages are *removed*. Running this in a virtual environment (instead
of `virtualenvs.create false`) didn't help.
See also python-poetry/poetry#3139
Poetry 1.2 fixes it:
https://python-poetry.org/blog/announcing-poetry-1.2.0a2/#package-operations-and-synchronization-improvements
@egormkn I have the exact same issue. Did you manage to resolve this? |
|
Hi team, what's the consideration around the following design decision? I have a similar situation where the package A is defined as optional in my pyproject.toml, and our team only install A when the corresponding platform doesn't provide package A (e.g. we use package A's CPU version, but if the platform has a GPU version of package A, we'll not install package A)
|
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
and others
Issue
poetry installremoves packages that are not supposed to be removed. I'm using Poetry in a Dockerfile to install the dependencies of my project. Before that, the parent image has also installed some dependencies that are necessary.As you can see, there are multiple dependencies that are removed. How can I prevent it?
The text was updated successfully, but these errors were encountered: