-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poetry cannot correctly select dependencies #5896
Comments
That's not a "repeated" message, poetry is (slowly) working its way through the many versions of awscli that are not compatible with the selections it has previously made at that point in the search. There are some performance improvements in the latest beta that might help a bit, but your best bet is almost certainly to specify that you want some recent version of awscli. That will allow poetry to fail much faster - and backtrack the search and find a solution. |
Thanks, I mean that it was repeatedly trying to solve. Adding in the |
We are having this problem since more than half a year. Last time clearing the poetry caches helped, this time we had to do more. |
I checked with the latest poetry version 1.2.0b2 and it did not help. Setting a minimum version for awscli was the only real solution for us. |
@mihirsamdarshi @mkniewallner could you please explain why did you close this issue? We are still facing the same problem with the latest Poetry release and it's reproducible in a stable way. The changed in the other issues didn't solve the problem and there is no other issue linked that is open. If there is no other issue tracking this performance issue, could you please reopen this one?
|
I am not sure if this is the problem or not for this specific issue but i am on poetry 1.5.1 and am able to get poetry to spin for a long time in a brand new project by just running
my understanding is that if any package selects urllib3 that is in conflict with botocore (ie >= 2.0) then your stuck downloading every boto3 library between the first time that you added the library (because poetry put ^1.x.x on whatever day you originally added it) and now (boto3 publishes new versions very often) would it be possible to do some kind of shallow discovery of libraries that publish a lot of versions in order to select a new transitive dependency version. in this case it still resolves after a long time of downloading 100s of boto versions but only because it rules out all versions of boto and restarts at the top with a lower urllib version. |
The dependency resolution is known to be quite slow of The options to improve this are limited by Poetry. Best way, as others explained above, is to limit the version range that Poetry should try. |
@finswimmer, would you please be so kind as to comment on what exactly limits Poetry's options to improve this, so that people facing this problem can understand why it's closed as not planned and they have to introduce an artificial lower limit to limit the packages that are scanned? My vague understanding was that there is no metadata service, so in some cases Poetry has to download the packages from the beginning of the day to extract the metadata first to make sure they don't match, and this causes the slowness. Is this understanding correct? If so, would it make sense to bring this up with the PyPI / PyPA people? |
P.S. Just saw that in #8823 @dimbleby explained:
I think that's as close to an explanation as I can get, but I'm still confused as to why backtracking through thousands of versions is a problem. It seems to me that if you have all the relevant constraints at hand, it shouldn't be difficult to solve. So is it true that constraints can only be obtained by downloading the packages in question, and that's the root cause of the issue? |
It depends:
|
But OP is using PyPI and boto3 provides wheels, so the problem is really in the efficiency of Python code in this case? |
Not sure if it is about Python or just network requests or the algorithm itself (independent from the programming language). |
TL;DR there are many moving parts between network, algorithms, and Python, but boto having daily releases and a ton of versions to check doesn't help. |
Well, I think it makes a huge difference. I remember that about two years ago my colleagues complaining that Poetry was saturating our 500 Mbit link, so it definitively felt like it's downloading half of the world. If nowadays it's "just" network requests and/or resolution code, then in theory something could be done about it at the Poetry side. But I see, I guess I can only get these detailed answers by looking at the code myself :( Thank you for the pointers! |
-vvv
option).Issue
With this particular
pyproject.toml
Poetry is unable to select a version ofawscli
, regardless of if I runpoetry update
,poetry install
, orpoetry lock
When running with
-vvv
it hangs with the following repeated message:The text was updated successfully, but these errors were encountered: