Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tomlkit lib slows poetry #7097

Closed
bwiklak opened this issue Nov 25, 2022 · 2 comments
Closed

Tomlkit lib slows poetry #7097

bwiklak opened this issue Nov 25, 2022 · 2 comments
Labels
status/duplicate Duplicate issues

Comments

@bwiklak
Copy link

bwiklak commented Nov 25, 2022

  • poetry: 1.2.2
  • python: 3.9.13
  • os: MacOS 12.6

Issue

Hi,
I'm changing our virtualenv based solution to poetry. We're running massive amount of tests in batches on multiple machines.
Every time the batch starts we'll need to check if current checkout lock file is in sync with dependencies.
I checked how does poetry install compares with pip install -r requirements for a clean/in-sync repo and had seen big differences:

On my mac m1 pro poetry install takes 1.5s vs 700ms for pip (user time)
On our oldest macs that are running tests it's up to 6.5s vs 2s.

It's quite long for every test batch so I started digging.
I was even more surprised when I found out that poetry uses cleaver hashing trick to speed up such case - so why such a difference between brute-force pip install?
After a bit of profiling I saw that majority of time is spent in parsing toml lock file.
As a proof of concept I did a quick&dirty rewrite on packages/locker.py and used a tomli lib to parse lock.
My results are now 0.56s user time vs 1.5s. Haven't check on older macs yet.

My question is - how would you see using a simpler/faster lib for toml file parsing for time critical parts?

@bwiklak bwiklak added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Nov 25, 2022
@Secrus
Copy link
Member

Secrus commented Nov 25, 2022

See #6562, it's already done, just on not yet released version. You can track release progress here: #6972

@Secrus Secrus closed this as completed Nov 25, 2022
@neersighted neersighted added status/duplicate Duplicate issues and removed kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Nov 25, 2022
@neersighted neersighted closed this as not planned Won't fix, can't repro, duplicate, stale Nov 25, 2022
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status/duplicate Duplicate issues
Projects
None yet
Development

No branches or pull requests

3 participants