pipenv lock hangs. It really does. #3827
Comments
ghost
changed the title
frostming
added
Category: Dependency Resolution
|
It seems a good solution, I'm +1 on this. Thanks for your efforts in it. Would you mind send a PR with the fix for this? |
|
I don't think I have ever felt so validated by a comment I left on github in hopes that one day in future someone would stumble upon it. (Turned out to be the near future.) <3 Thank you for this. |
ghost
mentioned this issue
Build times absolutely affect Dependency resolution is an NP hard problem, there is no hack or easy trick around this, and in python it is also a problem that sometimes requires building artifacts. If you have identified and can avoid extra downloads that is excellent, but I do want to be clear: building is often a part of locking. @jackiekazil apologies if you felt invalidated by the previous responses, we are all definitely aware that locking is slow, and as I mentioned in the other threads on the topic I agree that there are likely multiple downloads occurring but am not precisely sure where and would need to see debugging info to make any progress, so if the accompanying PR here addresses that it is awesome |
|
I'm going back to pip. This issue has been around for so long, and is the main blockage for new users. Makes the whole project seem a little bit unprofessional to me. |
|
I have also gone back to pip. Every time an issue is opened, it is ignored. Pipenv, fundamentally, is unusable. Lock fails for me even with 10 relatively small packages. Its a shame because this was a great idea, only to be ruined by the locking mechanism. |
|
my very first installation by pipenv got stuck by stopless "Locking", it just a small package and has already been install succeeded. 1 hour later, it was still there... annoying. |
|
(posting this in the hopes that some reports of hanging are due to the same issue I encountered) After multiple attempts to debug this, I kept seeing the same package causing problems (in my case, it was configparser, but I don't believe that's particularly relevant). I managed to replicate how pipenv runs resolver.py and recreated that environment in order to run the resolver.py under pdb. Under those conditions, I found that resolver.py would get stuck sitting in a sleep loop buried in a call stack that looked like it was trying to do some lockfile operations. When I looked at the directory it was trying to create in order to acquire the lock, it became clear what the problem was. The directory already existed and, in fact, had been on my filesystem for months (the length of time I have been having issues). So in short, once I found: I moved the that subtree to another location and then was able to successfully run pipenv lock again. So for Mac OS, something like 'find ~/Library/Caches/pipenv/http | grep lock' might help highlight issues of this sort. |
|
Well, I really cannot understand this. Long-time ago, pypi doesn't contain metadata of the packages. There is no other choice but downloading all packages to calculate the hash. But it is 2019 now, pypi provides what you need. In the past, I recommended this tool to others. Said "Oh, this is the next generation of pip.", "It is perfect, it not only manage virtual envs for you but also lock your dependencies.", "Just use it." ... |
|
Thanks @michio-nikaido |
|
had the same problem. no error, verbose does nothing. really frustrating. the ** needs globstar |
|
I was really convinced by stuff I read about trying out pipenv. Of course, I immediately had this problem with a ridiculously long locking time. After googling around a bunch, it seems this is a persistent problem, with seemingly several issues open on it. It clearly should be the number one focus for everyone contributing, but it clearly isn't. |
|
It was very confusing, but I figured it out. My internet connection was slow. Check to make sure you're on a solid internet connection. I was on what I thought was a pretty good internet connection where I could browse the web. Here's the result: solid internet connection: takes almost 2 minutes, appears to be working, but slow not so solid internet connection: takes at least 15 minutes, appears to not be working I think if it gave a message after 5 minutes to check that you have a good internet connection that might reduce the number of inquiries. Especially if it could report how many packages (or package metadata, not sure what it's downloading during lock) it could download per minute and what a normal rate is. |
|
Including a progress on the lock command might help "solve" this issue. I just waited for a 15 minute lock, tought for sure it wouldn't work anymore :/ |
|
Having the same issue here... will go back to pip.. |
|
Yes, a progress bar or some meaningful debug output might have made it more tolerable. Right now it's absolutely unclear how long you have to wait or will it ever complete. |
|
Just installed pipenv version 2018.11.26 and encountering this same problem when trying to install opencv and matplotlib, having to wait an excessively long time for pipenv to acquire a lock. This really needs to be addressed. |
|
I have the same problem in all of my projects. Minimal working exampledjbrown@DESKTOP-65P6D75:~$ mkdir test
djbrown@DESKTOP-65P6D75:~$ cd test
djbrown@DESKTOP-65P6D75:~/test$ pipenv install --dev pylint --verbose
Creating a virtualenv for this project…
Pipfile: /home/djbrown/test/Pipfile
Using /usr/bin/python3 (3.6.9) to create virtualenv…
⠸ Creating virtual environment...Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/python3
Also creating executable in /home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/python
Installing setuptools, pip, wheel...done.
✔ Successfully created virtual environment!
Virtualenv location: /home/djbrown/.local/share/virtualenvs/test-PW-auWy_
Creating a Pipfile for this project…
Installing pylint…
⠋ Installing...Installing 'pylint'
$ ['/home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/pip', 'install', '--verbose', '--upgrade', 'pylint', '-i', 'https://pypi.org/simple']
Adding pylint to Pipfile's [dev-packages]…
✔ Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
⠇ Locking... |
|
I have that same problem, pipenv stuck on: for hours. Help!!! |
|
Same here |
|
Same problem. I have a decent internet connection and I stopped my HD streaming while it was running. |
|
Same problem. |
Takes too long to processWhy am I complaining about this? It takes Creating a virtualenv for this project…
Pipfile: /app/Pipfile
Using /usr/local/bin/python (3.6.10) to create virtualenv…
⠹ Creating virtual environment...Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /home/app/.local/share/virtualenvs/app-4ijtfRKB/bin/python
Installing setuptools, pip, wheel...
done.
✔ Successfully created virtual environment!
Virtualenv location: /home/app/.local/share/virtualenvs/app-4ijtfRKB
Installing dependencies from Pipfile.lock (8ec9bf)…
Docker Cache busted during build
Possible causes
$ docker images | grep "<none>"
<none> <none> a6cf739cad9b 23 minutes ago 676MB
<none> <none> 14c14622bb53 About an hour ago 1.49GB
<none> <none> 37af7cfaff6d 3 hours ago 1.49GB
<none> <none> 38c52c39290f 4 hours ago 1.49GB
<none> <none> 72d90c0851fc 4 hours ago 1.49GB
<none> <none> 396ac84d5c09 4 hours ago 1.49GB
<none> <none> c98718725272 5 hours ago 1.49GB
<none> <none> c836d99b5b81 5 hours ago 1.49GB
<none> <none> 2973e08171f7 6 hours ago 1.49GB
<none> <none> 789e39892958 21 hours ago 1.49GB
<none> <none> 5dd243cca7e3 25 hours ago 1.49GB
<none> <none> d345e13492bd 25 hours ago 1.49GB
<none> <none> 43e2e2ae0aba 27 hours ago 1.49GB
<none> <none> b5ac09477eee 28 hours ago 1.49GB
<none> <none> 75188f13c688 28 hours ago 1.49GB
<none> <none> c5e7a082fa51 39 hours ago 1.49GB
<none> <none> 9edadee2549c 39 hours ago 676MB
---> Using cache
---> 6fd43325091c
Step 14/28 : RUN mkdir -p /app
---> Using cache
---> 6f0b7023aa17
Step 15/28 : WORKDIR /app
---> Using cache
---> f56f4bd66bd9
Step 16/28 : COPY Pipfile /app/Pipfile
---> dcbf499a8a6e
Step 17/28 : COPY Pipfile.lock /app/Pipfile.lock
---> 44517cc21c0a
RUN apk update \
Step 18/28 : RUN su-exec app pipenv install --deploy --python=$(which python) --dev
---> Running in 518de56e6a85Testing Pipfile only
$ echo "FROM alpine\nCOPY Pipfile /Pipfile" > Dockerfile.Pipfile && docker build -f Dockerfile.Pipfile -t pipfile:sha .
Sending build context to Docker daemon 400.3MB
Step 1/2 : FROM alpine
---> 961769676411
Step 2/2 : COPY Pipfile /Pipfile
---> Using cache
---> 30efbfd31ef3
Successfully built 30efbfd31ef3
Successfully tagged pipfile:sha
Possible Solution
$ docker images | grep "<none>" | awk '{print $3}' | xargs docker rmi -f
Deleted: sha256:a6cf739cad9bd268a06ff8434121108ae2a95f1a5c05961f62706a844bdc49fc
Deleted: sha256:8dd421ccebab91274a58d5f61567af996f13034e0f59bb2d9f4369e3da6ef0d0
Deleted: sha256:14c14622bb53df2e82f37780c2dd232901198630a986e71b361593e075603a45
Deleted: sha256:07e292542ede3db39d16e41059dac0062691ee7117c4946b9c67a982c5c575a9
Deleted: sha256:81631b2307d3966ee3ba959b0f9ba85db116c4bed240eac8ea9f1eb253796cb8
Deleted: sha256:dd79559a7a15d065ba5f8722f4566f1bbd17ec0f77bd58359e6cd71aedbb2391
Deleted: sha256:d17c782c2507a28c19bc9a984a11a6b4df24790609eececd14dc1be9c152cc5a
Deleted: sha256:b2310eae698e4abc874e81331bf93f6e87385966f02b01f08a067f42fa73aa7f
Deleted: sha256:7d4320440119b2803f6e11c7d8bd7e3a2d348f8697f246602c0bbbb02bb41539
Deleted: sha256:52ac5df58a4f1ba73e4b1d787b5a29b48f1be7c27aee407ab59ed6e08a79053c
Deleted: sha256:281790a15bddda7d0b08dcfc6364a9dc7ef0f4e46d24846afd0de2d6b8331c3b
Deleted: sha256:e6b7049cebd3bd5fa238565754c5cada22d5f78efda8f62206faef83f0ed83e9
Deleted: sha256:f7e1fab55b452200edd64dda1fd19bd4f90c11a15b7f692b1986d5419c43f82c
....
....
Testing the original docker builds started working again and I could be reusing the ---> 3d7c755ae22d
Step 14/28 : RUN mkdir -p /iks
---> Using cache
---> 9b68987bb88c
Step 15/28 : WORKDIR /iks
---> Using cache
---> a3d9938e1b02
Step 16/28 : COPY Pipfile /iks
---> Using cache
---> cbe7844a6111
Step 17/28 : COPY Pipfile.lock /iks
---> Using cache
---> 7db6c35db9c8
Step 18/28 : RUN su-exec iks pipenv install --deploy --python=$(which python) --dev
---> Using cache
---> d1a69218a465 |
|
we had to stop using |
pipenv is really slow and sometimes it hangs for a really long time. Perhaps because I'm using a private PyPi instance that makes it worse, but that works fine with plain pip. This seems to be a common problem - see and the several other linked issues there: pypa/pipenv#3827 So far, this is just the result of grep/search/replace. Haven't tested this at all and there's more work to do. Just want to start by getting some feedback if this is going to be something that everyone is OK with. Closes hashicorp#324
pipenv is really slow and sometimes it hangs for a really long time. Perhaps because I'm using a private PyPi instance that makes it worse, but that works fine with plain pip. This seems to be a common problem - see and the several other linked issues there: pypa/pipenv#3827 So far, this is just the result of grep/search/replace. Haven't tested this at all and there's more work to do. Just want to start by getting some feedback if this is going to be something that everyone is OK with. Closes hashicorp#324
|
+1, could not get locking to conclude even once in Docker. |
|
I'm a new user to pipenv. It wasn't clear to me what on earth it was doing during the delay. Having default feedback indicating that it's performing a network action (ideally with progress bar) or that it's in the middle of building something, would have helped. |
|
if pipenv locking gets stuck somewhere first skip the lock part by --skip-lock it works. |
|
Subscribers to this issue, please test the performance against the latest master branch. Feedbacks are welcome |
Hi, I was able to get things done using the latest from main branch. Thanks. It would still be great to have a verbose mode to be sure what's being processed. But, for the main topic of the issue (it hangs on locking), for me, it worked like a charm. |
|
+1, the pipenv is stuck pinning packages from VCS |
|
@ZhukovAlexander which version of pipenv after you using? |
|
@Mause it's 2018.11.26. I've already found the problem. I needed to configure git credentials so git can authenticate when cloning packages from VCS. Otherwise pipenv was stuck waiting for prompt, basically, without telling anything. |
|
I'd also suggest updating to the latest version so you don't have further issues |
### Changes * Implements minesweeper gym env * Adds unit tests * Changes to using `virtualenv` over `pipenv` due to [locking taking an eternity](pypa/pipenv#3827)
|
Still slow locking, had it running for hours without anything happening. It should be mentioned its a large project with 35 packages where 2 are local folders. The verbose flag does nothing in telling me if anything is happening or what step is hanging, it just hangs and taunts me with the dancing pipenv, version 2020.11.15 |
|
I just started using pipenv and now stumbled across this problem. For me happening the same long stuck time without any feedback about whats happening while creating the Pipfile.lock. Libs effected were Numpy, SciPy and Matplotlib. |
|
Update: My problem was related to an unreachable source, when i removed it it worked properly again. I would suggest having a timeout and/or show source reaching attempts when running in --verbose |
|
Just wow.. I tried using pipenv for the first time today and just wasted an hour trying to troubleshoot before discovering this known issue. Can't believe it has been around for so long and no warning about it. Back to conda for me.. |
|
This was generated on a poor vpn network connection. most of the time is spent in SSL.py read() |
2022.1.8 (2022-01-08) ===================== Bug Fixes --------- - Remove the extra parentheses around the venv prompt. `#4877 <https://github.com/pypa/pipenv/issues/4877>`_ - Fix a bug of installation fails when extra index url is given. `#4881 <https://github.com/pypa/pipenv/issues/4881>`_ - Fix regression where lockfiles would only include the hashes for releases for the platform generating the lockfile `#4885 <https://github.com/pypa/pipenv/issues/4885>`_ - Fix the index parsing to reject illegal requirements.txt. `#4899 <https://github.com/pypa/pipenv/issues/4899>`_ 2021.11.23 (2021-11-23) ======================= Bug Fixes --------- - Update ``charset-normalizer`` from ``2.0.3`` to ``2.0.7``, this fixes an import error on Python 3.6. `#4865 <https://github.com/pypa/pipenv/issues/4865>`_ - Fix a bug of deleting a virtualenv that is not managed by Pipenv. `#4867 <https://github.com/pypa/pipenv/issues/4867>`_ - Fix a bug that source is not added to ``Pipfile`` when index url is given with ``pipenv install``. `#4873 <https://github.com/pypa/pipenv/issues/4873>`_ 2021.11.15 (2021-11-15) ======================= Bug Fixes --------- - Return an empty dict when ``PIPENV_DONT_LOAD_ENV`` is set. `#4851 <https://github.com/pypa/pipenv/issues/4851>`_ - Don't use ``sys.executable`` when inside an activated venv. `#4852 <https://github.com/pypa/pipenv/issues/4852>`_ Vendored Libraries ------------------ - Drop the vendored ``jinja2`` dependency as it is not needed any more. `#4858 <https://github.com/pypa/pipenv/issues/4858>`_ - Update ``click`` from ``8.0.1`` to ``8.0.3``, to fix a problem with bash completion. `#4860 <https://github.com/pypa/pipenv/issues/4860>`_ - Drop unused vendor ``chardet``. `#4862 <https://github.com/pypa/pipenv/issues/4862>`_ Improved Documentation ---------------------- - Fix the documentation to reflect the fact that special characters must be percent-encoded in the URL. `#4856 <https://github.com/pypa/pipenv/issues/4856>`_ 2021.11.9 (2021-11-09) ====================== Features & Improvements ----------------------- - Replace ``click-completion`` with ``click``'s own completion implementation. `#4786 <https://github.com/pypa/pipenv/issues/4786>`_ Bug Fixes --------- - Fix a bug that ``pipenv run`` doesn't set environment variables correctly. `#4831 <https://github.com/pypa/pipenv/issues/4831>`_ - Fix a bug that certifi can't be loaded within ``notpip``'s vendor library. This makes several objects of ``pip`` fail to be imported. `#4833 <https://github.com/pypa/pipenv/issues/4833>`_ - Fix a bug that ``3.10.0`` can be found be python finder. `#4837 <https://github.com/pypa/pipenv/issues/4837>`_ Vendored Libraries ------------------ - Update ``pythonfinder`` from ``1.2.8`` to ``1.2.9``. `#4837 <https://github.com/pypa/pipenv/issues/4837>`_ 2021.11.5.post0 (2021-11-05) ============================ Bug Fixes --------- - Fix a regression that ``pipenv shell`` fails to start a subshell. `#4828 <https://github.com/pypa/pipenv/issues/4828>`_ - Fix a regression that ``pip_shims`` object isn't imported correctly. `#4829 <https://github.com/pypa/pipenv/issues/4829>`_ 2021.11.5 (2021-11-05) ====================== Features & Improvements ----------------------- - Avoid sharing states but create project objects on demand. So that most integration test cases are able to switch to a in-process execution method. `#4757 <https://github.com/pypa/pipenv/issues/4757>`_ - Shell-quote ``pip`` commands when logging. `#4760 <https://github.com/pypa/pipenv/issues/4760>`_ Bug Fixes --------- - Ignore empty .venv in rood dir and create project name base virtual environment `#4790 <https://github.com/pypa/pipenv/issues/4790>`_ Vendored Libraries ------------------ - Update vendored dependencies - ``attrs`` from ``20.3.0`` to ``21.2.0`` - ``cerberus`` from ``1.3.2`` to ``1.3.4`` - ``certifi`` from ``2020.11.8`` to ``2021.5.30`` - ``chardet`` from ``3.0.4`` to ``4.0.0`` - ``click`` from ``7.1.2`` to ``8.0.1`` - ``distlib`` from ``0.3.1`` to ``0.3.2`` - ``idna`` from ``2.10`` to ``3.2`` - ``importlib-metadata`` from ``2.0.0`` to ``4.6.1`` - ``importlib-resources`` from ``3.3.0`` to ``5.2.0`` - ``jinja2`` from ``2.11.2`` to ``3.0.1`` - ``markupsafe`` from ``1.1.1`` to ``2.0.1`` - ``more-itertools`` from ``5.0.0`` to ``8.8.0`` - ``packaging`` from ``20.8`` to ``21.0`` - ``pep517`` from ``0.9.1`` to ``0.11.0`` - ``pipdeptree`` from ``1.0.0`` to ``2.0.0`` - ``ptyprocess`` from ``0.6.0`` to ``0.7.0`` - ``python-dateutil`` from ``2.8.1`` to ``2.8.2`` - ``python-dotenv`` from ``0.15.0`` to ``0.19.0`` - ``pythonfinder`` from ``1.2.5`` to ``1.2.8`` - ``requests`` from ``2.25.0`` to ``2.26.0`` - ``shellingham`` from ``1.3.2`` to ``1.4.0`` - ``six`` from ``1.15.0`` to ``1.16.0`` - ``tomlkit`` from ``0.7.0`` to ``0.7.2`` - ``urllib3`` from ``1.26.1`` to ``1.26.6`` - ``zipp`` from ``1.2.0`` to ``3.5.0`` Add new vendored dependencies - ``charset-normalizer 2.0.3`` - ``termcolor 1.1.0`` - ``tomli 1.1.0`` - ``wheel 0.36.2`` `#4747 <https://github.com/pypa/pipenv/issues/4747>`_ - Drop the dependencies for Python 2.7 compatibility purpose. `#4751 <https://github.com/pypa/pipenv/issues/4751>`_ - Switch the dependency resolver from ``pip-tools`` to `pip`. Update vendor libraries: - Update ``requirementslib`` from ``1.5.16`` to ``1.6.1`` - Update ``pip-shims`` from ``0.5.6`` to ``0.6.0`` - New vendor ``platformdirs 2.4.0`` `#4759 <https://github.com/pypa/pipenv/issues/4759>`_ Improved Documentation ---------------------- - remove prefixes on install commands for easy copy/pasting `#4792 <https://github.com/pypa/pipenv/issues/4792>`_ - Officially drop support for Python 2.7 and Python 3.5. `#4261 <https://github.com/pypa/pipenv/issues/4261>`_ 2021.5.29 (2021-05-29) ====================== Bug Fixes --------- - Fix a bug where passing --skip-lock when PIPFILE has no [SOURCE] section throws the error: "tomlkit.exceptions.NonExistentKey: 'Key "source" does not exist.'" `#4141 <https://github.com/pypa/pipenv/issues/4141>`_ - Fix bug where environment wouldn't activate in paths containing & and $ symbols `#4538 <https://github.com/pypa/pipenv/issues/4538>`_ - Fix a bug that ``importlib-metadata`` from the project's dependencies conflicts with that from ``pipenv``'s. `#4549 <https://github.com/pypa/pipenv/issues/4549>`_ - Fix a bug where ``pep508checker.py`` did not expect double-digit Python minor versions (e.g. "3.10"). `#4602 <https://github.com/pypa/pipenv/issues/4602>`_ - Fix bug where environment wouldn't activate in paths containing () and [] symbols `#4615 <https://github.com/pypa/pipenv/issues/4615>`_ - Fix bug preventing use of pipenv lock --pre `#4642 <https://github.com/pypa/pipenv/issues/4642>`_ Vendored Libraries ------------------ - Update ``packaging`` from ``20.4`` to ``20.8``. `#4591 <https://github.com/pypa/pipenv/issues/4591>`_ 2020.11.15 (2020-11-15) ======================= Features & Improvements ----------------------- - Support expanding environment variables in requirement URLs. `#3516 <https://github.com/pypa/pipenv/issues/3516>`_ - Show warning message when a dependency is skipped in locking due to the mismatch of its markers. `#4346 <https://github.com/pypa/pipenv/issues/4346>`_ Bug Fixes --------- - Fix a bug that executable scripts with leading backslash can't be executed via ``pipenv run``. `#4368 <https://github.com/pypa/pipenv/issues/4368>`_ - Fix a bug that VCS dependencies always satisfy even if the ref has changed. `#4387 <https://github.com/pypa/pipenv/issues/4387>`_ - Restrict the acceptable hash type to SHA256 only. `#4517 <https://github.com/pypa/pipenv/issues/4517>`_ - Fix the output of ``pipenv scripts`` under Windows platform. `#4523 <https://github.com/pypa/pipenv/issues/4523>`_ - Fix a bug that the resolver takes wrong section to validate constraints. `#4527 <https://github.com/pypa/pipenv/issues/4527>`_ Vendored Libraries ------------------ - Update vendored dependencies: - ``colorama`` from ``0.4.3`` to ``0.4.4`` - ``python-dotenv`` from ``0.10.3`` to ``0.15.0`` - ``first`` from ``2.0.1`` to ``2.0.2`` - ``iso8601`` from ``0.1.12`` to ``0.1.13`` - ``parse`` from ``1.15.0`` to ``1.18.0`` - ``pipdeptree`` from ``0.13.2`` to ``1.0.0`` - ``requests`` from ``2.23.0`` to ``2.25.0`` - ``idna`` from ``2.9`` to ``2.10`` - ``urllib3`` from ``1.25.9`` to ``1.26.1`` - ``certifi`` from ``2020.4.5.1`` to ``2020.11.8`` - ``requirementslib`` from ``1.5.15`` to ``1.5.16`` - ``attrs`` from ``19.3.0`` to ``20.3.0`` - ``distlib`` from ``0.3.0`` to ``0.3.1`` - ``packaging`` from ``20.3`` to ``20.4`` - ``six`` from ``1.14.0`` to ``1.15.0`` - ``semver`` from ``2.9.0`` to ``2.13.0`` - ``toml`` from ``0.10.1`` to ``0.10.2`` - ``cached-property`` from ``1.5.1`` to ``1.5.2`` - ``yaspin`` from ``0.14.3`` to ``1.2.0`` - ``resolvelib`` from ``0.3.0`` to ``0.5.2`` - ``pep517`` from ``0.8.2`` to ``0.9.1`` - ``zipp`` from ``0.6.0`` to ``1.2.0`` - ``importlib-metadata`` from ``1.6.0`` to ``2.0.0`` - ``importlib-resources`` from ``1.5.0`` to ``3.3.0`` `#4533 <https://github.com/pypa/pipenv/issues/4533>`_ Improved Documentation ---------------------- - Fix suggested pyenv setup to avoid using shimmed interpreter `#4534 <https://github.com/pypa/pipenv/issues/4534>`_ 2020.11.4 (2020-11-04) ====================== Features & Improvements ----------------------- - Add a new command ``pipenv scripts`` to display shortcuts from Pipfile. `#3686 <https://github.com/pypa/pipenv/issues/3686>`_ - Retrieve package file hash from URL to accelerate the locking process. `#3827 <https://github.com/pypa/pipenv/issues/3827>`_ - Add the missing ``--system`` option to ``pipenv sync``. `#4441 <https://github.com/pypa/pipenv/issues/4441>`_ - Add a new option pair ``--header/--no-header`` to ``pipenv lock`` command, which adds a header to the generated requirements.txt `#4443 <https://github.com/pypa/pipenv/issues/4443>`_ Bug Fixes --------- - Fix a bug that percent encoded characters will be unquoted incorrectly in the file URL. `#4089 <https://github.com/pypa/pipenv/issues/4089>`_ - Fix a bug where setting PIPENV_PYTHON to file path breaks environment name `#4225 <https://github.com/pypa/pipenv/issues/4225>`_ - Fix a bug that paths are not normalized before comparison. `#4330 <https://github.com/pypa/pipenv/issues/4330>`_ - Handle Python major and minor versions correctly in Pipfile creation. `#4379 <https://github.com/pypa/pipenv/issues/4379>`_ - Fix a bug that non-wheel file requirements can be resolved successfully. `#4386 <https://github.com/pypa/pipenv/issues/4386>`_ - Fix a bug that ``pexept.exceptions.TIMEOUT`` is not caught correctly because of the wrong import path. `#4424 <https://github.com/pypa/pipenv/issues/4424>`_ - Fix a bug that compound TOML table is not parsed correctly. `#4433 <https://github.com/pypa/pipenv/issues/4433>`_ - Fix a bug that invalid Python paths from Windows registry break ``pipenv install``. `#4436 <https://github.com/pypa/pipenv/issues/4436>`_ - Fix a bug that function calls in ``setup.py`` can't be parsed rightly. `#4446 <https://github.com/pypa/pipenv/issues/4446>`_ - Fix a bug that dist-info inside ``venv`` directory will be mistaken as the editable package's metadata. `#4480 <https://github.com/pypa/pipenv/issues/4480>`_ - Make the order of hashes in resolution result stable. `#4513 <https://github.com/pypa/pipenv/issues/4513>`_ Vendored Libraries ------------------ - Update ``tomlkit`` from ``0.5.11`` to ``0.7.0``. `#4433 <https://github.com/pypa/pipenv/issues/4433>`_ - Update ``requirementslib`` from ``1.5.13`` to ``1.5.14``. `#4480 <https://github.com/pypa/pipenv/issues/4480>`_ Improved Documentation ---------------------- - Discourage homebrew installation in installation guides. `#4013 <https://github.com/pypa/pipenv/issues/4013>`_
|
If we can get more verbose log in the lock phase will be very helpful since there're different root causes. |
|
I ran the same I believe either it's a bug in pipenv caused by the circular references at pypi.org or python.org, or else these two sites are curated at different intervals? |
|
@Amondale What version are you on? If its the latest If you find that its still an issue, I suggest opening a new issue with a reproduceable test case, or look at the linked issues off that PR and decide if they match the issue you are facing. |
|
Just tested this with Then I ran |
|
We have done some performance enhancements to pipenv in recent times, including a big install optimization which released 2022.8.31. For an independent comparison of benchmarks, please have a look at: lincolnloop.github.io/python-package-manager-shootout For new issues related to performance: open a new issue report. |
and others
This is a followup to #2681, which was closed without addressing the issue for some users.
Also #3812, #3829, SO and someone's blog rant.
Summary
pipenv lockdownloads every available artifact of installed packages and their dependencies. It does this to calculate their hashes, even when the artifact url includes the hash in a fragment. For some large packages, such as scipy, which have large dependencies and many artifacts per version, this behavior can result in unreasonably long delays for some users (893MB vs. 50MB download). It's also bad netiquette towards pypi.Symptoms
pipenv lockhanging when installing scipy#2681 was closed by @techalchemy with a comment suggesting the delay is due to lengthy build times (which don't affect
pipenv lock), but asked users to provide steps to reproduce.note:
All the packages fetched have wheels.
Details
pipenv.lockcallsResolver.resolve()which enables all artifactspipenv/pipenv/patched/piptools/resolver.py
Lines 68 to 73 in 4c00352
For a common setup consisting of scipy, pandas and numpy, here's the list of artifacts:
Artifacts Queued for Hash Retrieval
That's a lot of sequential network round-trips to wait through, and most of these are for different platforms than the installation. But that's not all.
Each artifact is passed to
HashCache.get_hash.pipenv/pipenv/patched/piptools/repositories/pypi.py
Lines 60 to 71 in 4c00352
The pypi artifact urls includes the sha256 hash, parsed into
new_location.hash, but it isn't used. If theHashCachedoesn't already hold the location key, the artifact is downloaded, hashed and then the hash is stored. This is the root cause of the delay, and in the above examples results in downloading 893MB of artifacts, compared to the 50MBs-worth which are installed.The problem disappears if the user is patient enough to wait it out, or if the connection is fast. However, it seems most users are surprised by the delay (me included), and expect it to be more or less instantaneous once the packages are installed.
As a quick verification, I patched the
get_hashmethod to use the hash fragment from the url if available, and thepipenv lockrun time dropped to a few seconds.Sidenote,
--verbosedoes not log network requests, so this was masked from view even when trying to debug.The text was updated successfully, but these errors were encountered: