Concurrent poetry install fails due to cache issue

[abowden1989]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
  • possibly related to Potentially a concurrency issue while installing packages #4568, hard to say
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • because this is a concurrency issue it doesn't always happen, so there isn't a guaranteed reproducer to execute with -vvv

• Ubuntu 18.04:v1.0.2 (docker container)
• Seen on multiple versions (issue seems to be in dependency)
• pyproject.toml not particularly relevant

Issue

We have a build script that runs the poetry install command concurrently to install into virtual environments for multiple different python versions. This fails sporadically in CI with an exception from the cachy library:

STDOUT:     Updating dependencies
STDOUT:     Resolving dependencies...
STDOUT:     
STDOUT:       ValueError
STDOUT:     
STDOUT:       invalid literal for int() with base 10: b''
STDOUT:     
STDOUT:       at /usr/local/lib/python3.6/dist-packages/cachy/stores/file_store.py:65 in _get_payload
STDOUT:            61│ 
STDOUT:            62│         with open(path, 'rb') as fh:
STDOUT:            63│             contents = fh.read()
STDOUT:            64│ 
STDOUT:         →  65│         expire = int(contents[:10])
STDOUT:            66│ 

The multiple concurrent poetry install commands are using the same cache directory, and it looks like the code in the cachy library is not thread safe - https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py.

It appears that if one process is performing a cache write at https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py#L102-L103 while another is doing a read at https://github.com/sdispater/cachy/blob/master/cachy/stores/file_store.py#L62-L63, the read could result in reading an empty file before the contents are written, and then L65 attempts to cast an empty set of bytes to an integer and throws.

Ideally the cache would be thread safe, but documentation saying that concurrent installs are not safe would be an alternative.

A possible workaround that we will attempt is to set the cache-dir in the config to different directories for different concurrent installs.

UPDATE:
The attempted workaround of setting a specific cache-dir in the config does not work, likely due to the issue described in #2445 meaning the repository cache is still shared. As a workaround we are now going to try setting both the poetry cache-dir and XDG_CACHE_HOME to different directories for different installs

[John15321]

Should this issue stay open or be closed?

[neersighted]

Fixed by #6186

[rrauenza]

If this ever happens again, a workaround is to just run poetry with flock,

/usr/bin/flock $(poetry config cache-dir).lock poetry <command>

It won't be concurrent.... but will work.

(Thanks for fixing!!)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.