Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

download function should avoid race-condition #11646

Closed
szha opened this issue Jul 11, 2018 · 1 comment · Fixed by #12572
Closed

download function should avoid race-condition #11646

szha opened this issue Jul 11, 2018 · 1 comment · Fixed by #12572

Comments

@szha
Copy link
Member

szha commented Jul 11, 2018

Currently the download function would directly write to target file path, which causes file integrity problem when multiple processes use the same logic to write to the same file.
https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/utils.py#L223-L226

Alternative is to write to a temporary file first, and only move to target path upon download completion after verifying file integrity. Temporary file should be cleaned up at the end. This should help solve issues like #11616

@nswamy
Copy link
Member

nswamy commented Jul 18, 2018

You could use tempfile.TemporaryFile or tempfile.NamedTemporaryFile that automatically cleans up after its closed

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants