Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[BUGFIX] fix model zoo parallel download #17336

Closed
wants to merge 3 commits into from
Closed

[BUGFIX] fix model zoo parallel download #17336

wants to merge 3 commits into from

Conversation

eric-haibin-lin
Copy link
Member

@eric-haibin-lin eric-haibin-lin commented Jan 16, 2020 •
edited
Loading

Description

Fixes #17332

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • Feature1, tests, (and when applicable, API doc)
  • Feature2, tests, (and when applicable, API doc)

Comments

  • If this change is a backward incompatible change, why must this change be made.
  • Interesting edge cases to note here

apeforest
apeforest reviewed Jan 17, 2020
View reviewed changes
Copy link
Contributor

@apeforest apeforest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently when I ran the mxnet_mnist example in horovod, the data-XXX folder remains even after the training finished. Does this PR also address the auto clean up issue?

python/mxnet/gluon/model_zoo/model_store.py
@@ -103,16 +106,16 @@ def get_model_file(name, root=os.path.join(base.data_dir(), 'models')):

util.makedirs(root)

zip_file_path = os.path.join(root, file_name+'.zip')
temp_zip_file_path = os.path.join(root, file_name+random_uuid+'.zip')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to use tempfile so it takes care of clean up automatically: https://docs.python.org/3/library/tempfile.html

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe tempfile is typically on a different filesystem, so there is no atomic rename operation to the target directory and filename available.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@leezu What do you mean by a different filesystem? Regarding name, I think you can also specify a file name using the NamedTemporaryfile class?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I overlooked the dir argument.

tests/python/unittest/test_gluon_model_zoo.py
threads = []
name = 'mobilenetv2_0.25'
for _ in range(10):
x = threading.Thread(target=fn, args=(name,))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use multiprocess to test? My understand is that in horovod get_model is parallelized at process level not thread level?

@eric-haibin-lin
Copy link
Member Author

Moved to #17372

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@apeforest apeforest apeforest left review comments

@leezu leezu leezu left review comments

@szha szha Awaiting requested review from szha szha is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Race condition in downloading model from model zoo in parallel
3 participants
@eric-haibin-lin @leezu @apeforest