Add non-existing resume_from_checkpoint acceptance for auto-resubmit #4402

tarepan · 2020-10-28T00:07:42Z

What does this PR do?

Enable resume_from_checkpoint file == empty in Trainer for auto-resume/resubmit.
Feature #4366

Before submitting

proposed Not-yet-existing resume_from_checkpoint for auto-resubmit #4366 <= Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?
Did you verify new and existing tests pass locally with your changes?
If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 👍

codecov · 2020-10-28T00:30:59Z

Codecov Report

Merging #4402 (743fe31) into master (ab7512d) will increase coverage by 0%.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #4402   +/-   ##
======================================
  Coverage      93%     93%           
======================================
  Files         134     134           
  Lines        9950    9955    +5     
======================================
+ Hits         9261    9266    +5     
  Misses        689     689

pytorch_lightning/trainer/connectors/checkpoint_connector.py

Add fsspec's HTTPFileSystem support through http extras. pl has supported remote http file (e.g. Lightning-AI#2925), so this commit do not add new functionality.

tchaton

Great addition !

pytorch_lightning/trainer/connectors/checkpoint_connector.py

tarepan · 2020-11-05T00:45:43Z

@justusschock Very sorry, I mistakenly make re-review request to you...

…lightning

tarepan · 2020-12-23T23:46:28Z

To anyone:
I encountered weird build-Docs CI error.
I failed to fix this error, so we cannot merge this PR.
Please help.

Summary

build-Docs failed with import error.
This PR did not touch any import statement or directory structure.
I cannot find the cause of this error.

Error

CircleCI build-Docs

[autosummary] generating autosummary for: BECOMING_A_CORE_CONTRIBUTOR.md, ...(※ abbreviated by me)
Failed to import 'pytorch_lightning.profiler': no module named pytorch_lightning.profiler

Warning, treated as error:
[autosummary] failed to import 'pytorch_lightning.callbacks.Callback': no module named pytorch_lightning.callbacks.Callback
Makefile:19: recipe for target 'html' failed
make: *** [html] Error 2

Exited with code exit status 2
CircleCI received exit code 2

Servey

Current master do not suffer from this error (e.g. CircleCI build-Docs Success in master).
This PR have pytorch_lightning.profiler, which is the module CI error says 'failed to import'

This PR follow the master branch, and do not touch any module import matters.
I guess that Sphinx through Make could have some relationship with this error.
For example, rebuild of long-untouch-file could reveal embedded/hidden bugs (because Sphinx through Make reuse unchanged files).

pytorch_lightning/trainer/connectors/checkpoint_connector.py

tarepan · 2021-01-02T12:16:23Z

The Build-Docs CI bug is fixed.

awaelchli

@Borda it goes to 1.2 branch yes?

tests/models/test_restore.py

requirements/docs.txt

Co-authored-by: Adrian Wälchli <[email protected]>

…existing_checkpoint

tarepan · 2021-01-04T23:58:44Z

Ready to Merge.

2 Red CIs are not from this PR.

CI build Docker / build-conda (3.8, 1.6)
CI build Docker / build-conda (3.7, 1.7)

The cause of both CI is Horovod build with GPU support was requested but this PyTorch installation does not support CUDA.

…4402) * Add empty resume_from_checkpoint acceptance #4366 * Fix general error catch with focused file check * Add fsspec HTTP extras Add fsspec's HTTPFileSystem support through http extras. pl has supported remote http file (e.g. #2925), so this commit do not add new functionality. * Fix potential too much logging in DDP * Add PR changelog * Add well-written argument explanation Co-authored-by: Adrian Wälchli <[email protected]> * Fix DDP-compatible restore logging Notify from where the states are restored. This feature temporally deleted as a result of PR review. With succeeding review, added with DDP compatibility. * Fix utility import pathes * Refactor load step commentaries * Refactor hpc ckpt suffix acquisition * Refactor restore/hpc_load match * Refactor hpc load trial * Refactor checkpoint dir check * Refactor unneeded function nest * Refactor nested If * Refactor duplicated cache clear * Refactor attempt flow with if/elif * Fix pip8 * Refactor hook commentary Co-authored-by: chaton <[email protected]> * Fix pep8 * Refactor hpc load checkpoint path acquisition * Fix pip8 * Fix typo Co-authored-by: Adrian Wälchli <[email protected]> * Fix typo Co-authored-by: Adrian Wälchli <[email protected]> * Fix doc Co-authored-by: Adrian Wälchli <[email protected]> * Refactor None Union type with Optional * Fix build-doc CI failure debuged in #5329 * Fix fsspec import during build-doc #5329 * Fix test epoch Co-authored-by: Adrian Wälchli <[email protected]> * Fix test with latest test models * . Co-authored-by: Adrian Wälchli <[email protected]> Co-authored-by: chaton <[email protected]> Co-authored-by: Jirka Borovec <[email protected]> Co-authored-by: Sean Naren <[email protected]> Co-authored-by: Roger Shieh <[email protected]> (cherry picked from commit b0051e8)

tarepan changed the title ~~Add empty resume_from_checkpoint acceptance for auto-resubmit~~ Add non-existing resume_from_checkpoint acceptance for auto-resubmit Oct 28, 2020

tarepan marked this pull request as ready for review October 29, 2020 09:59

tarepan requested review from ananyahjha93, awaelchli, Borda, justusschock, nateraw, SeanNaren, tchaton, teddykoker and williamFalcon as code owners October 29, 2020 09:59

SeanNaren reviewed Oct 29, 2020

View reviewed changes

pytorch_lightning/trainer/connectors/checkpoint_connector.py Outdated Show resolved Hide resolved

awaelchli added checkpointing Related to checkpointing feature Is an improvement or enhancement labels Oct 30, 2020

tarepan added 3 commits October 30, 2020 22:01

Add empty resume_from_checkpoint acceptance Lightning-AI#4366

43f989d

Fix general error catch with focused file check

21b60b7

Add fsspec HTTP extras

1b29438

Add fsspec's HTTPFileSystem support through http extras. pl has supported remote http file (e.g. Lightning-AI#2925), so this commit do not add new functionality.

tarepan force-pushed the feature/4366_non_existing_checkpoint branch from a752190 to 1b29438 Compare October 30, 2020 13:04

tarepan added 2 commits October 31, 2020 16:40

Merge branch 'master' into feature/4366_non_existing_checkpoint

697ebc3

Merge branch 'master' into feature/4366_non_existing_checkpoint

fb418b8

tchaton reviewed Nov 2, 2020

View reviewed changes

pytorch_lightning/trainer/connectors/checkpoint_connector.py Outdated Show resolved Hide resolved

Fix potential too much logging in DDP

0d9d763

justusschock approved these changes Nov 3, 2020

View reviewed changes

tarepan added 2 commits November 4, 2020 08:30

Merge branch 'master' into feature/4366_non_existing_checkpoint

742ce48

Merge branch 'master' into feature/4366_non_existing_checkpoint

a128112

tarepan requested review from SeanNaren, justusschock and tchaton November 5, 2020 00:42

tarepan added 4 commits December 14, 2020 03:55

Merge branch 'master' into feature/4366_non_existing_checkpoint

9fcb140

Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…

79da4ff

…lightning

Merge branch 'master' into feature/4366_non_existing_checkpoint

1ac9f6a

Merge branch 'master' into feature/4366_non_existing_checkpoint

7a7ec4b

Borda enabled auto-merge (squash) December 23, 2020 16:22

Merge branch 'master' into feature/4366_non_existing_checkpoint

5c031f4

rohitgr7 reviewed Dec 26, 2020

View reviewed changes

pytorch_lightning/trainer/connectors/checkpoint_connector.py Show resolved Hide resolved

Merge branch 'master' into feature/4366_non_existing_checkpoint

292662f

s-rog disabled auto-merge December 28, 2020 03:22

Merge branch 'master' into feature/4366_non_existing_checkpoint

70ea89c

tarepan mentioned this pull request Jan 2, 2021

Draft PR: Circle CI build-doc debug #5329

Closed

1 task

tarepan added 2 commits January 2, 2021 17:47

Fix build-doc CI failure debuged in Lightning-AI#5329

77bf8c1

Fix fsspec import during build-doc Lightning-AI#5329

383e40e

awaelchli reviewed Jan 2, 2021

View reviewed changes

tests/models/test_restore.py Outdated Show resolved Hide resolved

requirements/docs.txt Outdated Show resolved Hide resolved

tarepan and others added 3 commits January 3, 2021 06:15

Fix test epoch

f6eb95a

Co-authored-by: Adrian Wälchli <[email protected]>

Fix test with latest test models

743fe31

.

1444238

Borda enabled auto-merge (squash) January 4, 2021 19:55

tarepan added 4 commits January 5, 2021 07:05

Merge remote-tracking branch 'upstream/master' into feature/4366_non_…

b7bdd64

…existing_checkpoint

Refactor argument doc of resume_from_checkpoint

9f43e53

Fix package extras strip for sphinx

6093989

Fix unnesessary dependency for docs

ca6e21e

Borda merged commit b0051e8 into Lightning-AI:master Jan 5, 2021

tarepan deleted the feature/4366_non_existing_checkpoint branch January 5, 2021 02:38

tarepan mentioned this pull request Jan 6, 2021

Replace/deprecate SLURM specific checkpointing #5373

Closed

carmocca mentioned this pull request Apr 28, 2021

Changes resume_from_checkpoint warning to error #7075

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add non-existing resume_from_checkpoint acceptance for auto-resubmit #4402

Add non-existing resume_from_checkpoint acceptance for auto-resubmit #4402

tarepan commented Oct 28, 2020 •

edited

Loading

codecov bot commented Oct 28, 2020 •

edited

Loading

tchaton left a comment

tarepan commented Nov 5, 2020

tarepan commented Dec 23, 2020 •

edited

Loading

tarepan commented Jan 2, 2021

awaelchli left a comment

tarepan commented Jan 4, 2021

Add non-existing resume_from_checkpoint acceptance for auto-resubmit #4402

Add non-existing resume_from_checkpoint acceptance for auto-resubmit #4402

Conversation

tarepan commented Oct 28, 2020 • edited Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

codecov bot commented Oct 28, 2020 • edited Loading

Codecov Report

tchaton left a comment

Choose a reason for hiding this comment

tarepan commented Nov 5, 2020

tarepan commented Dec 23, 2020 • edited Loading

Summary

Error

Servey

tarepan commented Jan 2, 2021

awaelchli left a comment

Choose a reason for hiding this comment

tarepan commented Jan 4, 2021

tarepan commented Oct 28, 2020 •

edited

Loading

codecov bot commented Oct 28, 2020 •

edited

Loading

tarepan commented Dec 23, 2020 •

edited

Loading