Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

graceful training cleanup after Keyboard Interrupt #856

Merged
merged 6 commits into from
Feb 22, 2020
Merged

graceful training cleanup after Keyboard Interrupt #856

merged 6 commits into from
Feb 22, 2020

Conversation

jeremyjordan
Copy link
Contributor

@jeremyjordan jeremyjordan commented Feb 15, 2020 •
edited
Loading

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • Did you read the contributor guideline?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #795

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

@jeremyjordan
Copy link
Contributor Author

We should be able to close #657 as well

@Borda Borda added the feature Is an improvement or enhancement label Feb 15, 2020
@Borda Borda added this to the 0.6.1 milestone Feb 15, 2020
Borda
Borda requested changes Feb 15, 2020
View reviewed changes
Copy link
Member

@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall the run_training_teardown() be called also after training?

pytorch_lightning/trainer/trainer.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
@jeremyjordan
Copy link
Contributor Author

@Borda updated to fix merge conflicts with master. were there any more changes requested?

Borda
Borda approved these changes Feb 17, 2020
View reviewed changes
Copy link
Member

@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@Borda
Copy link
Member

Borda commented Feb 17, 2020

@jeremyjordan just fixed another conflict, pls check it...

@jeremyjordan
Copy link
Contributor Author

@Borda looks good! thanks for addressing that

@Borda Borda added the ready PRs ready to be merged label Feb 17, 2020
@Borda Borda requested a review from williamFalcon February 17, 2020 22:16
@Borda Borda mentioned this pull request Feb 18, 2020
Closed
@williamFalcon
Copy link
Contributor

@jeremyjordan rebase?

@jeremyjordan
Copy link
Contributor Author

jeremyjordan commented Feb 22, 2020 •
edited
Loading

@williamFalcon rebased and ready for review! unfortunately the diff renders poorly

@williamFalcon williamFalcon merged commit e05586c into Lightning-AI:master Feb 22, 2020
@ethanwharris ethanwharris mentioned this pull request Feb 25, 2020
Closed
4 tasks
@Borda Borda changed the title graceful training cleanup after KeyboardInterrupt graceful training cleanup after Keyboard Interrupt Mar 8, 2020
tullie pushed a commit to tullie/pytorch-lightning that referenced this pull request Apr 3, 2020
@jeremyjordan @Borda @tullie
extract training teardown into method, catch KeyboardInterrupt (Light…
e2a20f0 
…ning-AI#856)

Co-authored-by: Jirka Borovec <[email protected]>
This was referenced Sep 2, 2021
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Borda Borda Borda approved these changes

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

Assignees
No one assigned
Labels
feature Is an improvement or enhancement ready PRs ready to be merged
Projects
None yet
Milestone
pl:0.7
Development

Successfully merging this pull request may close these issues.

support graceful training cleanup after Keyboard Interrupt
3 participants
@jeremyjordan @Borda @williamFalcon