Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix] Resolve bug with multiple optimizers and toggle. #5574

Merged
merged 22 commits into from
Jan 25, 2021
Merged

[bugfix] Resolve bug with multiple optimizers and toggle. #5574

merged 22 commits into from
Jan 25, 2021

Conversation

tchaton
Copy link
Contributor

@tchaton tchaton commented Jan 19, 2021 •
edited
Loading

What does this PR do?

In case of multiple optimizers, toggle optimizer was called.
It would lose user requieres_grad state by setting everything to True or False.

This is erroneous in case there are any parameters which are explicitly kept non-trainable.

Fixes #5292 #5642

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified
  • Check that target branch and milestone match!

Did you have fun?

Make sure you had fun coding 🙃

@pep8speaks
Copy link

pep8speaks commented Jan 19, 2021 •
edited
Loading

Hello @tchaton! Thanks for updating this PR.

Line 183:121: E501 line too long (150 > 120 characters)

Comment last updated at 2021-01-25 20:48:57 UTC

@tchaton tchaton self-assigned this Jan 19, 2021
@tchaton tchaton added this to the 1.1.x milestone Jan 19, 2021
@tchaton tchaton added bug Something isn't working priority: 1 Medium priority task labels Jan 19, 2021
@tchaton tchaton mentioned this pull request Jan 19, 2021
Closed
12 tasks
@carmocca carmocca self-requested a review January 19, 2021 15:05
rohitgr7
rohitgr7 approved these changes Jan 19, 2021
View reviewed changes
pytorch_lightning/core/lightning.py
@@ -1166,16 +1166,44 @@ def toggle_optimizer(self, optimizer: Optimizer, optimizer_idx: int):

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just use the opt_idx to get the optimizer down there... just to unify the arguments with untoggle_optimizer. Either way is fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds great ! I will do so.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan to do it in this PR?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tchaton ^^

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rohitgr7 @carmocca @tchaton please see #5775

@codecov
Copy link

codecov bot commented Jan 19, 2021 •
edited
Loading

Codecov Report

Merging #5574 (9f11c76) into master (e87424a) will decrease coverage by 0%.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #5574   +/-   ##
======================================
- Coverage      93%     93%   -0%     
======================================
  Files         135     135           
  Lines       10034   10051   +17     
======================================
+ Hits         9369    9383   +14     
- Misses        665     668    +3

carmocca
carmocca reviewed Jan 19, 2021
View reviewed changes
@tchaton tchaton enabled auto-merge (squash) January 21, 2021 08:13
SeanNaren
SeanNaren reviewed Jan 21, 2021
View reviewed changes
@tchaton tchaton added the ready PRs ready to be merged label Jan 25, 2021
@mergify mergify bot removed the has conflicts label Jan 25, 2021
@tchaton tchaton merged commit c76cc23 into master Jan 25, 2021
@tchaton tchaton deleted the bugfix/toggle branch January 25, 2021 23:45
Borda pushed a commit that referenced this pull request Feb 4, 2021
@tchaton @Borda
[bugfix] Resolve bug with multiple optimizers and toggle. (#5574)
8753cf9 
* fix toggle_optimizer

* update doc

* resolve bug

* update

* Update pytorch_lightning/core/lightning.py

Co-authored-by: Rohit Gupta <[email protected]>

* update on comments

* update on comments

* update

Co-authored-by: Rohit Gupta <[email protected]>

(cherry picked from commit c76cc23)
Borda pushed a commit that referenced this pull request Feb 4, 2021
@tchaton @Borda
[bugfix] Resolve bug with multiple optimizers and toggle. (#5574)
02620f3 
* fix toggle_optimizer

* update doc

* resolve bug

* update

* Update pytorch_lightning/core/lightning.py

Co-authored-by: Rohit Gupta <[email protected]>

* update on comments

* update on comments

* update

Co-authored-by: Rohit Gupta <[email protected]>

(cherry picked from commit c76cc23)
Borda pushed a commit that referenced this pull request Feb 4, 2021
@tchaton @Borda
[bugfix] Resolve bug with multiple optimizers and toggle. (#5574)
f8c7548 
* fix toggle_optimizer

* update doc

* resolve bug

* update

* Update pytorch_lightning/core/lightning.py

Co-authored-by: Rohit Gupta <[email protected]>

* update on comments

* update on comments

* update

Co-authored-by: Rohit Gupta <[email protected]>

(cherry picked from commit c76cc23)
Borda pushed a commit that referenced this pull request Feb 4, 2021
@tchaton @Borda
[bugfix] Resolve bug with multiple optimizers and toggle. (#5574)
cc38d4c 
* fix toggle_optimizer

* update doc

* resolve bug

* update

* Update pytorch_lightning/core/lightning.py

Co-authored-by: Rohit Gupta <[email protected]>

* update on comments

* update on comments

* update

Co-authored-by: Rohit Gupta <[email protected]>

(cherry picked from commit c76cc23)
@phammanhhiep phammanhhiep mentioned this pull request Jul 5, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ananthsub ananthsub ananthsub left review comments

@Borda Borda Borda left review comments

@jain-anshul jain-anshul jain-anshul left review comments

@rohitgr7 rohitgr7 rohitgr7 approved these changes

@williamFalcon williamFalcon williamFalcon approved these changes

@SeanNaren SeanNaren SeanNaren approved these changes

@carmocca carmocca carmocca approved these changes

@justusschock justusschock justusschock approved these changes

@awaelchli awaelchli Awaiting requested review from awaelchli

Assignees

@tchaton tchaton

Labels
bug Something isn't working priority: 1 Medium priority task ready PRs ready to be merged
Projects
None yet
Milestone
pl:1.1.x
Development

Successfully merging this pull request may close these issues.

For multiple optimizers, model.toggle_optimizer() is setting the requires_grad=True for all the params.
10 participants
@tchaton @pep8speaks @ananthsub @williamFalcon @Borda @SeanNaren @carmocca @jain-anshul @justusschock @rohitgr7