[sharded plugin] Fix check for fp16 precision #7825

shuyingsunshine21 · 2021-06-03T19:47:20Z

What does this PR do?

Fixes the bug, precision passed in trainer is 16, but PrecisionPlugin will convert this to other format, like MixedPrecisionPlugin.precision is mixed (https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/plugins/precision/mixed.py)

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

…lightning pull latest code

…oint_consolidate Update test_all_gather_grad.py

This reverts commit 9d4a2b8.

…1-checkpoint_consolidate" This reverts commit c5053da, reversing changes made to 0d23d75.

This reverts commit 0d23d75.

This reverts commit 70fe5da.

This reverts commit a9aae99.

This reverts commit ea74906.

This reverts commit bf70e43.

This reverts commit f172101.

This reverts commit 536c132.

This reverts commit 3a9fde9.

This reverts commit 7a369f4.

…lightning

This reverts commit 8222dc9.

This reverts commit 6c095b2.

This reverts commit 250d0aa.

This reverts commit 8651d54.

This reverts commit dcdcd29.

…-lightning

ananthsub · 2021-06-03T20:28:17Z

pytorch_lightning/plugins/training_type/sharded.py

@@ -54,7 +54,7 @@ def _reinit_optimizers_with_oss(self):
                optim_class = type(optimizer)
                zero_optimizer = OSS(params=optimizer.param_groups, optim=optim_class, **optimizer.defaults)
                if _FAIRSCALE_OSS_FP16_BROADCAST_AVAILABLE:
-                    is_fp16 = self.lightning_module.trainer.precision == 16
+                    is_fp16 = self.lightning_module.trainer.precision == "mixed"


Suggested change

is_fp16 = self.lightning_module.trainer.precision == "mixed"

precision = self.lightning_module.trainer.precision

is_fp16 = precision == "mixed" or precision == 16

just in case we enable true fp16 later

pytorch_lightning/plugins/training_type/sharded.py

…e21/pytorch-lightning into fix_precision_bug

shuyingsunshine21 · 2021-06-04T05:54:55Z

would like to add unittest for assert the wrapped optimizer's attribute broadcast_fp16 is set correctly, but realized that there might have difficulty for multi-nodes setting.

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: ananthsub <[email protected]> (cherry picked from commit ca89a7f)

Shuying Sun and others added 30 commits March 23, 2021 12:06

Fix some test errors

89f284d

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…

80cfbff

…lightning pull latest code

checkpoint consolidation

536c132

Update ddp_spawn.py

f172101

Update test_metric_result_integration.py

bf70e43

Update test_results.py

ea74906

Update utils.py

a9aae99

Update utils.py

70fe5da

Update test_all_gather_grad.py

0d23d75

Update test_all_gather_grad.py

ca6f98b

Merge pull request #1 from shuyingsunshine21/shuyingsunshine21-checkp…

c5053da

…oint_consolidate Update test_all_gather_grad.py

Update test_results.py

9d4a2b8

Revert "Update test_results.py"

7635b4f

This reverts commit 9d4a2b8.

Revert "Merge pull request #1 from shuyingsunshine21/shuyingsunshine2…

d64f90c

…1-checkpoint_consolidate" This reverts commit c5053da, reversing changes made to 0d23d75.

Revert "Update test_all_gather_grad.py"

dcdcd29

This reverts commit 0d23d75.

Revert "Update utils.py"

8651d54

This reverts commit 70fe5da.

Revert "Update utils.py"

15f4b9e

This reverts commit a9aae99.

Revert "Update test_results.py"

250d0aa

This reverts commit ea74906.

Revert "Update test_metric_result_integration.py"

6c095b2

This reverts commit bf70e43.

Revert "Update ddp_spawn.py"

8222dc9

This reverts commit f172101.

Revert "checkpoint consolidation"

3a9fde9

This reverts commit 536c132.

Revert "Revert "checkpoint consolidation""

7a369f4

This reverts commit 3a9fde9.

Revert "Revert "Revert "checkpoint consolidation"""

b4a0b9e

This reverts commit 7a369f4.

Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…

5cf1db1

…lightning

Revert "Revert "Update ddp_spawn.py""

0ce7e05

This reverts commit 8222dc9.

Revert "Revert "Update test_metric_result_integration.py""

fe9736d

This reverts commit 6c095b2.

Revert "Revert "Update test_results.py""

c314ef6

This reverts commit 250d0aa.

Revert "Revert "Update utils.py""

c3feda0

This reverts commit 8651d54.

Revert "Revert "Update test_all_gather_grad.py""

c759477

This reverts commit dcdcd29.

Merge branch 'master' of https://github.com/shuyingsunshine21/pytorch…

7a8e540

…-lightning

ananthsub reviewed Jun 3, 2021

View reviewed changes

ananthsub changed the title ~~Fix precision bug~~ [sharded plugin] Fix check for is fp16 precision Jun 3, 2021

ananthsub changed the title ~~[sharded plugin] Fix check for is fp16 precision~~ [sharded plugin] Fix check for fp16 precision Jun 3, 2021

Shuying Sun added 2 commits June 3, 2021 14:03

comments

749e277

add changlog

6597fec

shuyingsunshine21 requested review from Borda, carmocca, kaushikb11 and williamFalcon as code owners June 3, 2021 21:33

ananthsub approved these changes Jun 4, 2021

View reviewed changes

pytorch_lightning/plugins/training_type/sharded.py Outdated Show resolved Hide resolved

justusschock approved these changes Jun 4, 2021

View reviewed changes

Update pytorch_lightning/plugins/training_type/sharded.py

7d628cb

ananthsub added the ready PRs ready to be merged label Jun 4, 2021

justusschock enabled auto-merge (squash) June 4, 2021 05:42

mergify bot added the has conflicts label Jun 4, 2021

Shuying Sun added 3 commits June 3, 2021 22:48

comment

f8138aa

Merge branch 'fix_precision_bug' of https://github.com/shuyingsunshin…

a82266e

…e21/pytorch-lightning into fix_precision_bug

rebase to resolve conflict

07a6207

auto-merge was automatically disabled June 4, 2021 05:52
Head branch was pushed to by a user without write access

mergify bot removed the has conflicts label Jun 4, 2021

awaelchli approved these changes Jun 4, 2021

View reviewed changes

awaelchli added the bug Something isn't working label Jun 4, 2021

awaelchli added this to the v1.3.x milestone Jun 4, 2021

awaelchli merged commit ca89a7f into Lightning-AI:master Jun 4, 2021

shuyingsunshine21 deleted the fix_precision_bug branch June 4, 2021 07:20

ananthsub mentioned this pull request Jun 7, 2021

IPU Integration 5/5 #7867

Merged

11 tasks

SeanNaren mentioned this pull request Jun 8, 2021

1.3.x cherries 🍒 #7467

Closed

SeanNaren pushed a commit that referenced this pull request Jun 8, 2021

[sharded plugin] Fix check for fp16 precision (#7825)

db77223

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: ananthsub <[email protected]> (cherry picked from commit ca89a7f)

lexierule pushed a commit that referenced this pull request Jun 9, 2021

[sharded plugin] Fix check for fp16 precision (#7825)

55f52a1

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: ananthsub <[email protected]> (cherry picked from commit ca89a7f)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sharded plugin] Fix check for fp16 precision #7825

[sharded plugin] Fix check for fp16 precision #7825

shuyingsunshine21 commented Jun 3, 2021 •

edited

Loading

ananthsub Jun 3, 2021

shuyingsunshine21 commented Jun 4, 2021

	is_fp16 = self.lightning_module.trainer.precision == "mixed"
	precision = self.lightning_module.trainer.precision
	is_fp16 = precision == "mixed" or precision == 16

[sharded plugin] Fix check for fp16 precision #7825

[sharded plugin] Fix check for fp16 precision #7825

Conversation

shuyingsunshine21 commented Jun 3, 2021 • edited Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

ananthsub Jun 3, 2021

Choose a reason for hiding this comment

shuyingsunshine21 commented Jun 4, 2021

shuyingsunshine21 commented Jun 3, 2021 •

edited

Loading