Set `find_unused_parameters=False` as the default #16611

awaelchli · 2023-02-02T14:53:24Z

What does this PR do?

Fixes #7330
Fixes #14486
Closes #12445
Part of #12398

The default value for DDP's find_unused_parameters in PyTorch is False. So far, Lightning has always flipped that to True because it tried to minimize the confusion of users who would otherwise see an error from PyTorch:

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 0 1
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error

This error would often occur when users dealt multiple optimizers (GAN) etc. After #16539, this issue is less significant. However, the drawback of setting find_unused_parameters is that you compromise on performance. PyTorch shows a warning:

Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

But obviously, the warning is pointing at the PyTorch DDP API that the Lightning user doesn't have access to. Thus, the warning is not very helpful.

Since Trainer 2.0 favors performance and parity with PyTorch, we will switch find_unused_parameters to False to align with PyTorch's defaults. Discussion from the past: #6219

Does your PR introduce any breaking changes? If yes, please list them.

In summary, these are the possible configurations a user can run into:

Case 1: User has unused parameters in their forward-backward

Trainer < 2.0: No errors, no warnings, expected performance hit
Trainer >= 2.0: User gets an error unless they set strategy=ddp_find_unused_parameters=True.

Case 2: User has no unused parameters in their forward-backward

Trainer < 2.0: User got a PyTorch warning because ddp_find_unused_parameters=True was the default, unexpected performance hit
Trainer >= 2.0: No errors, no warnings, no performance hit

A breaking change occurs for users of Case 1 when switching to >= 2.0.
The rest of users see one less warning and a speed up in their training.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @justusschock @awaelchli

for more information, see https://pre-commit.ci

…' into feature/find-unused-params-false

src/lightning/pytorch/strategies/hpu_parallel.py

github-actions · 2023-02-06T12:02:12Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (slow, macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (slow, windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/hpu_parallel.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/benchmarks/test_sync_batchnorm_parity.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/strategies/test_ddp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/hpu_parallel.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/benchmarks/test_sync_batchnorm_parity.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/strategies/test_ddp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
pytorch-lightning.Benchmark	success	✅

These checks are required after the changes to tests/tests_pytorch/benchmarks/test_sync_batchnorm_parity.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/hpu_parallel.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/benchmarks/test_sync_batchnorm_parity.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/strategies/test_ddp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/hpu_parallel.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/benchmarks/test_sync_batchnorm_parity.py, tests/tests_pytorch/callbacks/test_stochastic_weight_avg.py, tests/tests_pytorch/strategies/test_ddp.py, tests/tests_pytorch/strategies/test_ddp_spawn_strategy.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/hpu_parallel.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, docs/source-pytorch/advanced/model_parallel.rst, docs/source-pytorch/advanced/strategy_registry.rst, docs/source-pytorch/extensions/strategy.rst.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/hpu_parallel.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/ddp_spawn.py, src/lightning/pytorch/strategies/hpu_parallel.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

src/lightning/pytorch/strategies/hpu_parallel.py

awaelchli added 2 commits February 2, 2023 14:06

switch the default

b98717d

update

19fc989

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 2, 2023

awaelchli added breaking change Includes a breaking change strategy: ddp DistributedDataParallel and removed pl Generic label for PyTorch Lightning package labels Feb 2, 2023

awaelchli added this to the 2.0 milestone Feb 2, 2023

awaelchli self-assigned this Feb 2, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

898e796

for more information, see https://pre-commit.ci

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 2, 2023

awaelchli added 4 commits February 2, 2023 15:54

changelog

d54d880

Merge remote-tracking branch 'origin/feature/find-unused-params-false…

ec101ed

…' into feature/find-unused-params-false

changelog

c41a32f

fix

0c57218

coreyjadams mentioned this pull request Feb 2, 2023

Add MPI cluster environment #16570

Merged

15 tasks

set true

3b6ee90

awaelchli commented Feb 2, 2023

View reviewed changes

src/lightning/pytorch/strategies/hpu_parallel.py Outdated Show resolved Hide resolved

awaelchli added 2 commits February 6, 2023 13:01

include hpu

91ffad3

Merge branch 'master' into feature/find-unused-params-false

e7eb409

awaelchli marked this pull request as ready for review February 6, 2023 12:01

awaelchli requested review from carmocca, justusschock, williamFalcon, edenlightning, lantiga and Borda as code owners February 6, 2023 12:01

missed removal

a66db33

awaelchli commented Feb 6, 2023

View reviewed changes

src/lightning/pytorch/strategies/hpu_parallel.py Show resolved Hide resolved

awaelchli mentioned this pull request Feb 6, 2023

Add Strategy.on_exception #16646

Merged

11 tasks

carmocca approved these changes Feb 6, 2023

View reviewed changes

justusschock approved these changes Feb 6, 2023

View reviewed changes

justusschock enabled auto-merge (squash) February 6, 2023 15:45

mergify bot added the ready PRs ready to be merged label Feb 6, 2023

Borda approved these changes Feb 6, 2023

View reviewed changes

justusschock merged commit cd0eedb into master Feb 6, 2023

justusschock deleted the feature/find-unused-params-false branch February 6, 2023 15:51

athitten mentioned this pull request Aug 2, 2023

Upgrade to pytorch lightning 2.0 NVIDIA/NeMo#6433

Merged

8 tasks

zhiqiangdon mentioned this pull request Aug 8, 2023

[AutoMM] Support muti-GPU inference with ddp autogluon/autogluon#3445

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set `find_unused_parameters=False` as the default #16611

Set `find_unused_parameters=False` as the default #16611

awaelchli commented Feb 2, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Feb 6, 2023 •

edited

Loading

Set find_unused_parameters=False as the default #16611

Set find_unused_parameters=False as the default #16611

Conversation

awaelchli commented Feb 2, 2023 • edited by github-actions bot Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

PR review

Did you have fun?

github-actions bot commented Feb 6, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

Set `find_unused_parameters=False` as the default #16611

Set `find_unused_parameters=False` as the default #16611

awaelchli commented Feb 2, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Feb 6, 2023 •

edited

Loading