Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SyncBatchNormPlugin #11754

Merged
merged 36 commits into from
Mar 1, 2022
Merged

Add SyncBatchNormPlugin #11754

merged 36 commits into from
Mar 1, 2022

Conversation

awaelchli
Copy link
Contributor

@awaelchli awaelchli commented Feb 5, 2022 •
edited by github-actions bot
Loading

What does this PR do?

Fixes #4349

  • Added a simple base class for plugins that wrap layers for syncing during forward/backward
  • Added a implementation for native sync batch norm
  • Added tests for all cases of configuration

This decouples the baked-in support for sync batchnorm in ddp- and ddp spawn strategies. This would for example enable adding a plugin for Apex-specific sync batch norm in the future.

TODO:

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @akihironitta

@awaelchli awaelchli added breaking change Includes a breaking change feature Is an improvement or enhancement strategy labels Feb 5, 2022
@awaelchli awaelchli added this to the 1.6 milestone Feb 5, 2022
ananthsub
ananthsub reviewed Feb 5, 2022
View reviewed changes
Copy link
Contributor

@ananthsub ananthsub left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Back when I filed the issue, my reasoning to make this a plugin was to support both pytorch native & Apex: https://nvidia.github.io/apex/_modules/apex/parallel.html#convert_syncbn_model

Almost all instances I've seen have moved to the native version. But i'm sure there are a handful of users who still might use the apex version. Making this an abstract interface will make it easier to support apex later. That could be a good first issue

@awaelchli
Copy link
Contributor Author

Ah interesting! Then we should probably aim for an abstract interface and a concrete default torch implementation. I'll explore it, thanks!

@carmocca carmocca requested a review from ananthsub February 22, 2022 17:23
@mergify mergify bot removed the has conflicts label Feb 25, 2022
@mergify mergify bot removed the has conflicts label Feb 25, 2022
kaushikb11
kaushikb11 approved these changes Feb 25, 2022
View reviewed changes
Copy link
Contributor

@kaushikb11 kaushikb11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

@mergify mergify bot added the ready PRs ready to be merged label Feb 25, 2022
@ananthsub ananthsub mentioned this pull request Feb 26, 2022
Closed
12 tasks
justusschock
justusschock approved these changes Feb 26, 2022
View reviewed changes
Copy link
Member

@justusschock justusschock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, 2 very minor questions :)

pytorch_lightning/plugins/layer_sync.py Show resolved Hide resolved
pytorch_lightning/strategies/parallel.py Show resolved Hide resolved
tchaton
tchaton approved these changes Mar 1, 2022
View reviewed changes
Copy link
Contributor

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

@mergify mergify bot removed the has conflicts label Mar 1, 2022
@awaelchli awaelchli force-pushed the feature/sync-batchnorm-plugin branch from 6265800 to 7947e21 Compare March 1, 2022 11:46
@awaelchli awaelchli enabled auto-merge (squash) March 1, 2022 11:47
@awaelchli awaelchli merged commit d4d1970 into master Mar 1, 2022
@awaelchli awaelchli deleted the feature/sync-batchnorm-plugin branch March 1, 2022 14:11
This was referenced Mar 2, 2022
Closed
Merged
facebook-github-bot pushed a commit to facebookresearch/recipes that referenced this pull request Mar 4, 2022
@tangbinh @facebook-github-bot
Add SyncBatchNormPlugin (#11754)
b0a2def 
Summary:
### New commit log messages
- [d4d197070 Add `SyncBatchNormPlugin` (#11754)](Lightning-AI/pytorch-lightning#11754)

Reviewed By: edward-io

Differential Revision: D34608350

fbshipit-source-id: eae33c6cd276fd02666cf7fb9136b4a7d471bd85
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@carmocca carmocca carmocca approved these changes

@kaushikb11 kaushikb11 kaushikb11 approved these changes

@justusschock justusschock justusschock approved these changes

@tchaton tchaton tchaton approved these changes

@ananthsub ananthsub ananthsub approved these changes

@Borda Borda Awaiting requested review from Borda Borda is a code owner

@SeanNaren SeanNaren Awaiting requested review from SeanNaren

@edenlightning edenlightning Awaiting requested review from edenlightning

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@rohitgr7 rohitgr7 Awaiting requested review from rohitgr7

Assignees
No one assigned
Labels
breaking change Includes a breaking change feature Is an improvement or enhancement ready PRs ready to be merged strategy
Projects
None yet
Milestone
pl:1.6
Development

Successfully merging this pull request may close these issues.

Support sync batch norm as a plugin
6 participants
@awaelchli @ananthsub @carmocca @tchaton @justusschock @kaushikb11