Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pass enabled down to _BackwardSyncControl #19577

Merged
merged 5 commits into from
Mar 8, 2024
Merged

Pass enabled down to _BackwardSyncControl #19577

merged 5 commits into from
Mar 8, 2024

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Mar 5, 2024 •
edited by github-actions bot
Loading

What does this PR do?

The strategy's BackwardSyncControl implementation might want to run custom code when gradient accumulation is no longer enabled. This PR allows it.

📚 Documentation preview 📚: https://pytorch-lightning--19577.org.readthedocs.build/en/19577/

cc @Borda @carmocca @justusschock @awaelchli

@carmocca carmocca added feature Is an improvement or enhancement fabric lightning.fabric.Fabric strategy labels Mar 5, 2024
@carmocca carmocca added this to the 2.3 milestone Mar 5, 2024
@carmocca carmocca self-assigned this Mar 5, 2024
@carmocca carmocca marked this pull request as ready for review March 5, 2024 17:34
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 5, 2024 •
edited
Loading

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.13, oldest) success ✅
pl-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
pl-cpu (macOS-11, lightning, 3.10, 2.1) success ✅
pl-cpu (macOS-11, lightning, 3.10, 2.2) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13, oldest) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2) success ✅
pl-cpu (windows-2022, lightning, 3.8, 1.13, oldest) success ✅
pl-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
pl-cpu (windows-2022, lightning, 3.10, 2.1) success ✅
pl-cpu (windows-2022, lightning, 3.10, 2.2) success ✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success ✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success ✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0) success ✅
pl-cpu (macOS-12, pytorch, 3.11, 2.1) success ✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0) success ✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1) success ✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0) success ✅
pl-cpu (windows-2022, pytorch, 3.11, 2.1) success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) (testing Lightning | latest) success ✅
pytorch-lightning (GPUs) (testing PyTorch | latest) success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 pytorch_lightning: Benchmarks
Check ID Status
lightning.Benchmarks success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 fabric: Docs
Check ID Status
docs-make (fabric, doctest) success ✅
docs-make (fabric, html) success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, lightning, 3.8, 1.13, oldest) success ✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
fabric-cpu (macOS-11, lightning, 3.11, 2.1) success ✅
fabric-cpu (macOS-11, lightning, 3.11, 2.2) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.13, oldest) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2) success ✅
fabric-cpu (windows-2022, lightning, 3.8, 1.13, oldest) success ✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
fabric-cpu (windows-2022, lightning, 3.11, 2.1) success ✅
fabric-cpu (windows-2022, lightning, 3.11, 2.2) success ✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13) success ✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13) success ✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0) success ✅
fabric-cpu (macOS-12, fabric, 3.11, 2.1) success ✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0) success ✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1) success ✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0) success ✅
fabric-cpu (windows-2022, fabric, 3.11, 2.1) success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py, tests/tests_fabric/strategies/test_ddp.py, tests/tests_fabric/strategies/test_fsdp.py, tests/tests_fabric/strategies/test_xla_fsdp.py, tests/tests_fabric/test_fabric.py.

🟢 lightning_fabric: Azure GPU
Check ID Status
lightning-fabric (GPUs) (testing Fabric | latest) success ✅
lightning-fabric (GPUs) (testing Lightning | latest) success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py, tests/tests_fabric/strategies/test_ddp.py, tests/tests_fabric/strategies/test_fsdp.py, tests/tests_fabric/strategies/test_xla_fsdp.py, tests/tests_fabric/test_fabric.py.

🟢 mypy
Check ID Status
mypy success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success ✅
install-pkg (ubuntu-22.04, app, 3.11) success ✅
install-pkg (ubuntu-22.04, fabric, 3.8) success ✅
install-pkg (ubuntu-22.04, fabric, 3.11) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.8) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.11) success ✅
install-pkg (ubuntu-22.04, lightning, 3.8) success ✅
install-pkg (ubuntu-22.04, lightning, 3.11) success ✅
install-pkg (ubuntu-22.04, notset, 3.8) success ✅
install-pkg (ubuntu-22.04, notset, 3.11) success ✅
install-pkg (macOS-12, app, 3.8) success ✅
install-pkg (macOS-12, app, 3.11) success ✅
install-pkg (macOS-12, fabric, 3.8) success ✅
install-pkg (macOS-12, fabric, 3.11) success ✅
install-pkg (macOS-12, pytorch, 3.8) success ✅
install-pkg (macOS-12, pytorch, 3.11) success ✅
install-pkg (macOS-12, lightning, 3.8) success ✅
install-pkg (macOS-12, lightning, 3.11) success ✅
install-pkg (macOS-12, notset, 3.8) success ✅
install-pkg (macOS-12, notset, 3.11) success ✅
install-pkg (windows-2022, app, 3.8) success ✅
install-pkg (windows-2022, app, 3.11) success ✅
install-pkg (windows-2022, fabric, 3.8) success ✅
install-pkg (windows-2022, fabric, 3.11) success ✅
install-pkg (windows-2022, pytorch, 3.8) success ✅
install-pkg (windows-2022, pytorch, 3.11) success ✅
install-pkg (windows-2022, lightning, 3.8) success ✅
install-pkg (windows-2022, lightning, 3.11) success ✅
install-pkg (windows-2022, notset, 3.8) success ✅
install-pkg (windows-2022, notset, 3.11) success ✅

These checks are required after the changes to src/lightning/fabric/fabric.py, src/lightning/fabric/strategies/ddp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/strategy.py, src/lightning/fabric/strategies/xla_fsdp.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@codecov Codecov
Copy link

codecov bot commented Mar 5, 2024 •
edited
Loading

Codecov Report

Merging #19577 (395a044) into master (f23b3b1) will decrease coverage by 25%.
Report is 2 commits behind head on master.
The diff coverage is 100%.

Additional details and impacted files 
@@            Coverage Diff            @@
##           master   #19577     +/-   ##
=========================================
- Coverage      84%      59%    -25%     
=========================================
  Files         424      419      -5     
  Lines       34897    34805     -92     
=========================================
- Hits        29295    20496   -8799     
- Misses       5602    14309   +8707

@carmocca carmocca requested a review from Borda March 6, 2024 01:16
@mergify mergify bot added the ready PRs ready to be merged label Mar 8, 2024
@carmocca carmocca merged commit 06eb3cc into master Mar 8, 2024
114 of 115 checks passed
@carmocca carmocca deleted the carmocca/bwd-sync-enabled branch March 8, 2024 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Borda Borda Borda approved these changes

@awaelchli awaelchli awaelchli approved these changes

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

Assignees

@carmocca carmocca

Labels
fabric lightning.fabric.Fabric feature Is an improvement or enhancement ready PRs ready to be merged run TPU strategy
Projects
None yet
Milestone
2.3
Development

Successfully merging this pull request may close these issues.
3 participants
@carmocca @awaelchli @Borda