Merge DDPStrategy and DDPSpawnStrategy in Lite #14952

awaelchli · 2022-09-29T20:26:52Z

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Yes.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @justusschock @awaelchli @carmocca

for more information, see https://pre-commit.ci

…e-spawn

src/lightning_lite/strategies/ddp.py

for more information, see https://pre-commit.ci

…e-spawn

src/lightning_lite/strategies/ddp.py

src/lightning_lite/strategies/launchers/multiprocessing.py

tests/tests_lite/test_connector.py

github-actions · 2022-12-10T00:11:41Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, pytorch, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (windows-2022, pytorch, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py, src/pytorch_lightning/lite/lite.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/pytorch_lightning/lite/lite.py, src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py, src/pytorch_lightning/lite/lite.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py, src/pytorch_lightning/lite/lite.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/pytorch_lightning/lite/lite.py.

🟢 lightning_lite: CPU workflow

Check ID	Status
lite-cpu (macOS-11, lite, 3.8, 1.11)	success	✅
lite-cpu (macOS-11, lite, 3.9, 1.12)	success	✅
lite-cpu (macOS-11, lite, 3.10, 1.13)	success	✅
lite-cpu (macOS-11, lite, 3.7, 1.10, oldest)	success	✅
lite-cpu (ubuntu-20.04, lite, 3.8, 1.10)	success	✅
lite-cpu (ubuntu-20.04, lite, 3.9, 1.11)	success	✅
lite-cpu (ubuntu-20.04, lite, 3.10, 1.12)	success	✅
lite-cpu (ubuntu-20.04, lite, 3.10, 1.13)	success	✅
lite-cpu (ubuntu-20.04, lite, 3.7, 1.10, oldest)	success	✅
lite-cpu (windows-2022, lite, 3.9, 1.11)	success	✅
lite-cpu (windows-2022, lite, 3.10, 1.12)	success	✅
lite-cpu (windows-2022, lite, 3.10, 1.13)	success	✅
lite-cpu (windows-2022, lite, 3.7, 1.10, oldest)	success	✅
lite-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
lite-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
lite-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py, tests/tests_lite/plugins/collectives/test_torch_collective.py, tests/tests_lite/strategies/launchers/test_subprocess_script.py, tests/tests_lite/strategies/test_deepspeed.py, tests/tests_lite/strategies/test_fairscale.py, tests/tests_lite/test_connector.py, tests/tests_lite/test_lite.py, tests/tests_lite/test_parity.py, tests/tests_lite/utilities/test_distributed.py.

🟢 lightning_lite: Azure GPU

Check ID	Status
lightning-lite (GPUs)	success	✅

These checks are required after the changes to src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py, tests/tests_lite/plugins/collectives/test_torch_collective.py, tests/tests_lite/strategies/launchers/test_subprocess_script.py, tests/tests_lite/strategies/test_deepspeed.py, tests/tests_lite/strategies/test_fairscale.py, tests/tests_lite/test_connector.py, tests/tests_lite/test_lite.py, tests/tests_lite/test_parity.py, tests/tests_lite/utilities/test_distributed.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py, src/pytorch_lightning/lite/lite.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.7)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, lite, 3.7)	success	✅
install-pkg (ubuntu-22.04, lite, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.7)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.7)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.7)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.7)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, lite, 3.7)	success	✅
install-pkg (macOS-12, lite, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.7)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.7)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.7)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.7)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, lite, 3.7)	success	✅
install-pkg (windows-2022, lite, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.7)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.7)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.7)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning_lite/connector.py, src/lightning_lite/lite.py, src/lightning_lite/strategies/__init__.py, src/lightning_lite/strategies/ddp.py, src/lightning_lite/strategies/ddp_spawn.py, src/lightning_lite/strategies/fairscale.py, src/lightning_lite/strategies/launchers/multiprocessing.py, src/pytorch_lightning/lite/lite.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

carmocca

Great!

src/lightning_lite/strategies/ddp.py

Co-authored-by: Carlos Mocholí <[email protected]>

(cherry picked from commit fcd3195)

awaelchli added 8 commits September 29, 2022 21:30

merge the two classes

b2a7abc

fix

711e094

debug

94253fe

test updates

a5cdfa1

update

e559d04

updates

9182710

fixes

a7fbc0a

add todo

422adba

awaelchli added fabric lightning.fabric.Fabric refactor labels Sep 29, 2022

github-actions bot added the pl Generic label for PyTorch Lightning package label Sep 29, 2022

awaelchli added code quality and removed pl Generic label for PyTorch Lightning package labels Sep 29, 2022

awaelchli added this to the pl:future milestone Sep 29, 2022

[pre-commit.ci] auto fixes from pre-commit.com hooks

7a9ea0f

for more information, see https://pre-commit.ci

github-actions bot added the pl Generic label for PyTorch Lightning package label Sep 29, 2022

Merge branch 'master' into lite/merge-spawn

8a78f2c

awaelchli mentioned this pull request Oct 16, 2022

Efficient gradient accumulation in LightningLite #14966

Merged

11 tasks

awaelchli added 4 commits November 27, 2022 00:53

Merge branch 'master' into lite/merge-spawn

432e641

fixes

c059eec

fix tests

cebf3c4

mypy

6f5604b

awaelchli added the fun Staff contributions outside working hours - to differentiate from the "community" label label Nov 27, 2022

awaelchli modified the milestones: future, v1.9 Nov 27, 2022

pre-commit-ci bot and others added 5 commits November 27, 2022 00:10

[pre-commit.ci] auto fixes from pre-commit.com hooks

2ccc677

for more information, see https://pre-commit.ci

fix import

c34c894

Merge remote-tracking branch 'origin/lite/merge-spawn' into lite/merg…

c0a83ce

…e-spawn

remove classes

1fbed69

remove tests

90e3c99

launcher

b08ec47

awaelchli commented Nov 27, 2022

View reviewed changes

src/lightning_lite/strategies/ddp.py Outdated Show resolved Hide resolved

pre-commit-ci bot and others added 4 commits November 27, 2022 00:51

[pre-commit.ci] auto fixes from pre-commit.com hooks

e8e859c

for more information, see https://pre-commit.ci

type

a15ff53

Merge remote-tracking branch 'origin/lite/merge-spawn' into lite/merg…

bbd796f

…e-spawn

resolve todo

b022eec

awaelchli mentioned this pull request Dec 7, 2022

Lite: Flatten XLAStrategy #15838

Merged

11 tasks

awaelchli added 3 commits December 9, 2022 23:29

Merge branch 'master' into lite/merge-spawn

1ea413c

resolve todo

c87b18c

update changelog

c5c0359

awaelchli commented Dec 10, 2022

View reviewed changes

awaelchli marked this pull request as ready for review December 10, 2022 00:11

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners December 10, 2022 00:11

carmocca approved these changes Dec 10, 2022

View reviewed changes

src/lightning_lite/strategies/ddp.py Outdated Show resolved Hide resolved

src/lightning_lite/strategies/ddp.py Outdated Show resolved Hide resolved

Apply suggestions from code review

59c1f5b

Co-authored-by: Carlos Mocholí <[email protected]>

justusschock approved these changes Dec 13, 2022

View reviewed changes

mergify bot added the ready PRs ready to be merged label Dec 13, 2022

Merge branch 'master' into lite/merge-spawn

be697be

awaelchli enabled auto-merge (squash) December 13, 2022 08:12

awaelchli merged commit 40596f0 into master Dec 13, 2022

awaelchli deleted the lite/merge-spawn branch December 13, 2022 08:53

awaelchli mentioned this pull request Dec 13, 2022

Fix DDPStrategy import in app framework after #14952 #16029

Merged

11 tasks

awaelchli added a commit that referenced this pull request Dec 13, 2022

Fix DDPStrategy import in app framework after #14952 (#16029)

fcd3195

awaelchli added a commit that referenced this pull request Dec 13, 2022

Fix DDPStrategy import in app framework after #14952 (#16029)

b165057

Borda pushed a commit that referenced this pull request Dec 14, 2022

Fix DDPStrategy import in app framework after #14952 (#16029)

b326fd1

(cherry picked from commit fcd3195)

Borda pushed a commit that referenced this pull request Dec 14, 2022

Fix DDPStrategy import in app framework after #14952 (#16029)

89e14c9

(cherry picked from commit fcd3195)

awaelchli mentioned this pull request Feb 19, 2023

Merge DDPStrategy and DDPSpawnStrategy in PL #16809

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge DDPStrategy and DDPSpawnStrategy in Lite #14952

Merge DDPStrategy and DDPSpawnStrategy in Lite #14952

awaelchli commented Sep 29, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Dec 10, 2022 •

edited

Loading

carmocca left a comment

Merge DDPStrategy and DDPSpawnStrategy in Lite #14952

Merge DDPStrategy and DDPSpawnStrategy in Lite #14952

Conversation

awaelchli commented Sep 29, 2022 • edited by github-actions bot Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

PR review

Did you have fun?

github-actions bot commented Dec 10, 2022 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

carmocca left a comment

Choose a reason for hiding this comment

awaelchli commented Sep 29, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Dec 10, 2022 •

edited

Loading