Add `Fabric.all_reduce` #16459

awaelchli · 2023-01-21T02:39:07Z

What does this PR do?

The Fabric object already has broadcast, barrier and all_gather exposed. This PR adds all_reduce.
This completes the exposure of the most commonly used collectives, conveniently on the fabric object.

For more specialized collectives, the user will be advised to use the new Collectives API in the future, which exposes a much larger set of functions.

Follow up to #16373

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @carmocca @justusschock @awaelchli

for more information, see https://pre-commit.ci

docs/source-pytorch/fabric/api/fabric_methods.rst

github-actions · 2023-01-21T02:44:24Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, pytorch, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (windows-2022, pytorch, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to docs/source-pytorch/fabric/advanced/distributed_communication.rst, docs/source-pytorch/fabric/api/fabric_methods.rst.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, fabric, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, fabric, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, fabric, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, fabric, 3.7, 1.10, oldest)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.10)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.9, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.10, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.7, 1.10, oldest)	success	✅
fabric-cpu (windows-2022, fabric, 3.9, 1.11)	success	✅
fabric-cpu (windows-2022, fabric, 3.10, 1.12)	success	✅
fabric-cpu (windows-2022, fabric, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.7, 1.10, oldest)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py, tests/tests_fabric/test_fabric.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py, tests/tests_fabric/test_fabric.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.7)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.7)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.7)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.7)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.7)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.7)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.7)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.7)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.7)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.7)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.7)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.7)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.7)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.7)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.7)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning_fabric/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

awaelchli added 4 commits January 20, 2023 23:54

collectives

d9e9297

documentation

5df2688

reduce docs

f4f996d

reset

4365096

github-actions bot added the pl Generic label for PyTorch Lightning package label Jan 21, 2023

awaelchli added fabric lightning.fabric.Fabric feature Is an improvement or enhancement docs Documentation related and removed pl Generic label for PyTorch Lightning package labels Jan 21, 2023

awaelchli added this to the 2.0 milestone Jan 21, 2023

awaelchli self-assigned this Jan 21, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

ec7f739

for more information, see https://pre-commit.ci

github-actions bot added the pl Generic label for PyTorch Lightning package label Jan 21, 2023

changelog

1a867c6

awaelchli commented Jan 21, 2023

View reviewed changes

docs/source-pytorch/fabric/api/fabric_methods.rst Show resolved Hide resolved

awaelchli marked this pull request as ready for review January 21, 2023 02:43

awaelchli requested review from carmocca, justusschock, edenlightning, lantiga and Borda as code owners January 21, 2023 02:43

awaelchli added the fun Staff contributions outside working hours - to differentiate from the "community" label label Jan 21, 2023

justusschock approved these changes Jan 23, 2023

View reviewed changes

carmocca approved these changes Jan 24, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Jan 24, 2023

Borda approved these changes Jan 24, 2023

View reviewed changes

mergify bot added has conflicts and removed ready PRs ready to be merged labels Jan 24, 2023

Merge branch 'master' into fabric/all-reduce

024fb73

awaelchli enabled auto-merge (squash) January 24, 2023 18:12

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Jan 24, 2023

awaelchli merged commit c87bb71 into master Jan 24, 2023

awaelchli deleted the fabric/all-reduce branch January 24, 2023 22:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Fabric.all_reduce` #16459

Add `Fabric.all_reduce` #16459

awaelchli commented Jan 21, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 21, 2023 •

edited

Loading

Add Fabric.all_reduce #16459

Add Fabric.all_reduce #16459

Conversation

awaelchli commented Jan 21, 2023 • edited by github-actions bot Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

github-actions bot commented Jan 21, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

Add `Fabric.all_reduce` #16459

Add `Fabric.all_reduce` #16459

awaelchli commented Jan 21, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 21, 2023 •

edited

Loading