Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TPU] Support setting the XLAFSDP policy with a set #18430

Merged
merged 9 commits into from
Aug 30, 2023
Merged

[TPU] Support setting the XLAFSDP policy with a set #18430

merged 9 commits into from
Aug 30, 2023

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Aug 29, 2023 •
edited by github-actions bot
Loading

What does this PR do?

Just as regular FSDP allows

cc @Borda @carmocca @justusschock @awaelchli @JackCaoG @steventk-g @Liyang90

@carmocca carmocca added feature Is an improvement or enhancement fabric lightning.fabric.Fabric strategy: xla labels Aug 29, 2023
@carmocca carmocca added this to the 2.1 milestone Aug 29, 2023
@carmocca carmocca self-assigned this Aug 29, 2023
@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label Aug 29, 2023
@carmocca carmocca mentioned this pull request Aug 29, 2023
Merged
@carmocca carmocca force-pushed the carmocca/xlafsdp-api branch from 9e06b8e to 601c5e9 Compare August 29, 2023 23:40
@carmocca carmocca marked this pull request as ready for review August 29, 2023 23:55
@github-actions
Copy link
Contributor

github-actions bot commented Aug 29, 2023 •
edited
Loading

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.11) success ✅
pl-cpu (macOS-11, lightning, 3.9, 1.12) success ✅
pl-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
pl-cpu (macOS-11, lightning, 3.10, 2.0) success ✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest) success ✅
pl-cpu (windows-2022, lightning, 3.8, 1.11) success ✅
pl-cpu (windows-2022, lightning, 3.9, 1.12) success ✅
pl-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
pl-cpu (windows-2022, lightning, 3.10, 2.0) success ✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest) success ✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success ✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success ✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0) success ✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0) success ✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0) success ✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/strategies/fsdp.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
[pytorch-lightning (GPUs) (testing Lightning latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=172189&view=logs&jobId=47e66f3c-897a-5428-da11-bf5c7745762e) success
[pytorch-lightning (GPUs) (testing PyTorch latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=172189&view=logs&jobId=3f274fac-2e11-54ca-487e-194c91f3ae9f) success

These checks are required after the changes to src/lightning/pytorch/strategies/fsdp.py, src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 pytorch_lightning: Benchmarks
Check ID Status
lightning.Benchmarks success ✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/strategies/fsdp.py.

🟢 fabric: Docs
Check ID Status
docs-make (fabric, doctest) success ✅
docs-make (fabric, html) success ✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 pytorch_lightning: Docs
Check ID Status
docs-make (pytorch, doctest) success ✅
docs-make (pytorch, html) success ✅

These checks are required after the changes to src/lightning/pytorch/strategies/fsdp.py.

🟢 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11) success ✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12) success ✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0) success ✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest) success ✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11) success ✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12) success ✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0) success ✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest) success ✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13) success ✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13) success ✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0) success ✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0) success ✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0) success ✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py, tests/tests_fabric/conftest.py, tests/tests_fabric/strategies/test_xla_fsdp.py, tests/tests_fabric/strategies/test_xla_fsdp_integration.py.

🟢 lightning_fabric: Azure GPU
Check ID Status
[lightning-fabric (GPUs) (testing Fabric latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=172191&view=logs&jobId=3f274fac-2e11-54ca-487e-194c91f3ae9f) success
[lightning-fabric (GPUs) (testing Lightning latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=172191&view=logs&jobId=47e66f3c-897a-5428-da11-bf5c7745762e) success

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py, tests/tests_fabric/conftest.py, tests/tests_fabric/strategies/test_xla_fsdp.py, tests/tests_fabric/strategies/test_xla_fsdp_integration.py.

🟢 mypy
Check ID Status
mypy success ✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/strategies/fsdp.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success ✅
install-pkg (ubuntu-22.04, app, 3.11) success ✅
install-pkg (ubuntu-22.04, fabric, 3.8) success ✅
install-pkg (ubuntu-22.04, fabric, 3.11) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.8) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.11) success ✅
install-pkg (ubuntu-22.04, lightning, 3.8) success ✅
install-pkg (ubuntu-22.04, lightning, 3.11) success ✅
install-pkg (ubuntu-22.04, notset, 3.8) success ✅
install-pkg (ubuntu-22.04, notset, 3.11) success ✅
install-pkg (macOS-12, app, 3.8) success ✅
install-pkg (macOS-12, app, 3.11) success ✅
install-pkg (macOS-12, fabric, 3.8) success ✅
install-pkg (macOS-12, fabric, 3.11) success ✅
install-pkg (macOS-12, pytorch, 3.8) success ✅
install-pkg (macOS-12, pytorch, 3.11) success ✅
install-pkg (macOS-12, lightning, 3.8) success ✅
install-pkg (macOS-12, lightning, 3.11) success ✅
install-pkg (macOS-12, notset, 3.8) success ✅
install-pkg (macOS-12, notset, 3.11) success ✅
install-pkg (windows-2022, app, 3.8) success ✅
install-pkg (windows-2022, app, 3.11) success ✅
install-pkg (windows-2022, fabric, 3.8) success ✅
install-pkg (windows-2022, fabric, 3.11) success ✅
install-pkg (windows-2022, pytorch, 3.8) success ✅
install-pkg (windows-2022, pytorch, 3.11) success ✅
install-pkg (windows-2022, lightning, 3.8) success ✅
install-pkg (windows-2022, lightning, 3.11) success ✅
install-pkg (windows-2022, notset, 3.8) success ✅
install-pkg (windows-2022, notset, 3.11) success ✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/strategies/fsdp.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@mergify mergify bot added the ready PRs ready to be merged label Aug 30, 2023
@carmocca carmocca merged commit fdda196 into master Aug 30, 2023
@carmocca carmocca deleted the carmocca/xlafsdp-api branch August 30, 2023 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Borda Borda Borda approved these changes

@awaelchli awaelchli awaelchli approved these changes

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

Assignees

@carmocca carmocca

Labels
fabric lightning.fabric.Fabric feature Is an improvement or enhancement pl Generic label for PyTorch Lightning package ready PRs ready to be merged strategy: xla
Projects
None yet
Milestone
2.1
Development

Successfully merging this pull request may close these issues.
3 participants
@carmocca @awaelchli @Borda