Support all half-precision modes in FSDP precision plugin #17807

awaelchli · 2023-06-11T21:27:39Z

What does this PR do?

Fixes #17657

Removes inheritance of MixedPrecision from the FSDP precision plugin
The plugin now handles both the full-precision, mixed-precision, and true half precision.

cc @Borda @justusschock @awaelchli @carmocca

for more information, see https://pre-commit.ci

…nto fabric/fsdp-precision-16-true

for more information, see https://pre-commit.ci

…nto fabric/fsdp-precision-16-true

…/pytorch-lightning#17807 is merged

for more information, see https://pre-commit.ci

…AI/lightning into fabric/fsdp-precision-16-true

for more information, see https://pre-commit.ci

src/lightning/fabric/connector.py

…AI/lightning into fabric/fsdp-precision-16-true

github-actions · 2023-07-04T14:25:12Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py.

🟢 fabric: Docs

Check ID	Status
make-doctest (fabric)	success	✅
make-html (fabric)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py, tests/tests_fabric/plugins/precision/test_fsdp.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/test_connector.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py, tests/tests_fabric/plugins/precision/test_fsdp.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/test_connector.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/fabric/connector.py, src/lightning/fabric/plugins/precision/amp.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/fsdp.py.

🟢 link-check

Check ID	Status
check-md-links / markdown-link-check	success	✅

These checks are required after the changes to src/lightning/fabric/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

src/lightning/fabric/plugins/precision/fsdp.py

refactor fsdp precision

606ef68

github-actions bot added the fabric lightning.fabric.Fabric label Jun 11, 2023

pre-commit-ci bot and others added 4 commits June 11, 2023 21:28

[pre-commit.ci] auto fixes from pre-commit.com hooks

e7cb399

for more information, see https://pre-commit.ci

imports

2075891

Merge remote-tracking branch 'origin/fabric/fsdp-precision-16-true' i…

c002b10

…nto fabric/fsdp-precision-16-true

x

cd439c3

awaelchli added precision: amp Automatic Mixed Precision feature Is an improvement or enhancement refactor fun Staff contributions outside working hours - to differentiate from the "community" label labels Jun 11, 2023

pre-commit-ci bot and others added 3 commits June 11, 2023 21:34

[pre-commit.ci] auto fixes from pre-commit.com hooks

fa8b717

for more information, see https://pre-commit.ci

import

a7a2b9a

Merge remote-tracking branch 'origin/fabric/fsdp-precision-16-true' i…

3d77c8f

…nto fabric/fsdp-precision-16-true

speediedan added a commit to speediedan/finetuning-scheduler that referenced this pull request Jun 22, 2023

disable fsdp 1.x mixed precision tests temporarily until Lightning-AI…

982848a

…/pytorch-lightning#17807 is merged

awaelchli added this to the 2.1 milestone Jul 1, 2023

awaelchli and others added 6 commits July 1, 2023 01:04

Merge branch 'master' into fabric/fsdp-precision-16-true

2a6effe

add connector test

4a59937

[pre-commit.ci] auto fixes from pre-commit.com hooks

7fd725a

for more information, see https://pre-commit.ci

add tests

686f39d

Merge branch 'fabric/fsdp-precision-16-true' of github.com:Lightning-…

afe9a74

…AI/lightning into fabric/fsdp-precision-16-true

[pre-commit.ci] auto fixes from pre-commit.com hooks

7ae7ed0

for more information, see https://pre-commit.ci

awaelchli commented Jul 1, 2023

View reviewed changes

src/lightning/fabric/connector.py Show resolved Hide resolved

awaelchli added 8 commits July 1, 2023 02:18

fix input types

d6088a9

fix docstring and types

babe86a

Merge branch 'fabric/fsdp-precision-16-true' of github.com:Lightning-…

2b4d623

…AI/lightning into fabric/fsdp-precision-16-true

fix handling 32-true

68a16ce

update

cf2aa6a

mypy

4f05d9c

chlog

4ccaf37

64 not supported

e06a4c3

awaelchli marked this pull request as ready for review July 4, 2023 14:24

awaelchli requested review from carmocca and justusschock as code owners July 4, 2023 14:24

awaelchli commented Jul 4, 2023

View reviewed changes

src/lightning/fabric/plugins/precision/fsdp.py Show resolved Hide resolved

carmocca approved these changes Jul 4, 2023

View reviewed changes

awaelchli added 3 commits July 4, 2023 20:34

decorate tests with runif min_torch=1.12

9f0f2bf

fix mypy issues

9ecb4e1

add min_cuda_gpus=1

8a1378d

mergify bot added the has conflicts label Jul 4, 2023

awaelchli added the ready PRs ready to be merged label Jul 6, 2023

mergify bot removed the ready PRs ready to be merged label Jul 6, 2023

Borda approved these changes Jul 6, 2023

View reviewed changes

Merge branch 'master' into fabric/fsdp-precision-16-true

5fd636c

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Jul 6, 2023

justusschock approved these changes Jul 7, 2023

View reviewed changes

Merge branch 'master' into fabric/fsdp-precision-16-true

6411c64

awaelchli enabled auto-merge (squash) July 7, 2023 18:59

Merge branch 'master' into fabric/fsdp-precision-16-true

bbf1b2e

awaelchli merged commit acc70d0 into master Jul 9, 2023

awaelchli deleted the fabric/fsdp-precision-16-true branch July 9, 2023 18:40

awaelchli mentioned this pull request Aug 2, 2023

Support true 16-bit precision with FSDP in Trainer #18219

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support all half-precision modes in FSDP precision plugin #17807

Support all half-precision modes in FSDP precision plugin #17807

awaelchli commented Jun 11, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jul 4, 2023 •

edited

Loading

Support all half-precision modes in FSDP precision plugin #17807

Support all half-precision modes in FSDP precision plugin #17807

Conversation

awaelchli commented Jun 11, 2023 • edited by github-actions bot Loading

What does this PR do?

github-actions bot commented Jul 4, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

awaelchli commented Jun 11, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jul 4, 2023 •

edited

Loading