True half-precision support in Trainer #18193

awaelchli · 2023-07-30T22:50:36Z

What does this PR do?

Implements true half-precision support in Trainer, with corresponding changes as done for Fabric in #17287.

Follow up work:

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda @justusschock @awaelchli

for more information, see https://pre-commit.ci

…ature/16-true-trainer

github-actions · 2023-08-01T12:15:06Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/plugins/__init__.py, src/lightning/pytorch/plugins/precision/__init__.py, src/lightning/pytorch/plugins/precision/half.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/plugins/precision/test_half.py, tests/tests_pytorch/strategies/test_common.py, tests/tests_pytorch/strategies/test_ddp.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/plugins/__init__.py, src/lightning/pytorch/plugins/precision/__init__.py, src/lightning/pytorch/plugins/precision/half.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, tests/tests_pytorch/plugins/precision/test_half.py, tests/tests_pytorch/strategies/test_common.py, tests/tests_pytorch/strategies/test_ddp.py, tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/pytorch/plugins/__init__.py, src/lightning/pytorch/plugins/precision/__init__.py, src/lightning/pytorch/plugins/precision/half.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/plugins/__init__.py, src/lightning/pytorch/plugins/precision/__init__.py, src/lightning/pytorch/plugins/precision/half.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py, docs/source-pytorch/api_references.rst, docs/source-pytorch/common/lightning_module.rst, docs/source-pytorch/common/precision_basic.rst, docs/source-pytorch/common/precision_intermediate.rst, docs/source-pytorch/common/trainer.rst, docs/source-pytorch/levels/advanced_level_19.rst, docs/source-pytorch/levels/advanced_level_20.rst, docs/source-pytorch/model/build_model_intermediate.rst.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/plugins/__init__.py, src/lightning/pytorch/plugins/precision/__init__.py, src/lightning/pytorch/plugins/precision/half.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/plugins/__init__.py, src/lightning/pytorch/plugins/precision/__init__.py, src/lightning/pytorch/plugins/precision/half.py, src/lightning/pytorch/trainer/connectors/accelerator_connector.py.

🟢 link-check

Check ID	Status
check-md-links / markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

justusschock

minor comment to clearly differentiate between half and mixed precision. Wording wasn't clear to me before

docs/source-pytorch/common/trainer.rst

lantiga

LGTM, after merging @justusschock 's suggestion on the description

Co-authored-by: Justus Schock <[email protected]>

True half-precision support in Trainer

290101e

github-actions bot added the pl Generic label for PyTorch Lightning package label Jul 30, 2023

awaelchli added feature Is an improvement or enhancement plugin trainer fun Staff contributions outside working hours - to differentiate from the "community" label and removed pl Generic label for PyTorch Lightning package labels Jul 30, 2023

awaelchli added this to the 2.1 milestone Jul 30, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

5b027b9

for more information, see https://pre-commit.ci

github-actions bot added the pl Generic label for PyTorch Lightning package label Jul 30, 2023

awaelchli and others added 3 commits July 31, 2023 01:11

add tests

ec1f397

[pre-commit.ci] auto fixes from pre-commit.com hooks

6417261

for more information, see https://pre-commit.ci

tests

853046a

awaelchli added the docs Documentation related label Jul 31, 2023

pre-commit-ci bot and others added 4 commits July 31, 2023 00:47

[pre-commit.ci] auto fixes from pre-commit.com hooks

49209c0

for more information, see https://pre-commit.ci

fix typo

c20c501

convert

33881bb

[pre-commit.ci] auto fixes from pre-commit.com hooks

a29359a

for more information, see https://pre-commit.ci

awaelchli mentioned this pull request Aug 1, 2023

Simplify double precision input and module conversion #18209

Merged

7 tasks

awaelchli removed the fun Staff contributions outside working hours - to differentiate from the "community" label label Aug 1, 2023

awaelchli and others added 7 commits August 1, 2023 12:53

reset

dc35ef4

docs

44bef1b

docs

8882e28

docs

b58b111

[pre-commit.ci] auto fixes from pre-commit.com hooks

1512556

for more information, see https://pre-commit.ci

changelog

1ff3c1f

Merge remote-tracking branch 'origin/feature/16-true-trainer' into fe…

049ed20

…ature/16-true-trainer

awaelchli marked this pull request as ready for review August 1, 2023 12:14

awaelchli requested review from carmocca and justusschock as code owners August 1, 2023 12:14

awaelchli requested review from williamFalcon, edenlightning, lantiga and Borda as code owners August 1, 2023 12:14

mergify bot added the has conflicts label Aug 1, 2023

Merge branch 'master' into feature/16-true-trainer

4a6b9aa

mergify bot removed the has conflicts label Aug 1, 2023

awaelchli and others added 2 commits August 1, 2023 14:32

reset

e47608e

[pre-commit.ci] auto fixes from pre-commit.com hooks

e515195

for more information, see https://pre-commit.ci

justusschock approved these changes Aug 1, 2023

View reviewed changes

docs/source-pytorch/common/trainer.rst Outdated Show resolved Hide resolved

Borda approved these changes Aug 1, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Aug 1, 2023

Merge branch 'master' into feature/16-true-trainer

4e289d1

lantiga approved these changes Aug 1, 2023

View reviewed changes

Update docs/source-pytorch/common/trainer.rst

718bbec

Co-authored-by: Justus Schock <[email protected]>

awaelchli mentioned this pull request Aug 1, 2023

XLA's 16-bit precision plugin selection is incorrect #18172

Closed

awaelchli added 2 commits August 2, 2023 12:57

Merge branch 'master' into feature/16-true-trainer

3e58327

bad merge

2ce56f1

awaelchli merged commit c5cb532 into master Aug 2, 2023

awaelchli deleted the feature/16-true-trainer branch August 2, 2023 11:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

True half-precision support in Trainer #18193

True half-precision support in Trainer #18193

awaelchli commented Jul 30, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023 •

edited

Loading

justusschock left a comment

lantiga left a comment

True half-precision support in Trainer #18193

True half-precision support in Trainer #18193

Conversation

awaelchli commented Jul 30, 2023 • edited Loading

What does this PR do?

PR review

github-actions bot commented Aug 1, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

justusschock left a comment

Choose a reason for hiding this comment

lantiga left a comment

Choose a reason for hiding this comment

awaelchli commented Jul 30, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023 •

edited

Loading