Simplify double precision input and module conversion #18209

awaelchli · 2023-08-01T10:29:48Z

What does this PR do?

Refactors the double precision logic for converting inputs and model weight conversion. This needs to be standardized and simplified for supporting half-precision: #18193

This is only possible after #17531
Fixes #17655
Closes #13000 (no longer needed)

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda @justusschock @awaelchli @tchaton @carmocca

github-actions · 2023-08-01T10:54:32Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/base.py, src/lightning/pytorch/plugins/precision/double.py, src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/strategy.py, tests/tests_pytorch/deprecated_api/test_no_removal_version.py, tests/tests_pytorch/overrides/test_base.py, tests/tests_pytorch/plugins/precision/test_double.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/base.py, src/lightning/pytorch/plugins/precision/double.py, src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/strategy.py, tests/tests_pytorch/deprecated_api/test_no_removal_version.py, tests/tests_pytorch/overrides/test_base.py, tests/tests_pytorch/plugins/precision/test_double.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/base.py, src/lightning/pytorch/plugins/precision/double.py, src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/strategy.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/base.py, src/lightning/pytorch/plugins/precision/double.py, src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/strategy.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/base.py, src/lightning/pytorch/plugins/precision/double.py, src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/strategy.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/overrides/base.py, src/lightning/pytorch/plugins/precision/double.py, src/lightning/pytorch/strategies/ddp.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/strategy.py.

🟢 link-check

Check ID	Status
check-md-links / markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

src/lightning/pytorch/plugins/precision/double.py

src/lightning/pytorch/strategies/strategy.py

src/lightning/pytorch/plugins/precision/double.py

for more information, see https://pre-commit.ci

Co-authored-by: Jirka Borovec <[email protected]>

lantiga

LGTM!

awaelchli added 2 commits August 1, 2023 12:25

refactor

e12d9f4

add strategy calls

f11c1c5

github-actions bot added the pl Generic label for PyTorch Lightning package label Aug 1, 2023

awaelchli added refactor deprecation Includes a deprecation precision: double Double precision code quality labels Aug 1, 2023

awaelchli added this to the 2.1 milestone Aug 1, 2023

awaelchli added 3 commits August 1, 2023 12:44

deprecation

c563238

changelog

bc02e95

test

630bf82

awaelchli marked this pull request as ready for review August 1, 2023 10:54

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners August 1, 2023 10:54

[pre-commit.ci] auto fixes from pre-commit.com hooks

884e1e0

for more information, see https://pre-commit.ci

awaelchli commented Aug 1, 2023

View reviewed changes

src/lightning/pytorch/plugins/precision/double.py Show resolved Hide resolved

awaelchli commented Aug 1, 2023

View reviewed changes

src/lightning/pytorch/plugins/precision/double.py Show resolved Hide resolved

justusschock approved these changes Aug 1, 2023

View reviewed changes

carmocca approved these changes Aug 1, 2023

View reviewed changes

src/lightning/pytorch/plugins/precision/double.py Show resolved Hide resolved

src/lightning/pytorch/strategies/strategy.py Outdated Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Aug 1, 2023

Borda approved these changes Aug 1, 2023

View reviewed changes

src/lightning/pytorch/plugins/precision/double.py Outdated Show resolved Hide resolved

awaelchli and others added 4 commits August 1, 2023 21:06

move convert module to connect()

b978963

mypy

8569fe6

[pre-commit.ci] auto fixes from pre-commit.com hooks

8d95a97

for more information, see https://pre-commit.ci

Update src/lightning/pytorch/plugins/precision/double.py

c59f79b

Co-authored-by: Jirka Borovec <[email protected]>

lantiga approved these changes Aug 1, 2023

View reviewed changes

Merge branch 'master' into refactor/double-precision

403886c

awaelchli merged commit a581a1f into master Aug 1, 2023

awaelchli deleted the refactor/double-precision branch August 1, 2023 23:37

awaelchli mentioned this pull request Nov 27, 2023

Delay Precision.convert_module until configure_model has run #19061

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify double precision input and module conversion #18209

Simplify double precision input and module conversion #18209

awaelchli commented Aug 1, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023 •

edited

Loading

lantiga left a comment

Simplify double precision input and module conversion #18209

Simplify double precision input and module conversion #18209

Conversation

awaelchli commented Aug 1, 2023 • edited Loading

What does this PR do?

PR review

github-actions bot commented Aug 1, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

lantiga left a comment

Choose a reason for hiding this comment

awaelchli commented Aug 1, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023 •

edited

Loading