Make the `FaultToleranceCheckpoint` callback opt-in #16512

carmocca · 2023-01-26T13:40:00Z

What does this PR do?

Fault-tolerance is composed of several items:

Automatic DataLoader state serialization
Manual DataLoader state serialization
Metrics/Logging results state
Sigterm handling to signal to terminate gracefully
Save a checkpoint when it is signaled to terminate gracefully

This PR changes (5) so it is not tied to the fault-tolerance feature, as it doesn't need to be and could be used for other purposes. In the process, the callback is renamed to be more explicit and made public. Tests are adapted to use it on-demand.

Does your PR introduce any breaking changes? If yes, please list them.

None

cc @Borda @justusschock @awaelchli @carmocca

github-actions · 2023-01-26T13:56:57Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, pytorch, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (windows-2022, pytorch, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to src/pytorch_lightning/callbacks/__init__.py, src/pytorch_lightning/callbacks/model_checkpoint.py, src/pytorch_lightning/callbacks/on_exception_checkpoint.py, src/pytorch_lightning/trainer/connectors/callback_connector.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/utilities/exceptions.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/optimization/test_optimizer_loop.py, tests/tests_pytorch/loops/test_loops.py, tests/tests_pytorch/run_standalone_tests.sh, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_auto_restart.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to examples/pl_fault_tolerant/automatic.py, src/pytorch_lightning/callbacks/__init__.py, src/pytorch_lightning/callbacks/model_checkpoint.py, src/pytorch_lightning/callbacks/on_exception_checkpoint.py, src/pytorch_lightning/trainer/connectors/callback_connector.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/utilities/exceptions.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/optimization/test_optimizer_loop.py, tests/tests_pytorch/loops/test_loops.py, tests/tests_pytorch/run_standalone_tests.sh, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_auto_restart.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/pytorch_lightning/callbacks/__init__.py, src/pytorch_lightning/callbacks/model_checkpoint.py, src/pytorch_lightning/callbacks/on_exception_checkpoint.py, src/pytorch_lightning/trainer/connectors/callback_connector.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/utilities/exceptions.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/optimization/test_optimizer_loop.py, tests/tests_pytorch/loops/test_loops.py, tests/tests_pytorch/run_standalone_tests.sh, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_auto_restart.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/pytorch_lightning/callbacks/__init__.py, src/pytorch_lightning/callbacks/model_checkpoint.py, src/pytorch_lightning/callbacks/on_exception_checkpoint.py, src/pytorch_lightning/trainer/connectors/callback_connector.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/utilities/exceptions.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/optimization/test_optimizer_loop.py, tests/tests_pytorch/loops/test_loops.py, tests/tests_pytorch/run_standalone_tests.sh, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_auto_restart.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/pytorch_lightning/callbacks/__init__.py, src/pytorch_lightning/callbacks/model_checkpoint.py, src/pytorch_lightning/callbacks/on_exception_checkpoint.py, src/pytorch_lightning/trainer/connectors/callback_connector.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/utilities/exceptions.py, docs/source-pytorch/api_references.rst.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to tests/tests_pytorch/run_standalone_tests.sh.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/pytorch_lightning/callbacks/__init__.py, src/pytorch_lightning/callbacks/model_checkpoint.py, src/pytorch_lightning/callbacks/on_exception_checkpoint.py, src/pytorch_lightning/trainer/connectors/callback_connector.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/utilities/exceptions.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.7)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.7)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.7)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.7)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.7)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.7)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.7)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.7)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.7)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.7)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.7)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.7)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.7)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.7)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.7)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/pytorch_lightning/callbacks/__init__.py, src/pytorch_lightning/callbacks/model_checkpoint.py, src/pytorch_lightning/callbacks/on_exception_checkpoint.py, src/pytorch_lightning/trainer/connectors/callback_connector.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/utilities/exceptions.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/pytorch_lightning/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

The is now opt-in

472ac7c

carmocca added feature Is an improvement or enhancement refactor fault tolerance pl Generic label for PyTorch Lightning package labels Jan 26, 2023

carmocca added this to the 2.0 milestone Jan 26, 2023

carmocca self-assigned this Jan 26, 2023

carmocca added 3 commits January 26, 2023 14:43

Update example

611f39c

Missed this

b83dbf1

api reference

76b6d41

carmocca marked this pull request as ready for review January 26, 2023 13:56

carmocca requested review from awaelchli, justusschock, williamFalcon, edenlightning, lantiga, Borda and tchaton as code owners January 26, 2023 13:56

carmocca added 2 commits January 26, 2023 15:34

Fixes

2973d2b

Update CHANGELOG

6ffd0f0

awaelchli approved these changes Jan 26, 2023

View reviewed changes

carmocca added 2 commits January 26, 2023 16:32

Missed these

4017320

Exclude on_exception from standalone

2fdb65a

carmocca added the breaking change Includes a breaking change label Jan 26, 2023

carmocca mentioned this pull request Jan 26, 2023

Remove result serialization (under fault tolerance) #16516

Merged

mergify bot added the has conflicts label Jan 27, 2023

Merge branch 'master' into refactor/on-exception-callback

d68731f

carmocca enabled auto-merge (squash) January 27, 2023 03:05

mergify bot removed the has conflicts label Jan 27, 2023

pre-commit-ci bot and others added 2 commits January 27, 2023 03:06

[pre-commit.ci] auto fixes from pre-commit.com hooks

5e7fc99

for more information, see https://pre-commit.ci

Undo

2d4fb7d

mergify bot added the has conflicts label Jan 27, 2023

justusschock approved these changes Jan 27, 2023

View reviewed changes

Merge branch 'master' into refactor/on-exception-callback

c56a0f1

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Jan 27, 2023

carmocca merged commit d562319 into master Jan 27, 2023

carmocca deleted the refactor/on-exception-callback branch January 27, 2023 13:02

carmocca mentioned this pull request Jan 30, 2023

Remove DataLoader serialization (under fault tolerance) #16533

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the `FaultToleranceCheckpoint` callback opt-in #16512

Make the `FaultToleranceCheckpoint` callback opt-in #16512

carmocca commented Jan 26, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 26, 2023 •

edited

Loading

Make the FaultToleranceCheckpoint callback opt-in #16512

Make the FaultToleranceCheckpoint callback opt-in #16512

Conversation

carmocca commented Jan 26, 2023 • edited by github-actions bot Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

github-actions bot commented Jan 26, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

Make the `FaultToleranceCheckpoint` callback opt-in #16512

Make the `FaultToleranceCheckpoint` callback opt-in #16512

carmocca commented Jan 26, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 26, 2023 •

edited

Loading