Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove result serialization (under fault tolerance) #16516

Merged
merged 7 commits into from
Jan 27, 2023

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Jan 26, 2023

What does this PR do?

Fault-tolerance is composed of several items:

  1. Automatic DataLoader state serialization
  2. Manual DataLoader state serialization
  3. Metrics/Logging results state
  4. Sigterm handling to signal to terminate gracefully
  5. Save a checkpoint when it is signaled to terminate gracefully

This PR removes (3). This might re-land in the future depending on future logging changes.

Requires #16512 to be merged first.

Does your PR introduce any breaking changes? If yes, please list them.

Removes support for serializing the logged results under the PL_FAULT_TOLERANCE_TRAINING environment variable

cc @Borda @justusschock @awaelchli @carmocca

@carmocca carmocca added refactor breaking change Includes a breaking change fault tolerance experimental pl Generic label for PyTorch Lightning package labels Jan 26, 2023
@carmocca carmocca added this to the 2.0 milestone Jan 26, 2023
@carmocca carmocca self-assigned this Jan 26, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jan 26, 2023

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11) success
pl-cpu (macOS-11, pytorch, 3.9, 1.12) success
pl-cpu (macOS-11, pytorch, 3.10, 1.13) success
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest) success
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10) success
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11) success
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12) success
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13) success
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest) success
pl-cpu (windows-2022, pytorch, 3.9, 1.11) success
pl-cpu (windows-2022, pytorch, 3.10, 1.12) success
pl-cpu (windows-2022, pytorch, 3.10, 1.13) success
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest) success
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11) success
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11) success
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11) success
pl-cpu (macOS-11, lightning, 3.8, 1.13) success
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13) success
pl-cpu (windows-2022, lightning, 3.8, 1.13) success

These checks are required after the changes to src/pytorch_lightning/loops/loop.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/trainer/connectors/logger_connector/result.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/test_loop_state_dict.py, tests/tests_pytorch/loops/test_loops.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) success

These checks are required after the changes to src/pytorch_lightning/loops/loop.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/trainer/connectors/logger_connector/result.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/test_loop_state_dict.py, tests/tests_pytorch/loops/test_loops.py.

🟢 pytorch_lightning: Azure HPU
Check ID Status
pytorch-lightning (HPUs) success

These checks are required after the changes to src/pytorch_lightning/loops/loop.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/trainer/connectors/logger_connector/result.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/test_loop_state_dict.py, tests/tests_pytorch/loops/test_loops.py.

🟢 pytorch_lightning: Azure IPU
Check ID Status
pytorch-lightning (IPUs) success

These checks are required after the changes to src/pytorch_lightning/loops/loop.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/trainer/connectors/logger_connector/result.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/loops/test_loop_state_dict.py, tests/tests_pytorch/loops/test_loops.py.

🟢 pytorch_lightning: Docs
Check ID Status
make-doctest (pytorch) success
make-html (pytorch) success

These checks are required after the changes to src/pytorch_lightning/loops/loop.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/trainer/connectors/logger_connector/result.py.

🟢 mypy
Check ID Status
mypy success

These checks are required after the changes to src/pytorch_lightning/loops/loop.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/trainer/connectors/logger_connector/result.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.7) success
install-pkg (ubuntu-22.04, app, 3.10) success
install-pkg (ubuntu-22.04, fabric, 3.7) success
install-pkg (ubuntu-22.04, fabric, 3.10) success
install-pkg (ubuntu-22.04, pytorch, 3.7) success
install-pkg (ubuntu-22.04, pytorch, 3.10) success
install-pkg (ubuntu-22.04, lightning, 3.7) success
install-pkg (ubuntu-22.04, lightning, 3.10) success
install-pkg (ubuntu-22.04, notset, 3.7) success
install-pkg (ubuntu-22.04, notset, 3.10) success
install-pkg (macOS-12, app, 3.7) success
install-pkg (macOS-12, app, 3.10) success
install-pkg (macOS-12, fabric, 3.7) success
install-pkg (macOS-12, fabric, 3.10) success
install-pkg (macOS-12, pytorch, 3.7) success
install-pkg (macOS-12, pytorch, 3.10) success
install-pkg (macOS-12, lightning, 3.7) success
install-pkg (macOS-12, lightning, 3.10) success
install-pkg (macOS-12, notset, 3.7) success
install-pkg (macOS-12, notset, 3.10) success
install-pkg (windows-2022, app, 3.7) success
install-pkg (windows-2022, app, 3.10) success
install-pkg (windows-2022, fabric, 3.7) success
install-pkg (windows-2022, fabric, 3.10) success
install-pkg (windows-2022, pytorch, 3.7) success
install-pkg (windows-2022, pytorch, 3.10) success
install-pkg (windows-2022, lightning, 3.7) success
install-pkg (windows-2022, lightning, 3.10) success
install-pkg (windows-2022, notset, 3.7) success
install-pkg (windows-2022, notset, 3.10) success

These checks are required after the changes to src/pytorch_lightning/loops/loop.py, src/pytorch_lightning/trainer/connectors/checkpoint_connector.py, src/pytorch_lightning/trainer/connectors/logger_connector/result.py.


Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@mergify mergify bot removed the has conflicts label Jan 27, 2023
@mergify mergify bot added the ready PRs ready to be merged label Jan 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking change Includes a breaking change experimental fault tolerance pl Generic label for PyTorch Lightning package ready PRs ready to be merged refactor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants