[Feat] Add support for fault tolerance within the ValidationLoop

[tchaton]

🚀 Feature

Motivation

Lightning 1.5 is going to introduce Fault Tolerant feature.
However, the feature is still requires some improvements. As a matter of fact, Lightning doesn't support to be restarted within the ValidationLoop yet.

This issue tracks this addition.

Pitch

Alternatives

Additional context

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

• Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

• Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

[awaelchli]

duplicate of #9130, this item is already included in the list there.