Checkpoint saving is not working in custom loops KFold.py example

[FinnBehrendt]

🐛 Bug

When using KFold.py from the custom loop example, the checkpoint callback does not save checkpoints properly. Only for the first few iterations checkpoints are saved. After some iterations, the checkpoints are not updated anymore.

To Reproduce

For me, this error appeared also for the plain example from here

Expected behavior

Checkpoints should be saved as specified in the checkpoint_callback

Environment

• PyTorch Lightning Version (e.g., 1.5.0): 1.5.10, 1.5.0, 1.6.0
• PyTorch Version (e.g., 1.10):1.10
• Python version (e.g., 3.9): 3.9
• OS (e.g., Linux): Linux
• CUDA/cuDNN version: 11.3
• GPU models and configuration: V100
• How you installed PyTorch (conda, pip, source): conda

cc @awaelchli @ananthsub @ninginthecloud @rohitgr7 @otaj @carmocca @justusschock

[JinLi711]

Hey @FinnBehrendt,

I've tried out the Kfold.py example in the master branch and I don't see your issue showing up. So maybe you can try the master version of Kfold.py.

When I set breakpoints in Kfold.py, I do see that checkpoints in lightning_logs/ are being updated continuously throughout the training process.

[JinLi711]

After investigating this issue more, I realized that checkpointing in Kfold.py is an issue. And I can reproduce your error after running Kfold.py a few more times.

I believe that what is happening is that after every fold, the checkpoint state is not reset. So for example, say that after fold 1 the minimum validation loss is 1.5 and the best checkpoint is saved. Then fold 2 trains and the minimum validation loss is 2.0. Because the checkpoint state is not reset, the best checkpoint in fold 2 is worse than the Trainer's minimum validation loss of 1.5 and will not save the checkpoint.

What we need to do is to make the checkpoints of each fold independent of another.

Here's a related issue #12300 that I opened.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[awaelchli]

Yes, the Kfold example is flawed and the checkpointing is just one example where things go wrong. To implement kfold properly, one needs to loop externally and use a new trainer every time and not share any state between the runs. If that makes sense. While one can probably try to fix the checkpointing issue, systematic issues with this internal loop still exist. I wonder if https://github.com/SkafteNicki/pl_cross has the same problem.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[carmocca]

It does have the same problem. We will be revisiting the kfold loop in the future
cc @rasbt @SkafteNicki

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[awaelchli]

Closing the issue since special support for custom loops was removed and the example with it too.