Raise a WARNING when someone tried to load the best checkpoint when one has not been set.

[aprbw]

🚀 Feature

Raise a WARNING when someone call:
Trainer.test(ckpt_path='best')
or
Trainer.test(ckpt_path='best')
if there is no user-defined ModelCheckpoint in callbacks

Motivation

I'm frustrated when there is a deadline approaching, and I need to update my PL to the latest version, and it breaks a bunch of things, and one of thing that silently break is the model checkpointing.

If someone is calling Trainer.test(ckpt_path='best') they are expecting the best model to be loaded, not the latest model. If it is loading the latest model instead (due to user-defined ModelCheckpoint was not set), please raise a warning and not just silently load the latest model. Freaked me out real bad, I literally lost sleep over this.

Pitch

Raise a WARNING when someone call:
Trainer.test(ckpt_path='best')
or
Trainer.test(ckpt_path='best')
if there is no user-defined ModelCheckpoint in callbacks

Alternatives

None

Additional context

None

cc @tchaton @justusschock @awaelchli @Borda @ananthsub @ninginthecloud @rohitgr7 @carmocca @jjenniferdai

[carmocca]

Hi @aprbw! Sorry that you had a bad experience with this.

When you create a Trainer(), we add a instance ModelCheckpoint() internally as long as you don't set Trainer(enable_model_checkpointing=False).

This ModelCheckpoint() instance by default has monitor=None which means it doesn't track any specific logged value, instead the last model saved.
Instances with this configuration still set best_model_path, the attribute checked with test(ckpt_path='best').

Even though this was confusing to you, this is expected behavior for other users who want to set 'best' and forget without having to take into account the particular checkpointing configuration.

I am not sure we can show a warning here because this is intended.

On a similar vein: #12485

[aprbw]

Sorry that you had a bad experience with this.

Sorry about my rant.

Instances with this configuration still set best_model_path, the attribute checked with test(ckpt_path='best').

I see, so there are cases where this is a desired behavior. However, I still think that this is still confusing. I'm also not sure what the correct solution should be.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!