Checkpoint naming broken

[neelr11]

🐛 Bug

I would like to be able to save checkpoints with custom names that include the value of my val_loss, ie. path/epoch_2-val_loss_0.2.hdf5 . The documentation for ModelCheckpoint suggests that this is possible using the filepath argument. This does not appear to be the case, since the source code calls os.mkdirs(filepath). I have also tried using the prefix argument, but it doesn't seem to be possible to pass it a format string containing a variable.

Expected behavior

The documentation claims that filepath='{epoch:02d}-{val_loss:.2f}.hdf5' will save a checkpoint at /path/epoch_2-val_loss_0.2.hdf5. Instead, it saves a checkpoint at {epoch:02d}-{val_loss:.2f}.hdf5/_ckpt_epoch_1.ckpt.

The issues in the documentation are two-fold:
-- It suggests that filepath can contain the directory + name of the checkpoint, when it seems like it should only contain the directory specifying where to save.
-- It suggests that it can 'contain named formatting options to be auto-filled', which also doesn't seem to be the case.

Is it possible to achieve this functionality with the prefix argument instead? If so, how?

[williamFalcon]

that’s already possible.

https://pytorch-lightning.readthedocs.io/en/latest/callbacks.html.

# save epoch and val_loss in name
ModelCheckpoint(filepath='{epoch:02d}-{val_loss:.2f}.hdf5')
# saves file like: /path/epoch_2-val_loss_0.2.hdf5

but as you mentioned, if that’s broken then it’s better to submit a PR to fix? :)

[williamFalcon]

@neelr11 did this work for you? is this still a bug?

[williamFalcon]

@neelr11 @Borda updates on this?

[neelr11]

@williamFalcon this is still a bug. I can submit a PR this weekend.

[versatran01]

I suggest following pytorch-ignite and just add the monitored name and value to the saved checkpoint.
So filepath would simply be a directory and saved checkpoint will be something like {prefix}ckpt_epoch{epoch}{monitor}{monitor_value}.ckpt

[Borda]

@neelr11 are you still interested in sending a PR?

[davinnovation]

As neerlr11 mentioned, 0.6.0-modelcallback didn't work as documented: supporting dynamic checkpoint string like filepath='{epoch:02d}-{val_loss:.2f}.hdf5' - waiting for update