Saving checkpoints instead of overwriting during validation

[wwerkk]

Just wondering if it is possible to save a new, separate checkpoint every time validation takes place instead of the default overwriting?
I realise it is quite consuming in terms of storage space, but on the other hand saving every hundred thousand steps or so would make it way easier to keep up with the phase 2 of training, since if training is left running too long some artefacts can occur and earlier checkpoints are impossible to retrieve.