Change to save checkpoint and run validation on N steps instead of epochs

[roedoejet]

Since validation on every epoch can cause performance problems on small datasets.

[roedoejet]

See Lightning-AI/pytorch-lightning#2534

[SamuelLarkin]

Notes:
TRAINER

• limit_val_batches (Union[int, float, None]) – How much of validation dataset to check (float = fraction, int = num_batches). Default: 1.0.
• val_check_interval (Union[int, float, None]) – How often to check the validation set. Pass a float in the range [0.0, 1.0] to check after a fraction of the training epoch. Pass an int to check after a fixed number of training batches. An int value can only be higher than the number of training batches when check_val_every_n_epoch=None, which validates after every N training batches across epochs or during iteration-based training. Default: 1.0.
• check_val_every_n_epoch (Optional[int]) – Perform a validation loop every after every N training epochs. If None, validation will be done solely based on the number of training batches, requiring val_check_interval to be an integer value. Default: 1.

val_check_interval was added in Support val_check_interval values higher than number of training batches and was merged on Apr 21, 2022.