allow val_check_interval to be larger than training dataset size

[cccntu]

🚀 Feature

allow val_check_interval to be larger than the number of the training batches in one epoch

Motivation

I am using a small datasets, so instead of specifying max_epochs in Trainer, I want to use max_steps and evaluate every val_check_interval , but when val_check_interval is larger than number of batches in training set, there is an error, like this:

`val_check_interval` (100) must be less than or equal to the number of the training batches (24). If you want to disable validation set `limit_val_batches` to 0.0 instead.

Pitch

val_check_interval should't be limited by the number of the training batches in one epoch, it should be a warning, not an error

Alternatives

I am currently using a wrapper to make it an iterable dataset, so it allows me to do that

Additional context

[rohitgr7]

#4409 similar

[gan3sh500]

@rohitgr7 @tchaton Could I work on this?

[rohitgr7]

@gan3sh500 yeah go ahead.

[gan3sh500]

@cccntu Did you try check_val_every_n_epoch for this purpose.

@rohitgr7 This issue can be fixed by removing exceptions to warning and checking trainer.total_batch_idx when val_check_batch greater than trainer.num_training_batches. Shall I open a PR?

It seems a bit redundant with check_val_every_n_epoch. Or shall we add float support for check_val_every_n_epoch?

[rohitgr7]

yeah I guess just deprecate check_val_every_n_epoch and let val_check_interval handle it using float values (1.0, 2.0, 3.0...). Also use release/1.2-dev as the base to avoid conflicts.

[gan3sh500]

@rohitgr7 Sounds good. It'll be a bit cleaner this way. I'll make the changes for this and PR.

[edenlightning]

@gan3sh500 any update on this?

[turian]

Related to this discussion: #6253

[cccntu]

@cccntu Did you try check_val_every_n_epoch for this purpose.
It seems a bit redundant with check_val_every_n_epoch. Or shall we add float support for check_val_every_n_epoch?

Sorry for the late reply. I use val_check_interval=n to evaluate every n steps, using check_val_every_n_epoch=n would be evaluating every n epochs.

Using together with max_steps=m, I can make sure the number of training steps don't change if the dataset size changes.

[kaushikb11]

Not exactly Priority P0. Moving it to P1

[Rizhiy]

Any updates on this? I really don't like training using epochs, so this feature would be quite useful for me.

[edenlightning]

@kaushikb11 what is left TODO here?

[shivammehta25]

Is this change merged? if yes then in which version? It is really useful, I am not sure why it is marked P2.