Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable trainer val_check_interval to be greater than number of the training batches #4406

Closed
Vozf opened this issue Oct 28, 2020 · 4 comments · Fixed by #11993
Closed

Enable trainer val_check_interval to be greater than number of the training batches #4406

Vozf opened this issue Oct 28, 2020 · 4 comments · Fixed by #11993
Labels
design Includes a design discussion feature Is an improvement or enhancement help wanted Open to be worked on
Milestone

Comments

@Vozf
Copy link
Contributor

Vozf commented Oct 28, 2020

Currently I can't set val_check_interval greater than number of the training batches
Error occurs

/site-packages/pytorch_lightning/trainer/data_loading.py", line 203, in reset_train_dataloader
    raise ValueError(
ValueError: `val_check_interval` (1000000) must be less than or equal to the number of the training batches (5). If you want to disable validation set `limit_val_batches` to 0.0 instead.

But it is a useful feature when validation takes considerate time. I want to do validation after multiple train epochs(but fixed number of steps, so I can't use check_val_every_n_epoch)

cc @Borda @tchaton @justusschock @awaelchli

@Vozf Vozf added bug Something isn't working help wanted Open to be worked on labels Oct 28, 2020
@justusschock
Copy link
Member

This would probably require #4086 first, right?

@Vozf
Copy link
Contributor Author

Vozf commented Oct 28, 2020

Yeah, seems like so

@awaelchli awaelchli added design Includes a design discussion feature Is an improvement or enhancement and removed bug Something isn't working labels Oct 28, 2020
@stale stale bot added the won't fix This will not be worked on label Nov 27, 2020
@rohitgr7 rohitgr7 removed the won't fix This will not be worked on label Nov 27, 2020
@stale stale bot added the won't fix This will not be worked on label Dec 27, 2020
@stale stale bot closed this as completed Jan 3, 2021
@Mi-Przystupa
Copy link

Has this issue been resolved? I'm using the latest version of pylightning and I seem to get this error when my number of training batches are less than my val_check_interval

@akihironitta akihironitta reopened this Nov 12, 2021
@stale stale bot removed the won't fix This will not be worked on label Nov 12, 2021
@Lightning-AI Lightning-AI deleted a comment from stale bot Nov 12, 2021
@Lightning-AI Lightning-AI deleted a comment from stale bot Nov 12, 2021
@stale
Copy link

stale bot commented Dec 15, 2021

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

@stale stale bot added the won't fix This will not be worked on label Dec 15, 2021
@awaelchli awaelchli added this to the 1.6 milestone Dec 15, 2021
@stale stale bot removed the won't fix This will not be worked on label Dec 15, 2021
@carmocca carmocca modified the milestones: 1.6, 1.7 Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design Includes a design discussion feature Is an improvement or enhancement help wanted Open to be worked on
Projects
None yet
7 participants