Dataloader is reloaded twice after resuming from checkpoint

[ninginthecloud]

🚀 Feature

Motivation

PyTorch Lightning reloads dataloader twice after resuming from checkpoint:
the first time reloads before train loop start [link];
The second reloads of default value for reload_dataloaders_every_n_epochs [link]
This behavior could be problematic:

• This behavior could bring user confusion. They expect reload_dataloaders_every_n_epochs = 1 forces the dataloaders to reload once every n epochs, but it loads multiple times sometimes.
• The dataloader loading behavior is inconsistent between resuming from checkpoint and fresh training start. This could break the internal states of dataloader, which requires users to keep track of global states.
• It may impact Fault tolerance training correctly captures the state of dataloader

Pitch

Let's remove self.reset_train_val_dataloaders(model) in _run_train() [link]
Because reset_*_dataloader() will be called in fit loop, there's no need to call it in trainer.

Alternatives

N/A

Additional context

cc: @awaelchli @ananthsub

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

• Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

• Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

[tchaton]

Hey @ninginthecloud,

Quite catch. I believe this logic could definitely benefit from a code-health refactor. Feel free to add it to the sprint if you plan to work on it.

Best,
T.C