Support multiple dataloaders with dataloader_iter
#18390
Conversation
carmocca
added
data handling
…ataloader_iter_multiple_dataloaders
carmocca
requested review from
awaelchli,
justusschock,
Borda and
williamFalcon
as code owners
⚡ Required checks status: All passing 🟢Groups summary🟢 pytorch_lightning: Tests workflowThese checks are required after the changes to 🟢 pytorch_lightning: Azure GPU
These checks are required after the changes to 🟢 pytorch_lightning: Benchmarks
These checks are required after the changes to 🟢 pytorch_lightning: Docs
These checks are required after the changes to 🟢 mypy
These checks are required after the changes to 🟢 installThese checks are required after the changes to Thank you for your contribution! 💜
|
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #18390 +/- ##
==========================================
- Coverage 85% 50% -35%
==========================================
Files 427 419 -8
Lines 33289 33186 -103
==========================================
- Hits 28291 16655 -11636
- Misses 4998 16531 +11533 |
| if isinstance(data_fetcher, _DataLoaderIterDataFetcher): | ||
| dataloader_iter = next(data_fetcher) | ||
| # hook's batch_idx and dataloader_idx arguments correctness cannot be guaranteed in this setting | ||
| batch = data_fetcher._batch |
There was a problem hiding this comment.
Did we store the fetched batch in memory before this PR as well? I think that previously the "batch" variable was actually the dataloader_iter. My understanding is that we need to store the batch so we can pass it to the callback hooks. An argument could probably be made that in case of dataloader_iter, the loop should probably not hold onto a batch (and perhaps pass None to the hooks).
What does this PR do?
Support multiple dataloaders with
dataloader_iterThis unblocks the NeMo team.
cc @justusschock @awaelchli @tchaton @Borda