Support recreating dataloaders during loop #248
Conversation
|
This pull request was exported from Phabricator. Differential Revision: D40539580 |
Codecov Report
@@ Coverage Diff @@
## master #248 +/- ##
==========================================
+ Coverage 89.22% 89.34% +0.11%
==========================================
Files 85 85
Lines 5272 5330 +58
==========================================
+ Hits 4704 4762 +58
Misses 568 568
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Summary: Pull Request resolved: pytorch#248 Sometimes users need to recreate the dataloaders at the start of the epoch during the overall training loop. To support this, we allow users to register a creation function when creating the state for training or fitting. We pass the state as an argument since users may want to use progress information such as the progress counters to reinitialize the dataloader. We don't add this support for `evaluate` or `predict` since those functions iterate through the corresponding dataloader just once. For `fit`, this allows flexibility to reload training & evaluation dataloaders independently during if desired Differential Revision: D40539580 fbshipit-source-id: c67d7be4d20ac25f0b65927a20b525d86f8b56a4
ananthsub
force-pushed
the
export-D40539580
branch
from
0ca1a11 to
a636d89
Compare
|
This pull request was exported from Phabricator. Differential Revision: D40539580 |
|
Hi @ananthsub! Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours needs attention. You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
Summary:
Sometimes users need to recreate the dataloaders at the start of the epoch during the overall training loop. To support this, we allow users to register a creation function when creating the state for training or fitting. We pass the state as an argument since users may want to use progress information such as the progress counters to reinitialize the dataloader.
We don't add this support for
evaluateorpredictsince those functions iterate through the corresponding dataloader just once.For
fit, this allows flexibility to reload training & evaluation dataloaders independently during if desiredDifferential Revision: D40539580