-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clean up dataloader logic #926
Merged
Merged
Changes from all commits
Commits
Show all changes
80 commits
Select commit
Hold shift + click to select a range
15bfd14
added get dataloaders directly using a getter
williamFalcon 6c7a37e
deleted decorator
williamFalcon 38e6991
added prepare_data hook
williamFalcon 37be7e7
refactored dataloader init
williamFalcon db1115e
refactored dataloader init
williamFalcon dbe3fc0
added dataloader reset flag and main loop
williamFalcon bb642b8
added dataloader reset flag and main loop
williamFalcon 6d646fd
added dataloader reset flag and main loop
williamFalcon 412899c
made changes
williamFalcon 252713f
made changes
williamFalcon de77ece
made changes
williamFalcon f723574
made changes
williamFalcon 9009c22
made changes
williamFalcon 74984e3
made changes
williamFalcon 838879b
made changes
williamFalcon 14f3a1d
made changes
williamFalcon 93f6b19
made changes
williamFalcon b021212
made changes
williamFalcon 45db9be
made changes
williamFalcon 20b5c62
made changes
williamFalcon 189bbb1
made changes
williamFalcon 0a45b1a
made changes
williamFalcon 767ad23
made changes
williamFalcon 56c0654
made changes
williamFalcon 783b5c7
made changes
williamFalcon 8183e82
made changes
williamFalcon 2a2a6ef
made changes
williamFalcon 8347a18
made changes
williamFalcon 176d62d
made changes
williamFalcon 4e3fb96
made changes
williamFalcon 51cc57f
made changes
williamFalcon c55bb0d
made changes
williamFalcon 6fe933b
made changes
williamFalcon d43c9e7
made changes
williamFalcon 05ab2db
made changes
williamFalcon d165b46
made changes
williamFalcon c5535e8
made changes
williamFalcon 1e56281
made changes
williamFalcon 7623a27
made changes
williamFalcon 36697f3
made changes
williamFalcon 803e72d
made changes
williamFalcon a8f3e19
made changes
williamFalcon 53598e2
made changes
williamFalcon cddbac8
made changes
williamFalcon e42b1b7
made changes
williamFalcon b83a7d7
made changes
williamFalcon d117727
made changes
williamFalcon 6e57368
made changes
williamFalcon de64175
made changes
williamFalcon 55d302d
made changes
williamFalcon df70d2e
made changes
williamFalcon 3635e61
made changes
williamFalcon f7a6382
made changes
williamFalcon abd2126
made changes
williamFalcon 90eef5e
fixed bad loaders
williamFalcon d047618
fixed bad loaders
williamFalcon eb9a380
fixed bad loaders
williamFalcon cb4f761
fixed bad loaders
williamFalcon cb8e977
fixed bad loaders
williamFalcon 82ec6ce
fixed bad loaders
williamFalcon 617dd32
fixed bad loaders
williamFalcon 97cf4c0
fixed bad loaders
williamFalcon 1671fbc
fixed bad loaders
williamFalcon 08eeb48
fixed error in .fit with loaders
williamFalcon d9cfcdb
fixed error in .fit with loaders
williamFalcon d2db8f2
fixed error in .fit with loaders
williamFalcon 35a8880
fixed error in .fit with loaders
williamFalcon cec5931
fixed error in .fit with loaders
williamFalcon 6c412ed
fixed error in .fit with loaders
williamFalcon 6287dd6
fixed error in .fit with loaders
williamFalcon dc403e8
fixed error in .fit with loaders
williamFalcon b2755f9
fixed error in .fit with loaders
williamFalcon 1be1cf6
fixed error in .fit with loaders
williamFalcon 83869c2
fixed error in .fit with loaders
williamFalcon 6bc9587
fixed error in .fit with loaders
williamFalcon 66f55d7
fixed error in .fit with loaders
williamFalcon 05ad57d
fixes #909
williamFalcon 9504461
fixes #909
williamFalcon c12cb92
bug fix
williamFalcon 3173ad3
Fixes #902
williamFalcon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
import traceback | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. add warning also here |
||
from functools import wraps | ||
import warnings | ||
|
||
|
||
def data_loader(fn): | ||
|
@@ -8,27 +9,9 @@ def data_loader(fn): | |
:param fn: | ||
:return: | ||
""" | ||
wraps(fn) | ||
attr_name = '_lazy_' + fn.__name__ | ||
@wraps(fn) | ||
def _get_data_loader(self): | ||
try: | ||
value = getattr(self, attr_name) | ||
except AttributeError: | ||
try: | ||
value = fn(self) # Lazy evaluation, done only once. | ||
if ( | ||
value is not None and | ||
not isinstance(value, list) and | ||
fn.__name__ in ['test_dataloader', 'val_dataloader'] | ||
): | ||
value = [value] | ||
except AttributeError as e: | ||
# Guard against AttributeError suppression. (Issue #142) | ||
traceback.print_exc() | ||
error = f'{fn.__name__}: An AttributeError was encountered: ' + str(e) | ||
raise RuntimeError(error) from e | ||
setattr(self, attr_name, value) # Memoize evaluation. | ||
return value | ||
w = 'data_loader decorator deprecated in 0.6.1. Will remove 0.8.0' | ||
warnings.warn(w) | ||
|
||
return _get_data_loader | ||
def inner_fx(self): | ||
return fn(self) | ||
return inner_fx |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,6 +16,13 @@ | |
from pytorch_lightning.core.memory import ModelSummary | ||
from pytorch_lightning.overrides.data_parallel import LightningDistributedDataParallel | ||
|
||
try: | ||
import torch_xla.core.xla_model as xm | ||
XLA_AVAILABLE = True | ||
|
||
except ImportError: | ||
XLA_AVAILABLE = False | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. rather
|
||
|
||
|
||
class LightningModule(ABC, GradInformation, ModelIO, ModelHooks): | ||
|
||
|
@@ -798,7 +805,9 @@ def optimizer_step(self, current_epoch, batch_idx, optimizer, optimizer_idx, sec | |
optimizer.zero_grad() | ||
|
||
""" | ||
if isinstance(optimizer, torch.optim.LBFGS): | ||
if self.trainer.use_tpu and XLA_AVAILABLE: | ||
xm.optimizer_step(optimizer) | ||
elif isinstance(optimizer, torch.optim.LBFGS): | ||
optimizer.step(second_order_closure) | ||
else: | ||
optimizer.step() | ||
|
@@ -868,7 +877,33 @@ def tbptt_split_batch(self, batch, split_size): | |
|
||
return splits | ||
|
||
@data_loader | ||
def prepare_data(self): | ||
"""Use this to download and prepare data. | ||
In distributed (GPU, TPU), this will only be called once | ||
|
||
:return: PyTorch DataLoader | ||
|
||
This is called before requesting the dataloaders | ||
|
||
.. code-block:: python | ||
|
||
model.prepare_data() | ||
model.train_dataloader() | ||
model.val_dataloader() | ||
model.test_dataloader() | ||
|
||
Example | ||
------- | ||
|
||
.. code-block:: python | ||
|
||
def prepare_data(self): | ||
download_imagenet() | ||
clean_imagenet() | ||
cache_imagenet() | ||
""" | ||
return None | ||
|
||
def train_dataloader(self): | ||
"""Implement a PyTorch DataLoader | ||
|
||
|
@@ -908,7 +943,6 @@ def tng_dataloader(self): # todo: remove in v0.8.0 | |
" and this method will be removed in v0.8.0", DeprecationWarning) | ||
return output | ||
|
||
@data_loader | ||
def test_dataloader(self): | ||
r""" | ||
|
||
|
@@ -942,7 +976,6 @@ def test_dataloader(self): | |
""" | ||
return None | ||
|
||
@data_loader | ||
def val_dataloader(self): | ||
r""" | ||
|
||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
duplicated