Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

training_forward assumes input has .copy() method for single GPU #553

Closed
MikeScarp opened this issue Nov 26, 2019 · 3 comments
Closed

training_forward assumes input has .copy() method for single GPU #553

MikeScarp opened this issue Nov 26, 2019 · 3 comments
Assignees
Labels
bug Something isn't working

Comments

@MikeScarp
Copy link
Contributor

Common bugs:

  1. Tensorboard not showing in Jupyter-notebook see issue 79.
  2. PyTorch 1.1.0 vs 1.2.0 support see FAQ

Describe the bug
After #532 , training with a DataLoader where batch does not have a .copy() method (such as tuple) will cause an exception. It doesn't seem to be the assumption that batch should always be a tensor, because we are passing it to transfer_batch_to_gpu, which does a lot of checking to handle different types differently.
Exception happens at https://github.com/williamFalcon/pytorch-lightning/blob/f2191b0cdf4305ae3a5ad2b1e404f99764a1a7c6/pytorch_lightning/trainer/train_loop_mixin.py#L293

To Reproduce
Steps to reproduce the behavior:

  1. Install bleeding edge build
  2. Train with an DataLoader where a batch is of type tuple with one GPU

Expected behavior
I would expect the copy to only happen when batch is of type tensor, or possibly apply the copy to elements inside the tuple if it is a tuple.

@MikeScarp MikeScarp added the bug Something isn't working label Nov 26, 2019
@williamFalcon
Copy link
Contributor

@MikeScarp good catch, silly oversight on our part. Mind submitting a PR?

@giovannipcarvalho
Copy link

It might not be a tuple either. I'm experiencing the same with torchtext's Batch class. In my case, data is already on the GPU.

@awaelchli
Copy link
Contributor

Looks to me like the problem described by @MikeScarp was solved. Master branch does not contain the copy anymore.

What @giovannipcarvalho mentions is also discussed here: #1245
So this may be closed? @MikeScarp

@Borda Borda closed this as completed Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants