wrap tpu tests with process decorator

[williamFalcon]

What does this PR do?

Fixes # (issue)

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[williamFalcon]

@Borda let's see if this works...
@dlibenzi @zcain117

[codecov]

Codecov Report

Merging #2582 into master will increase coverage by 2%.
The diff coverage is 18%.

@@           Coverage Diff            @@
##           master   #2582     +/-   ##
========================================
+ Coverage      89%     91%     +2%     
========================================
  Files          80      70     -10     
  Lines        7531    5770   -1761     
========================================
- Hits         6738    5252   -1486     
+ Misses        793     518    -275     

[Borda]

when was this function added? and never used till now?

[williamFalcon]

i added it a few days ago when the spawn stuff came out...

[williamFalcon]

@dlibenzi maybe the bug is about when we download data?

The current approach is:

• download on global rank = 0
• then spawn the multi-gpu stuff (.spawn())
  (this means inside the spawn call we don't download data)

Trace here (https://github.com/PyTorchLightning/pytorch-lightning/pull/2582/checks?check_run_id=861612147)

    dataloader = dataloader_fx()
  File "/pytorch-lightning/tests/base/model_valid_dataloaders.py", line 29, in val_dataloader__long
    num_samples=15000, digits=(0, 1, 2, 5, 8)), batch_size=32)
  File "/pytorch-lightning/tests/base/datasets.py", line 161, in __init__
    download=download
  File "/pytorch-lightning/tests/base/datasets.py", line 64, in __init__
    self.data, self.targets = torch.load(os.path.join(self.cached_folder_path, data_file))
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 854, in _load
    result = unpickler.load()
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 846, in persistent_load
    load_tensor(data_type, size, key, _maybe_decode_ascii(location))
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 834, in load_tensor
    storage = zip_file.get_storage_from_record(name, size, dtype).storage()
RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading file data/94756640: file read failed

Exception in device=TPU:4: unexpected EOF, expected 7791310 more bytes. The file might be corrupted.

[mergify]

This pull request is now in conflict... :(

[Borda]

@williamFalcon is this still valid? adding barriers looks reasonable...

[mergify]

This pull request is now in conflict... :(

[mergify]

This pull request is now in conflict... :(

[Borda]

implemented in #2632