TPU available: true when there are no TPUs

[dalmia]

🐛 Bug

I am using a DGX machine (and so, no TPUs), but on initiating Trainer, it logs TPU available: True. This ends up returning Missing XLA configuration when I run my script.

To Reproduce

Code sample

Simply running the following lines on my machine:

>> trainer = pl.Trainer(gpus=[0])                                                                                                                 
GPU available: True, used: True
TPU available: True, using: 0 TPU cores

Expected behavior

>> trainer = pl.Trainer(gpus=[0])                                                                                                                 
GPU available: True, used: True
TPU available: False, using: 0 TPU cores

Environment

* CUDA:
        - GPU:
                - Tesla V100-SXM2-32GB
        - available:         True
        - version:           10.2
* Packages:
        - numpy:             1.18.2
        - pyTorch_debug:     False
        - pyTorch_version:   1.6.0
        - pytorch-lightning: 0.9.0
        - tensorboard:       2.2.0
        - tqdm:              4.45.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - 
        - processor:         x86_64
        - python:            3.6.9
        - version:           #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019

[Borda]

sounds like some misconfiguration issue, are interested in sending a PR? 🐰

[dalmia]

Sure. I realized that the bug is in this script.

Specifically:

try:
    import torch_xla.core.xla_model as xm
except ImportError:
    XLA_AVAILABLE = False
else:
    XLA_AVAILABLE = True

So, if the environment has torch_xla installed but no TPU, then this error is thrown. If I use an environment without torch_xla, it works fine. So, is this something that should be fixed in the codebase or something that the user should take care of? @Borda

[Borda]

yes, we had the XLA detection as a temporal solution as we did not expect someone would install XLA without having TPU...
so, pls send a PR, I think that we have this patter in several files...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alimoezzi]

I faced this issue on gcloud vm with gpu but not tpu.

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/xla_device.py", line 32, in inner_f
    queue.put(func(*args, **kwargs))
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/xla_device.py", line 73, in _is_device_tpu
    return (xm.xrt_world_size() > 1) or bool(xm.get_xla_supported_devices("TPU"))
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/core/xla_model.py", line 137, in get_xla_supported_devices
    xla_devices = _DEVICES.value
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/utils/utils.py", line 32, in value
    self._value = self._gen_fn()
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/core/xla_model.py", line 19, in <lambda>
    _DEVICES = xu.LazyProperty(lambda: torch_xla._XLAC._xla_get_devices())
RuntimeError: tensorflow/compiler/xla/xla_client/computation_client.cc:273 : Missing XLA configuration

The torch_xla was installed by default in the Deep learning image and the only solution was to manually uninstall torch_xla
I used PL latest version 1.6.4

[apezzi]

@realsarm this worked for me, on gcloud with one gpu but not tpu.
On the shell launched pip uninstall torch_xla.

Thanks a lot :)