Trainer: fix support for non-distributed PyTorch

[adamjstewart]

What does this PR do?

Before this change, Lightning Trainers did not work with PyTorch unless it was built with distributed support. On macOS, this is rarely the case. Using a trainer would result in the following error:

>       trainer.test(model=model, datamodule=datamodule)

tests/trainers/test_byol.py:63: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.spack/.spack-env/view/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:862: in test
    return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
../.spack/.spack-env/view/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:650: in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
../.spack/.spack-env/view/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:909: in _test_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
../.spack/.spack-env/view/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1166: in _run
    results = self._run_stage()
../.spack/.spack-env/view/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1249: in _run_stage
    return self._run_evaluate()
../.spack/.spack-env/view/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1294: in _run_evaluate
    with self.profiler.profile(f"run_{self.state.stage}_evaluation"), _evaluation_context(self.accelerator):
../.spack/.spack-env/._view/bye6vsynw4rzmsits3aqqhz6ipopfcbi/lib/python3.9/contextlib.py:119: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

accelerator = <pytorch_lightning.accelerators.cpu.CPUAccelerator object at 0x12ab0fa90>

    @contextmanager
    def _evaluation_context(accelerator: Accelerator) -> Generator:
        # inference mode is not supported with gloo backend (#9431),
        # and HPU & TPU accelerators.
        context_manager_class = (
            torch.inference_mode
>           if not (dist.is_initialized() and dist.get_backend() == "gloo")
            and not isinstance(accelerator, HPUAccelerator)
            and not isinstance(accelerator, TPUAccelerator)
            else torch.no_grad
        )
E       AttributeError: module 'torch.distributed' has no attribute 'is_initialized'

../.spack/.spack-env/view/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:2799: AttributeError

PyTorch's distributed API is slightly different from its CUDA API. If torch.distributed.is_available() returns False, then the rest of the API, including torch.distributed.is_initialized(), does not even exist. This PR first checks to make sure torch.distributed.is_available() is True before checking torch.distributed.is_initialized().

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

[awaelchli]

@adamjstewart Thanks! I checked and it does not look like we have other occurrences of this type. Would you mind adding an entry in the "Fixed" section of the CHANGELOG? Thanks