Skip to content

Commit

Permalink
Update installation instructions for FairScale (#5099)
Browse files Browse the repository at this point in the history
Co-authored-by: Jirka Borovec <[email protected]>
  • Loading branch information
edenlightning and Borda authored Dec 12, 2020
1 parent 63fb7f9 commit 5f34f2b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/source/multi_gpu.rst
Original file line number Diff line number Diff line change
Expand Up @@ -663,7 +663,7 @@ It is highly recommended to use Sharded Training in multi-GPU environments where
A technical note: as batch size scales, storing activations for the backwards pass becomes the bottleneck in training. As a result, sharding optimizer state and gradients becomes less impactful.
Work within the future will bring optional sharding to activations and model parameters to reduce memory further, but come with a speed cost.

To use Sharded Training, you need to first install FairScale using the command below or install all extras using ``pip install pytorch-lightning["extra"]``.
To use Sharded Training, you need to first install FairScale using the command below.

.. code-block:: bash
Expand Down

0 comments on commit 5f34f2b

Please sign in to comment.