Accelerator Refactor: Precision Plugins

[justusschock]

What does this PR do?

Adds precision plugins from #5616
To be merged after #5715

Co-Authored with @awaelchli

[pep8speaks]

Hello @justusschock! Thanks for updating this PR.

• In the file pytorch_lightning/plugins/precision/precision_plugin.py:

Line 26:121: E501 line too long (136 > 120 characters)
Line 40:121: E501 line too long (139 > 120 characters)

Comment last updated at 2021-01-31 17:46:16 UTC

[ananthsub]

@SeanNaren if we want to do ZeRO-v3 and offload more to CPU memory, is that handled within the training type plugin entirely? is there any dependency on the accelerator in which the plugin is contained?

[SeanNaren]

yeah handled explicitly within the training type plugin, or more technically, within the sharded DDP/OSS class itself. I think from what I recall, an offload_device will be exposed giving the user control (in this case, our accelerator control)

[SeanNaren]

ooo I didn't see this, this is nice :)

[Borda]

lets fix import...

[codecov]

Codecov Report

Merging #5718 (60445ad) into release/1.2-dev (3bacac7) will decrease coverage by 0%.
The diff coverage is n/a.

@@               Coverage Diff                @@
##           release/1.2-dev   #5718    +/-   ##
================================================
- Coverage               89%     89%    -0%     
================================================
  Files                  173     173            
  Lines                12495   12339   -156     
================================================
- Hits                 11175   11017   -158     
- Misses                1320    1322     +2