Samyamr/zero offload correctness#359

Merged

samyam merged 2 commits intostaging-zero-dual-v2from

samyamr/zero-offload-correctness

Sep 4, 2020

Commits on Sep 3, 2020

fixing gradient accumulation for zero offload
samyam
committed

Commits on Sep 4, 2020

Bug fixes. ZeRO Stage 1,2 and Offload all produce the same loss with gradient accumulation step of 2
samyam
committed