Merge the latest ds branch to forked branch#3

Merged

trajepl merged 38 commits intotrajepl:masterfrom

deepspeedai:master

Aug 1, 2022

Commits on Jul 21, 2022

Commits on Jul 25, 2022

Commits on Jul 26, 2022

Commits on Jul 28, 2022

Trajepl/nebula ckpt engine (#2085)

authored
Graceful exit on failures for multi-node runs (#2008)

Jerry Mannil
and
tjruwase
authored
fix: fix BF16_Optimizer compatibility issue with optimizer state 0-dim tensor (#2152)

shjwudp
and
tjruwase
authored

Commits on Jul 29, 2022

Fix random token-generation issue + MP-checkpoint loading/saving (#2132)

authored
Added retain_graph as a kwarg to the main engine backward function (#1149)

ncilfone
and
jeffra
authored
Elastic Training support in DeepSpeed (#2153) (#2156)

authored
prevent cuda 10 builds of inference kernels on ampere (#2157)
jeffra
authored

Commits on Jul 30, 2022

Commits on Jul 31, 2022

Release swap buffers for persisted params (#2089)
tjruwase
authored

Commits on Aug 1, 2022

Tensor parallelism for Mixture of Experts (#2074)

authored