Another thing to merge. (MY EYES HURT)#1

Merged

bobisapotato merged 199 commits intobobisai:masterfrom

deepspeedai:master

Jan 24, 2021

Commits on Jul 21, 2020

only global rank 0 can log tensorboard data; avoid multi gpu/node race for the log directory (#296)
arashashari
authored

Commits on Jul 22, 2020

Commits on Jul 23, 2020

updates to amp to support grad clip and grad accumulation (#290)
jeffra
authored

Commits on Jul 24, 2020

Commits on Jul 25, 2020

DeepSpeed webinar announcement (#301)
Shaden Smith
authored

Commits on Jul 27, 2020

Update README.md (#302)
jeffra
authored

Commits on Jul 28, 2020

Commits on Aug 1, 2020

NameError: name 'mpu' is not defined (#305)
MannyKayy
authored

Commits on Aug 7, 2020

Removing () from assertion. (#307)
Shaden Smith
authored
Add webinar link (#309)
jeffra
authored

Commits on Aug 8, 2020

updates website gems after kramdown alert (#311)
Shaden Smith
authored

Commits on Aug 10, 2020

Fix+tests for get_lr from lr_scheduler before training starts (#310)
jeffra
authored

Commits on Aug 12, 2020

Commits on Aug 13, 2020

Commits on Aug 14, 2020

bump DSE (#317)
jeffra
authored

Commits on Aug 18, 2020

Turn off multi-node launch if only 1 node (#322)
jeffra
authored

Commits on Aug 27, 2020

Add code owners for DeepSpeed team (#335)
jeffra
authored

Commits on Aug 28, 2020

bump DSE
jeffra
committed

Commits on Aug 31, 2020

Update deepspeed_checkpointing.py (#336)

samyam
and
jeffra
authored

Commits on Sep 2, 2020

Commits on Sep 3, 2020

Commits on Sep 4, 2020

Jekyll installation instructions (#351)
Shaden Smith
authored

Commits on Sep 5, 2020

Commits on Sep 6, 2020

Update installation instructions (#362)
tjruwase
authored
Update Sparse Attention Tutorial (#357)

arashashari
and
arashashari
authored

Commits on Sep 10, 2020

ZeRO-Offload release (#391)

authored
Pipeline parallel training engine. (#392)

Shaden Smith
and
jeffra
authored
Update documentation for 1-bit Adam (#388)

awan-10
and
jeffra
authored
Fix datatype issue with sparse attention softmax (#363)

jeffra
and
Shaden Smith
authored
Add openmpi to dockerfile
jeffra
authored
ZeRO tutorials (#384)

authored
fix for 16GB v100 nodes (#393)
jeffra
authored
Sparse attention: updating code tag in documentation (#394)

arashashari
and
jeffra
authored
Minjiaz/zero offload (#382)

minjiaz
and
jeffra
authored
Adding sparse attention news index item (#376)

arashashari
and
jeffra
authored
Landing page updates (#395)

authored
Update README.md
jeffra
authored
Website edits (#398)

Shaden Smith
and
jeffra
authored
update docker image and bump DSE

jeffra
authored and
Ubuntu
committed
only add 1bit adam reqs if mpi is installed, update cond build for cpu-adam (#400)
jeffra
authored
bump DSE and doc tweak

jeffra
authored and
Ubuntu
committed
Update README.md
jeffra
authored
Update _config.yml
jeffra
authored
Update news site with press release link
jeffra
authored
Update ZeRO-Offload blog post link (#401)

authored
remove old pt file
jeffra
committed
readthedocs upgrade (#402)
Shaden Smith
authored

Commits on Sep 11, 2020

Commits on Sep 13, 2020

scales throughput by logging freq (#408)
Shaden Smith
authored

Commits on Sep 15, 2020

Commits on Sep 16, 2020

Add Linear warmup+decay lr schedule (#414)
tjruwase
authored
Minor doc fixes (#417)

authored
Overflow fix (#416)
Shaden Smith
authored
Fix a typo in comments (#415)

eric-haibin-lin
and
jeffra
authored

Commits on Sep 17, 2020

readthedocs yaml configuration (#410)

Shaden Smith
and
jeffra
authored
Fix few typos in the docs (#418)
gowthamvadisetti
authored
Remove pip --use-feature (#419)
jeffra
authored

Commits on Sep 18, 2020

Activation checkpointing bugfix and unit tests (#420)

Shaden Smith
and
jeffra
authored
Revert "Activation checkpointing bugfix and unit tests (#420)" (#422)

jeffra
and
ShadenSmith
authored
Fix activation checkpoint unit tests for GPU systems (#421)
Shaden Smith
authored

Commits on Sep 21, 2020

Commits on Sep 22, 2020

support dynamic sequence length in transformer kernels (#424)

RezaYazdaniAminabadi
and
conglongli
authored

Commits on Sep 24, 2020

Commits on Sep 25, 2020

Update pipeline.md (#439)
eric-haibin-lin
authored
link fix part two :-) (#441)
Shaden Smith
authored
unit test rename (#442)
Shaden Smith
authored

Commits on Sep 28, 2020

fix typos (#446)
eric-haibin-lin
authored

Commits on Sep 29, 2020

Disable default installation of CPU Adam (#450)
tjruwase
authored

Commits on Oct 1, 2020

Use parentesis around min and max to enable Windows build (#449)

authored

Commits on Oct 5, 2020

Update engine.py (#458)
Zixxy
authored

Commits on Oct 10, 2020

Add DeepSpeed_Adam optimizer (#468)

authored

Commits on Oct 12, 2020

Commits on Oct 14, 2020

Add support for p100 in transformer kernels (#470)
jeffra
authored

Commits on Oct 19, 2020

updating website dependencies (#475)
Shaden Smith
authored

Commits on Oct 30, 2020

Commits on Nov 5, 2020

Fixing CPU-Adam convergence issue (#503)
RezaYazdaniAminabadi
authored

Commits on Nov 9, 2020

Commits on Nov 10, 2020

Commits on Nov 11, 2020

Commits on Nov 12, 2020

Commits on Nov 14, 2020

Commits on Nov 18, 2020

Commits on Nov 20, 2020

Fix unbalanced gradients bug in ZeRO-2 gradient accumulation (#545)
tjruwase
authored

Commits on Nov 21, 2020

Support non-tensor state in checkpoint (#548)
tjruwase
authored

Commits on Nov 23, 2020

Commits on Nov 28, 2020

[doc] typo fix and clarification (#563)
stas00
authored

Commits on Dec 1, 2020

Commits on Dec 2, 2020

Commits on Dec 3, 2020

Commits on Dec 4, 2020

Fix potential random layout inconsistency issues in sparse attention modules (#534)

authored

Commits on Dec 7, 2020

Commits on Dec 8, 2020

Pipeline warnings and checkpoint portability (#588)
Shaden Smith
authored

Commits on Dec 14, 2020

implement missing get_last_lr (#595)

stas00
and
jeffra
authored

Commits on Dec 15, 2020

[doc] xref to hostfile discussion (#604)
stas00
authored
Fixes for RTD build errors (#606)

jeffra
and
Shaden Smith
authored

Commits on Dec 17, 2020

Transformer-kernel - supporting any arbitrary sequence-length (#587)

RezaYazdaniAminabadi
and
jeffra
authored

Commits on Dec 18, 2020

Ability to initialize distributed backend outside deepspeed runtime (#608)
jeffra
authored

Commits on Dec 23, 2020

Elastic training support (#602)

jeffra
and
samyam
authored

Commits on Jan 4, 2021

Commits on Jan 5, 2021

Commits on Jan 6, 2021

Module replacement support (#586)

authored

Commits on Jan 7, 2021

Commits on Jan 12, 2021

Handle actvitation checkpointing args that are None or non-tensors (#660)
Shaden Smith
authored

Commits on Jan 13, 2021

Commits on Jan 14, 2021

Validate consistent ckpt tags across ranks (#667)
jeffra
authored

Commits on Jan 19, 2021

add zero-offload paper (#680)
jeffra
authored

Commits on Jan 20, 2021

[tutorials] typos (#676)

stas00
and
jeffra
authored
make test_pipe more stable (#683)
Shaden Smith
authored
Fix ZeRO 2 + Pipelining (#677)
leogao2
authored