Skip to content

document deepspeed.initialize() #644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeffra merged 5 commits into from
Jan 8, 2021
Merged

document deepspeed.initialize() #644

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ec9686a
document deepspeed.initialize()
stas00 Jan 7, 2021
d604845
add a note that NCCL is used by default
stas00 Jan 7, 2021
1a128db
Merge branch 'master' into dist-init
jeffra Jan 7, 2021
8663232
add a link to init_distributed doc
stas00 Jan 8, 2021
59ebe3b
Merge branch 'dist-init' of github.com:stas00/DeepSpeed into dist-init
stas00 Jan 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/_tutorials/getting-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@ construct and manage the training optimizer, data loader, and the learning rate
scheduler based on the parameters passed to `deepspeed.initialize` and the
DeepSpeed [configuration file](#deepspeed-configuration).

If you already have a distributed environment setup, you'd need to replace:

```python
torch.distributed.init_process_group(...)
```

with:

```python
deepspeed.init_distributed()
```

But if you don't need the distributed environment setup until after `deepspeed.initialize()` you don't have to use this function, as DeepSpeed will automatically initialize the distributed environment during its `initialize`. Regardless, you will need to remove `torch.distributed.init_process_group` if you already had one.


### Training

Expand Down