From ec9686a9e800f7008ce51f925e34b499929eb9af Mon Sep 17 00:00:00 2001 From: Stas Bekman Date: Thu, 7 Jan 2021 15:13:36 -0800 Subject: [PATCH 1/3] document deepspeed.initialize() --- docs/_tutorials/getting-started.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/_tutorials/getting-started.md b/docs/_tutorials/getting-started.md index 21268802d6c8..72b90a3e59d1 100644 --- a/docs/_tutorials/getting-started.md +++ b/docs/_tutorials/getting-started.md @@ -31,6 +31,20 @@ construct and manage the training optimizer, data loader, and the learning rate scheduler based on the parameters passed to `deepspeed.initialize` and the DeepSpeed [configuration file](#deepspeed-configuration). +If you already have a distributed environment setup, you'd need to replace: + +```python +torch.distributed.init_process_group(...) +``` + +with: + +```python +deepspeed.init_distributed() +``` + +But if you don't need the distributed environment setup until after `deepspeed.initialize()` you don't have to use this function, as DeepSpeed will automatically initialize the distributed environment during its `initialize`. Regardless, you will need to remove `torch.distributed.init_process_group` if you already had one. + ### Training From d60484551343df2f78c108d73f6d07d3ce28f33b Mon Sep 17 00:00:00 2001 From: Stas Bekman Date: Thu, 7 Jan 2021 15:39:50 -0800 Subject: [PATCH 2/3] add a note that NCCL is used by default --- docs/_tutorials/getting-started.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/_tutorials/getting-started.md b/docs/_tutorials/getting-started.md index 72b90a3e59d1..39a013bc0ef8 100644 --- a/docs/_tutorials/getting-started.md +++ b/docs/_tutorials/getting-started.md @@ -43,7 +43,9 @@ with: deepspeed.init_distributed() ``` -But if you don't need the distributed environment setup until after `deepspeed.initialize()` you don't have to use this function, as DeepSpeed will automatically initialize the distributed environment during its `initialize`. Regardless, you will need to remove `torch.distributed.init_process_group` if you already had one. +The default is to use the NCCL backend, which DeepSpeed has been thoroughly tested with, but other backends can be passed as an argument to `init_distributed`. + +But if you don't need the distributed environment setup until after `deepspeed.initialize()` you don't have to use this function, as DeepSpeed will automatically initialize the distributed environment during its `initialize`. Regardless, you will need to remove `torch.distributed.init_process_group` if you already had it in place. ### Training From 866323252009067ae7e32ee7ec4b7bfa5c777eac Mon Sep 17 00:00:00 2001 From: Stas Bekman Date: Thu, 7 Jan 2021 16:48:10 -0800 Subject: [PATCH 3/3] add a link to init_distributed doc --- docs/_tutorials/getting-started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/_tutorials/getting-started.md b/docs/_tutorials/getting-started.md index 39a013bc0ef8..5c53f2084f3c 100644 --- a/docs/_tutorials/getting-started.md +++ b/docs/_tutorials/getting-started.md @@ -43,7 +43,7 @@ with: deepspeed.init_distributed() ``` -The default is to use the NCCL backend, which DeepSpeed has been thoroughly tested with, but other backends can be passed as an argument to `init_distributed`. +The default is to use the NCCL backend, which DeepSpeed has been thoroughly tested with, but you can also [override the default](https://deepspeed.readthedocs.io/en/latest/initialize.html#distributed-initialization). But if you don't need the distributed environment setup until after `deepspeed.initialize()` you don't have to use this function, as DeepSpeed will automatically initialize the distributed environment during its `initialize`. Regardless, you will need to remove `torch.distributed.init_process_group` if you already had it in place.