Web edits

README.md

@@ -72,7 +72,7 @@ optimizations on advanced hyperparameter tuning and optimizers. For example:
 * DeepSpeed trains GPT2 (1.5 billion parameters) 3.75x faster than state-of-art, NVIDIA
   Megatron on Azure GPUs.
 
-  *Read more*: [GPT tutorial](./docs/tutorials/MegatronGPT2Tutorial.md)
+  *Read more*: [GPT tutorial](https://www.deepspeed.ai/tutorials/megatron/)
 
 
 
@@ -106,10 +106,10 @@ combination. ZeRO boosts the scaling capability and efficiency further.
   significant performance gains compared to using model parallelism alone.
 
   *Read more*: [technical report](https://arxiv.org/abs/1910.02054),
-  and [GPT tutorial](./docs/tutorials/MegatronGPT2Tutorial.md).
+  and [GPT tutorial](https://www.deepspeed.ai/tutorials/megatron/).
   <!-- and [QANet tutorial](../../Tutorials/QANetTutorial.md). -->
 
-![DeepSpeed-vs-Megatron](./docs/figures/DeepSpeed-vs-Megatron.png)
+![DeepSpeed-vs-Megatron](./docs/assets/images/DeepSpeed-vs-Megatron.png)
 <p align="center">
 <em>The figure depicts system throughput improvements of DeepSpeed (combining ZeRO-powered data parallelism with model parallelism of NVIDIA Megatron-LM) over using Megatron-LM alone.</em>
 </p>
@@ -121,7 +121,7 @@ optimizers such as [LAMB](https://arxiv.org/abs/1904.00962). These improve the
 effectiveness of model training and reduce the number of samples required to
 convergence to desired accuracy.
 
-*Read more*: [Tuning tutorial](./docs/tutorials/1Cycle.md),
+*Read more*: [Tuning tutorial](https://www.deepspeed.ai/tutorials/1Cycle/),
 <!---
  and *BERT Tutorial*: Coming Soon.
 
@@ -137,43 +137,43 @@ Only a few lines of code changes are needed to enable a PyTorch model to use Dee
 ## Features
 
 Below we provide a brief feature list, see our detailed [feature
-overview](./docs/features.md) for descriptions and usage.
+overview](https://www.deepspeed.ai/features/) for descriptions and usage.
 
-* [Distributed Training with Mixed Precision](./docs/features.md#distributed-training-with-mixed-precision)
+* [Distributed Training with Mixed Precision](https://www.deepspeed.ai/features/#distributed-training-with-mixed-precision)
     * 16-bit mixed precision
     * Single-GPU/Multi-GPU/Multi-Node
-* [Model Parallelism](./docs/features.md#model-parallelism)
+* [Model Parallelism](https://www.deepspeed.ai/features/#model-parallelism)
     * Support for Custom Model Parallelism
     * Integration with Megatron-LM
-* [Memory and Bandwidth Optimizations](./docs/features.md#memory-and-bandwidth-optimizations)
+* [Memory and Bandwidth Optimizations](https://www.deepspeed.ai/features/#memory-and-bandwidth-optimizations)
     * The Zero Redundancy Optimizer (ZeRO)
     * Constant Buffer Optimization (CBO)
     * Smart Gradient Accumulation
-* [Training Features](./docs/features.md#training-features)
+* [Training Features](https://www.deepspeed.ai/features/#training-features)
     * Simplified training API
     * Gradient Clipping
     * Automatic loss scaling with mixed precision
-* [Training Optimizers](./docs/features.md#training-optimizers)
+* [Training Optimizers](https://www.deepspeed.ai/features/#training-optimizers)
     * Fused Adam optimizer and arbitrary `torch.optim.Optimizer`
     * Memory bandwidth optimized FP16 Optimizer
     * Large Batch Training with LAMB Optimizer
     * Memory efficient Training with ZeRO Optimizer
-* [Training Agnostic Checkpointing](./docs/features.md#training-agnostic-checkpointing)
-* [Advanced Parameter Search](./docs/features.md#advanced-parameter-search)
+* [Training Agnostic Checkpointing](https://www.deepspeed.ai/features/#training-agnostic-checkpointing)
+* [Advanced Parameter Search](https://www.deepspeed.ai/features/#advanced-parameter-search)
     * Learning Rate Range Test
     * 1Cycle Learning Rate Schedule
-* [Simplified Data Loader](./docs/features.md#simplified-data-loader)
-* [Performance Analysis and Debugging](./docs/features.md#performance-analysis-and-debugging)
+* [Simplified Data Loader](https://www.deepspeed.ai/features/#simplified-data-loader)
+* [Performance Analysis and Debugging](https://www.deepspeed.ai/features/#performance-analysis-and-debugging)
 
 
 # Getting Started
 
 
 ## Installation
 
-* Please see our [Azure tutorial](docs/azure.md) to get started with DeepSpeed on Azure!
+* Please see our [Azure tutorial](https://www.deepspeed.ai/tutorials/azure/) to get started with DeepSpeed on Azure!
 * If you're not on Azure, we recommend using our docker image via `docker pull deepspeed/deepspeed:latest` which contains a pre-installed version of DeepSpeed and all the necessary dependencies.
-* If you want to install DeepSpeed manually, we provide an install script [install.sh](install.sh) to help install on a local machine or across an entire cluster.
+* If you want to install DeepSpeed manually, we provide an install script `install.sh` to help install on a local machine or across an entire cluster.
 
 ## Writing DeepSpeed Models
 DeepSpeed model training is accomplished using the DeepSpeed engine. The engine
@@ -280,7 +280,7 @@ the `step` value is stored as part of the `client_sd`.
 DeepSpeed features can be enabled, disabled, or configured using a config JSON
 file that should be specified as `args.deepspeed_config`. A sample config file
 is shown below. For a full set of features see [core API
-doc](https://microsoft.github.io/DeepSpeed/docs/htmlfiles/api/full/index.html).
+doc](https://deepspeed.readthedocs.io/en/latest/).
 
 ```json
 {
@@ -412,13 +412,13 @@ as the hostname.
 
 | Article                                                                                        | Description                                  |
 | ---------------------------------------------------------------------------------------------- | -------------------------------------------- |
-| [DeepSpeed Features](./docs/features.md)                                                       |  DeepSpeed features                          |
-| [DeepSpeed JSON Configuration](./docs/config_json.md)                                          |  Configuring DeepSpeed                       |
-| [API Documentation]( https://microsoft.github.io/DeepSpeed/docs/htmlfiles/api/full/index.html) |  Generated DeepSpeed API documentation       |
-| [CIFAR-10 Tutorial](./docs/tutorials/CIFAR-10.md)                                              |  Getting started with CIFAR-10 and DeepSpeed |
-| [Megatron-LM Tutorial](./docs/tutorials/MegatronGPT2Tutorial.md)                               |  Train GPT2 with DeepSpeed and Megatron-LM   |
-| [Learning Rate Range Test Tutorial](./docs/tutorials/lrrt.md)                                  |  Faster training with large learning rates   |
-| [1Cycle Tutorial](./docs/tutorials/1Cycle.md)                                                  |  SOTA learning schedule in DeepSpeed         |
+| [DeepSpeed Features](https://www.deepspeed.ai/features/)                                       |  DeepSpeed features                          |
+| [DeepSpeed JSON Configuration](https://www.deepspeed.ai/docs/config_json/)                     |  Configuring DeepSpeed                       |
+| [API Documentation](https://deepspeed.readthedocs.io/en/latest/)                               |  Generated DeepSpeed API documentation       |
+| [CIFAR-10 Tutorial](https://www.deepspeed.ai/tutorials/CIFAR-10)                               |  Getting started with CIFAR-10 and DeepSpeed |
+| [Megatron-LM Tutorial](https://www.deepspeed.ai/tutorials/megatron/)                           |  Train GPT2 with DeepSpeed and Megatron-LM   |
+| [Learning Rate Range Test Tutorial](https://www.deepspeed.ai/tutorials/lrrt/)                  |  Faster training with large learning rates   |
+| [1Cycle Tutorial](https://www.deepspeed.ai/tutorials/1Cycle/)                                  |  SOTA learning schedule in DeepSpeed         |
 
 
 

azure/README.md

@@ -1 +1 @@
-../docs/azure.md
+../docs/_tutorials/azure.md

docs/_config.yml

@@ -33,25 +33,28 @@ collections:
 defaults:
   - scope:
       path: ""
-      type: posts
     values:
       layout: single
       author_profile: false
       read_time: false
       comments: false
-      share: true
+      share: false
       related: false
       sneak_preview: false
-  # _tutorials
-  - scope:
-      path: ""
-      type: tutorials
-    values:
-      layout: single
       toc: true
       toc_label: "Contents"
       sidebar:
         nav: "lnav"
+  - scope:
+      path: "_pages"
+    values:
+      permalink: /docs/:title:output_ext
+  - scope:
+      path: ""
+      type: posts
+    values:
+      layout: single
+      share: true
 
 timezone: America/Los_Angeles
 breadcrumbs: true

docs/_data/navigation.yml

@@ -11,13 +11,33 @@ main:
     url: https://github.com/microsoft/DeepSpeed
 
 lnav:
-  - title: "DeepSpeed Documentation"
+  - title: "Feature Overview"
+    url: /features/
   - title: "Getting Started"
     url: /getting-started/
     children:
       - title: "Installation"
         url: /getting-started/#installation
-      - title: "Configuration"
-        url: /getting-started/#deepspeed-configuration
-  - title: "DeepSpeed Features"
-    url: /features/
+      - title: "Writing Models"
+        url: /getting-started/#writing-deepspeed-models
+      - title: "Training"
+        url: /getting-started/#training
+      - title: "Launching"
+        url: /getting-started/#launching-deepspeed-training
+  - title: "Configuration"
+    url: /docs/config_json/
+  - title: "Tutorials"
+    url: /tutorials/
+    children:
+      - title: "Getting Started on Azure"
+        url: /tutorials/azure/
+      - title: "CIFAR-10"
+        url: /tutorials/cifar-10/
+      - title: "Megatron-LM GPT2"
+        url: /tutorials/megatron/
+      - title: "1-Cycle Schedule"
+        url: /tutorials/1Cycle/
+      - title: "Learning Rate Range Test"
+        url: /tutorials/lrrt/
+  - title: "Contributing"
+    url: /contributing/

docs/_pages/config_json.md

@@ -0,0 +1,193 @@
+---
+title: "DeepSpeed Configuration JSON"
+permalink: /docs/config_json.html
+---
+## REQUIRED DeepSpeed Config JSON Parameters
+
+***train\_batch\_size***: [integer]
+
+| Value                                                        | Example |
+| ------------------------------------------------------------ | ------- |
+| The effective training batch size. This is the amount of data samples that leads to one step of model update. ***train\_batch\_size*** is aggregated by the batch size that a single GPU processes in one forward/backward pass (a.k.a., ***train\_step\_batch\_size***),  the gradient accumulation steps (a.k.a., ***gradient\_accumulation\_steps***), and the number of GPUs. | `32`      |
+
+## OPTIONAL DeepSpeed Config JSON Parameters
+
+### Batch Size Related Parameters
+
+***train\_micro\_batch\_size\_per\_gpu***: [integer]
+
+| Description                                                  | Default                      |
+| ------------------------------------------------------------ | ---------------------------- |
+| Batch size to be processed by one GPU in one step (without gradient accumulation). When specified, ***gradient\_accumulation\_steps*** is automatically calculated using ***train\_batch\_size*** and number of GPUs. Should not be concurrently specified with ***gradient\_accumulation\_steps*** in the configuration JSON. | ***train\_batch\_size*** value |
+
+***gradient\_accumulation\_steps***: [integer]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| Number of training steps to accumulate gradients before averaging and applying them. This feature is sometimes useful to improve scalability since it results in less frequent communication of gradients between steps. Another impact of this feature is the ability to train with larger batch sizes per GPU. When specified, ***train\_step\_batch\_size*** is automatically calculated using ***train\_batch\_size*** and number of GPUs. Should not be concurrently specified with ***train\_step\_batch\_size*** in the configuration JSON. | `1`       |
+
+
+
+### Optimizer Parameters
+
+***optimizer***: [dictionary]
+
+| Fields | Value                                                        | Example                        |
+| ------ | ------------------------------------------------------------ | ------------------------------ |
+| type   | The optimizer name. DeepSpeed natively supports Adam and LAMB optimizers and will import other optimizers from [torch](https://pytorch.org/docs/stable/optim.html). | `"Adam"`                         |
+| params | Dictionary of parameters to instantiate optimizer. The parameter names must match the optimizer constructor signature (e.g., for [Adam](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam)). | `{"lr": 0.001, "eps": 1e-8}` |
+
+  Example of ***optimizer***
+
+```json
+"optimizer": {
+    "type": "Adam",
+    "params": {
+      "lr": 0.001,
+      "betas": [
+        0.8,
+        0.999
+      ],
+      "eps": 1e-8,
+      "weight_decay": 3e-7
+    }
+  }
+```
+
+### Scheduler Parameters
+
+***scheduler***: [dictionary]
+
+| Fields | Value                                                        | Example                        |
+| ------ | ------------------------------------------------------------ | ------------------------------ |
+| type   | The scheduler name. See [here](https://deepspeed.readthedocs.io/en/latest/deepspeed.pt.html) for list of support schedulers. | `"1Cycle"`                      |
+| params | Dictionary of parameters to instantiate scheduler. The parameter names should match scheduler constructor signature. | `{"lr": 0.001, "eps": 1e-8}` |
+
+Example of ***scheduler***
+
+```json
+ "scheduler": {
+      "type": "WarmupLR",
+      "params": {
+          "warmup_min_lr": 0,
+          "warmup_max_lr": 0.001,
+          "warmup_num_steps": 1000
+      }
+  }  
+```
+
+### Communication options
+
+***fp32\_allreduce***: [boolean]
+
+| Description                          | Default |
+| ------------------------------------ | ------- |
+| During gradient averaging perform allreduce with 32 bit values | `false`   |
+
+***disable\_allgather***: [boolean]
+
+| Description                  | Default |
+| ---------------------------- | ------- |
+| Disable allgather when using ZeRO optimizer and instead use broadcast | `false`  
+
+***prescale\_gradients***: [boolean]
+
+| Description                            | Default |
+| -------------------------------------- | ------- |
+| Scale gradients before doing allreduce | `false`   |
+
+***sparse\_gradients***: [boolean]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| Enable sparse compression of [torch.nn.Embedding](https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding) gradients. | `false`    |
+
+
+### FP16 training options
+
+***zero\_optimization***: [boolean]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| Enable ZeRO memory optimization wrapper for FP16 Training. Currently compatible only with Adam optimizer. | `false`   |
+
+***fp16***: [dictionary]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| Configuration for using mixed precision/FP16 training that leverages [NVIDIA's Apex package](https://nvidia.github.io/apex/). An example, including the available dictionary keys is illustrated below. | None    |
+
+```json
+"fp16": {
+    "enabled": true,
+    "loss_scale": 0,
+    "initial_scale_power": 32,
+    "loss_scale_window": 1000,
+    "hysteresis": 2,
+ 	"min_loss_scale": 1
+}
+```
+
+***fp16:enabled***: [boolean]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| ***enabled*** is a **fp16** parameter indicating whether or not FP16 training enabled. | `false`   |
+
+***fp16:loss\_scale***: [float]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| ***loss\_scale*** is a ***fp16*** parameter representing the loss scaling value for FP16 training. The default value of 0.0 results in dynamic loss scaling, otherwise the value will be used for static fixed loss scaling. | `0.0`     |
+
+***fp16:initial\_scale\_power***: [integer]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| ***initial\_loss\_scale\_power*** is a **fp16** parameter representing the power of the initial dynamic loss scale value. The actual loss scale is computed as 2<sup>***initial\_loss\_scale\_power***</sup>. | `32`      |
+
+***fp16:loss\_scale\_window***: [integer]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| ***loss\_scale\_window*** is a **fp16** parameter representing the window over which to raise/lower the dynamic loss scale value. | `1000`    |
+
+***fp16:hysteresis***: [integer]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| ***hysteresis*** is a **fp16** parameter representing the delay shift in dynamic loss scaling. | `2`       |
+
+***fp16:min\_loss\_scale***: [integer]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| ***min\_loss\_scale*** is  a **fp16** parameter representing the minimum dynamic loss scale value. | `1000`    |
+
+### Gradient Clipping
+
+***gradient\_clipping***: [float]
+
+| Description                         | Default |
+| ----------------------------------- | ------- |
+| Enable gradient clipping with value | `0`      |
+
+### Logging
+
+***steps\_per\_print***: [integer]
+
+| Description | Default |
+| ----------- | ------- |
+| Print train loss every N steps | `10` |
+
+***wall\_clock\_breakdown***: [boolean]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| Enable timing of the latency of forward/backward/update training phases | `false`   |
+
+***dump_state***: [boolean]
+
+| Description                                                  | Default |
+| ------------------------------------------------------------ | ------- |
+| Print out state information of DeepSpeed object after initialization | `false`   |