Add cifar10 example using TE+DeepSpeed-Zero+MS-AMP (#143)

**Description**
Add cifar10 example using TE+DeepSpeed-Zero+MS-AMP
There is a 1P customer who only uses DeepSpeed ZeRO and the model is
transformer based. Purely replacing Linear with FP8Linear will not get
performance gain. So we suggest them to use TransformerEngine for
speeding up and MS-AMP for saving memory. This is a good reference to
them.

**Major Revision**
- Add example
- Add document
- Fix a bug in mnist_ddp example

Author: tocean

.gitignore

@@ -138,3 +138,5 @@ dmypy.json
 
 # Cython debug symbols
 cython_debug/
+
+examples/data

docs/getting-started/run-msamp.md

@@ -40,4 +40,10 @@ deepspeed cifar10_deepspeed.py --deepspeed --deepspeed_config ds_config_msamp.js
 deepspeed cifar10_deepspeed.py --deepspeed --deepspeed_config ds_config_zero_msamp.json
 ```
 
+### 4. Run cifar10 using deepspeed-ZeRO + TE with msamp enabled
+
+```bash
+deepspeed cifar10_deepspeed_te.py --deepspeed --deepspeed_config ds_config_zero_te_msamp.json
+```
+
 For more comprehensive examples, please go to [MS-AMP-Examples](https://github.com/Azure/MS-AMP-Examples).

docs/user-tutorial/usage.md

@@ -31,11 +31,13 @@ For enabling MS-AMP in DeepSpeed, add one line of code `from msamp import deepsp
 ```json
 "msamp": {
   "enabled": true,
-  "opt_level": "O1|O2|O3"
+  "opt_level": "O1|O2|O3",
+  "use_te": false
 }
 ```
 
 "O3" is designed for FP8 in ZeRO optimizer, so please make sure ZeRO is enabled when using "O3".
+"use_te" is designed for Transformer Engine, if you have already used Transformer Engine in your model, don't forget to set "use_te" to true.
 
 ## Usage in Megatron-DeepSpeed and Megatron-LM