[Dependency Update] Dependency update doc #15045

stu1130 · 2019-05-22T21:03:20Z

Description

document dependency update

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Comments

karan6181 · 2019-05-23T18:02:28Z

@mxnet-label-bot add [Doc, pr-awaiting-review]

vandanavk · 2019-06-16T19:23:19Z

@marcoabreu @perdasilva @aaronmarkham for review

karan6181 · 2019-07-19T00:27:35Z

@marcoabreu @perdasilva @aaronmarkham Gentle Ping!

tools/dependencies/README.md

aaronmarkham · 2019-07-23T16:18:38Z

tools/dependencies/README.md

+# Take CUDA 10 for example
+wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux
+chmod +x cuda_10.0.130_410.48_linux && sudo ./cuda_10.0.130_410.48_linux
+# Installation except:


What does this mean?

aaronmarkham · 2019-07-23T16:19:51Z

tools/dependencies/README.md

+### CUDA, cuDNN, NCCL
+#### 1. Environment Setup
+```
+# Take Ubuntu 16.04 for example


You could move this out of the code block and introduce what is going on here.

tools/dependencies/README.md

apeforest · 2019-07-23T17:50:08Z

tools/dependencies/README.md

+```
+# Download the ImageNet on http://image-net.org/download and make record file
+# Install prerequisite package
+pip2 install psutil --user


our benchmark script is written in python2 and not compatible with python3

apeforest · 2019-07-23T17:50:50Z

tools/dependencies/README.md

+
+# Install python frontend
+cd python
+pip3 install -e . --pre


maybe just pip? Let users to pick the right version (maybe thorugh viurtualenv)

tools/dependencies/README.md

apeforest · 2019-07-23T17:53:26Z

tools/dependencies/README.md

+# Wait for 10 - 30 mins, you will find libmxnet.so under the incubator-mxnet/lib
+
+# Install python frontend
+cd python


This can be simplified by pip install -e python

apeforest · 2019-07-23T17:54:25Z

tools/dependencies/README.md

+pip install -e . --pre
+# Test MXNet
+>>> import mxnet as mx
+>>> mx.nd.ones((2, 5) ctx=mx.gpu(0))


Maybe you can replace this test using the mxnet.runtime.feature_list()

apeforest · 2019-07-23T17:55:58Z

tools/dependencies/README.md

+# Clone the testing script
+git clone https://github.com/rahul003/deep-learning-benchmark-mirror.git
+# command 
+python2 benchmark_runner.py --framework mxnet --metrics-policy metrics_parameters_images_top_1 --task-name metrics_parameters_images_top_1 --metrics-suffix test --num-gpus 8 --command-to-execute 'python mxnet_benchmark/train_imagenet.py --use-rec --batch-size 128 --dtype float32 --num-data-workers 40 --num-epochs 3 --gpus 0,1,2,3,4,5,6,7 --lr 0.4 --warmup-epochs 5 --last-gamma --mode symbolic --model resnet50_v1b --rec-train /home/ubuntu/data/train-passthrough.rec --rec-train-idx /home/ubuntu/data/train-passthrough.idx --rec-val /home/ubuntu/data/val-passthrough.rec --rec-val-idx /home/ubuntu/data/val-passthrough.idx' --data-set data


This benchmark AI flow is outdated and I think it's also not public. Why not just run training script from example directly?

Also maybe use mnist as an example here. ImageNet is too big.

apeforest · 2019-07-24T18:11:04Z

tools/dependencies/README.md

+
+The dependencies could be categorized by several groups: BLAS libraries, CPU-based performance boost library, i.e. MKLDNN and GPU-based performance boosting library including CUDA, cuDNN, NCCL. and others including OpenCV, Numpy, S3-related, PS-lite dependencies. The list below shows all the dependencies and their version. Except for CUDA, cuDNN, NCCL which the user is required to install on their environments, we statically link those dependencies into libmxnet.so when we build PyPi package. By doing this, the user can take advantage of these dependencies without being worry about it.
+
+| Dependencies  | MXNet Version |


Maybe reword this? It's not MXNet version, but version of each package.

apeforest

small comment. otherwise LGTM. thanks for your contribution!

aaronmarkham · 2019-08-06T18:10:07Z

@stu1130 Did you want to make any further updates to this?

stu1130 · 2019-08-06T18:12:48Z

@aaronmarkham I think it's good to go

* first draft * 2nd draft * complete the draft * capitalize the comment * address the comment * address comment

stu1130 requested a review from szha as a code owner May 22, 2019 21:03

stu1130 changed the title ~~Dependecy update doc~~ [Dependency Update] Dependency update doc May 22, 2019

marcoabreu added Doc pr-awaiting-review PR is waiting for code review labels May 23, 2019

stu1130 force-pushed the dependecy_update_doc branch 4 times, most recently from 0185d13 to 9fca137 Compare May 31, 2019 06:44

stu1130 added 4 commits June 2, 2019 14:51

first draft

4b9b397

2nd draft

ec80ecf

complete the draft

e39329b

capitalize the comment

40dad43

stu1130 force-pushed the dependecy_update_doc branch from 9fca137 to 40dad43 Compare June 2, 2019 21:52

aaronmarkham suggested changes Jul 23, 2019

View reviewed changes

address the comment

b793ede

apeforest reviewed Jul 23, 2019

View reviewed changes

tools/dependencies/README.md Show resolved Hide resolved

apeforest reviewed Jul 23, 2019

View reviewed changes

stu1130 force-pushed the dependecy_update_doc branch from da90049 to 4c1c654 Compare July 23, 2019 19:11

address comment

09f6734

stu1130 force-pushed the dependecy_update_doc branch from 4c1c654 to 09f6734 Compare July 23, 2019 19:12

apeforest reviewed Jul 24, 2019

View reviewed changes

apeforest approved these changes Jul 24, 2019

View reviewed changes

aaronmarkham approved these changes Aug 6, 2019

View reviewed changes

aaronmarkham merged commit 51c07e5 into apache:master Aug 6, 2019

anirudhacharya pushed a commit to anirudhacharya/mxnet that referenced this pull request Aug 20, 2019

[Dependency Update] Dependency update doc (apache#15045)

69c0460

* first draft * 2nd draft * complete the draft * capitalize the comment * address the comment * address comment

stu1130 deleted the dependecy_update_doc branch January 10, 2020 03:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dependency Update] Dependency update doc #15045

[Dependency Update] Dependency update doc #15045

stu1130 commented May 22, 2019

karan6181 commented May 23, 2019

vandanavk commented Jun 16, 2019

karan6181 commented Jul 19, 2019

aaronmarkham Jul 23, 2019

stu1130 Jul 23, 2019

aaronmarkham Jul 23, 2019

apeforest Jul 23, 2019

stu1130 Jul 23, 2019 •

edited

Loading

apeforest Jul 23, 2019

apeforest Jul 23, 2019 •

edited

Loading

apeforest Jul 23, 2019

apeforest Jul 23, 2019

apeforest Jul 23, 2019

apeforest Jul 24, 2019 •

edited

Loading

apeforest left a comment

aaronmarkham commented Aug 6, 2019

stu1130 commented Aug 6, 2019


		The dependencies could be categorized by several groups: BLAS libraries, CPU-based performance boost library, i.e. MKLDNN and GPU-based performance boosting library including CUDA, cuDNN, NCCL. and others including OpenCV, Numpy, S3-related, PS-lite dependencies. The list below shows all the dependencies and their version. Except for CUDA, cuDNN, NCCL which the user is required to install on their environments, we statically link those dependencies into libmxnet.so when we build PyPi package. By doing this, the user can take advantage of these dependencies without being worry about it.

		\| Dependencies \| MXNet Version \|

[Dependency Update] Dependency update doc #15045

[Dependency Update] Dependency update doc #15045

Conversation

stu1130 commented May 22, 2019

Description

Checklist

Essentials

Changes

Comments

karan6181 commented May 23, 2019

vandanavk commented Jun 16, 2019

karan6181 commented Jul 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stu1130 Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apeforest Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apeforest Jul 24, 2019 • edited Loading

Choose a reason for hiding this comment

apeforest left a comment

Choose a reason for hiding this comment

aaronmarkham commented Aug 6, 2019

stu1130 commented Aug 6, 2019

stu1130 Jul 23, 2019 •

edited

Loading

apeforest Jul 23, 2019 •

edited

Loading

apeforest Jul 24, 2019 •

edited

Loading