[QA] why not all examples run out of the box?

[matt32106]

In the last three months, i have tried several examples in R or python, with a number of versions of mxnet. Very often, the examples throw errors of one kind or another when you run them 'as is'.

Do you think it would be possible to make sure all examples run before accepting a PR?
In fact, why not use the various examples as test cases where results could be checked against expected values (using fixed seeds)?

Open issues with examples problems:

• [example][rcnn][python] cv2.error #5731,"[example][rcnn][python] cv2.error","2017-04-07T18:17:26Z"
• Running error in the example code of googlenet #5714,"Running error in the example code of googlenet","2017-04-06T15:56:31Z"
• Fix bugs in distributed example of TrainMnist.scala in MXNet-Scala #5648,"Fix bugs in distributed example of TrainMnist.scala in MXNet-Scala","2017-03-31T10:50:30Z"
• Exception when using CPP package with GPU (TitanX) with mlp() example #5633,"Exception when using CPP package with GPU (TitanX) with mlp() example","2017-03-30T11:19:44Z"
• cpp-package lenet example throws exception #5621,"cpp-package lenet example throws exception","2017-03-29T21:36:08Z"
• running lazy evaluation code example outputs "Illegal instruction" #5601,"running lazy evaluation code example outputs ""Illegal instruction""","2017-03-28T09:25:57Z"
• example/rcnn mistake--KeyError: 'width' #5596,"example/rcnn mistake--KeyError: 'width'","2017-03-28T03:06:48Z"
• example/ssd error Check failed: op != nullptr Operator MultiBoxPrior is not registered #5550,"example/ssd error Check failed: op != nullptr Operator MultiBoxPrior is not registered","2017-03-24T02:00:18Z"
• I have error in executing python_loss example #5480,"I have error in executing python_loss example","2017-03-19T10:17:51Z"
• Where is gen_data.py in example/bi-lstm-sort/ #5448,"Where is gen_data.py in example/bi-lstm-sort/","2017-03-17T02:47:37Z"
• issue in example warpctc/toy_ctc.py #5435,"issue in example warpctc/toy_ctc.py","2017-03-16T14:51:17Z"
• When I run ssd example on cpu computer, I get "Check failed: temp.size() >= num_negative (0 vs. 3) error" error. #5431,"When I run ssd example on cpu computer, I get ""Check failed: temp.size() >= num_negative (0 vs. 3) error"" error.","2017-03-16T12:03:55Z"
• Using MKL causes C++ layer blow up while running lstm_bucketing example #5314,"Using MKL causes C++ layer blow up while running lstm_bucketing example","2017-03-09T01:08:03Z"
• mxnet example training cannot get expected result after 0.5.0 #5209,"mxnet example training cannot get expected result after 0.5.0","2017-03-02T10:19:48Z"
• Missing gen_data.py in bi-lstm-sort example #5201,"Missing gen_data.py in bi-lstm-sort example","2017-03-01T22:38:39Z"
• mxnet-notebook example failed. #5196,"mxnet-notebook example failed.","2017-03-01T10:25:01Z"
• Fine-tune in R: end-to-end example and some troubles #5149,"Fine-tune in R: end-to-end example and some troubles","2017-02-25T18:54:38Z"
• [R] image classification examples are broken #5080,"[R] image classification examples are broken","2017-02-20T22:32:36Z"
• bugs in example/numpy-ops/custom_softmax.py #5068,"bugs in example/numpy-ops/custom_softmax.py","2017-02-20T04:30:23Z"
• MNIST training example not exiting on CPU instances after training completion #5065,"MNIST training example not exiting on CPU instances after training completion","2017-02-20T00:18:06Z"
• MXNET R, GPU speed-up much less for regression example than classification #5052,"MXNET R, GPU speed-up much less for regression example than classification","2017-02-18T04:50:29Z"
• The example/warpctc/ocr_predict.py has a bug, it does not provide init_states values and run the ocr_predict.py several times and will get different results.  #5047,"The example/warpctc/ocr_predict.py has a bug, it does not provide init_states values and run the ocr_predict.py several times and will get different results. ","2017-02-17T12:47:05Z"
• [scala-package] Running example on spark has the error java bin #4967,"[scala-package] Running example on spark has the error java bin","2017-02-10T09:02:05Z"
• mxnet example error #4956,"mxnet example error","2017-02-09T13:05:16Z"
• SSD example crashes on CPUs #4853,"SSD example crashes on CPUs","2017-02-01T22:33:14Z"
• Alexnet and googlenet symbol in example/image-classification is wrong #4826,"Alexnet and googlenet symbol in example/image-classification is wrong","2017-01-29T20:43:01Z"
• problem when running warp-ctc example on latest mxnet version #4668,"problem when running warp-ctc example on latest mxnet version","2017-01-14T03:03:47Z"
• about example error #4648,"about example error","2017-01-12T08:54:40Z"
• mxnet/example/autoencoder/mnist_sae.py not converge with the specified parameters #4640,"mxnet/example/autoencoder/mnist_sae.py not converge with the specified parameters","2017-01-11T17:52:58Z"
• CPP inference example is broken #4553,"CPP inference example is broken","2017-01-05T21:49:43Z"
• Get a core dump when trying to run the Scala example from the website #4540,"Get a core dump when trying to run the Scala example from the website","2017-01-05T11:03:38Z"
• Prediction fails in Handwritten Digit Recognition example #4318,"Prediction fails in Handwritten Digit Recognition example","2016-12-21T08:41:51Z"
• python example/gan/dcgan.py error #4299,"python example/gan/dcgan.py error","2016-12-20T04:34:13Z"
• imagenet example training error #4272,"imagenet example training error","2016-12-18T00:26:54Z"
• Something wrong when I ran the example of image-classification #4172,"Something wrong when I ran the example of image-classification","2016-12-09T15:10:45Z"
• example/cpp/image classfication can not work in Windows #4000," example/cpp/image classfication can not work in Windows","2016-11-27T04:46:32Z"
• Can't run example code with Python 3 #3995,"Can't run example code with Python 3","2016-11-27T01:41:11Z"
• problem with bi-lstm-sort example  #3965,"problem with bi-lstm-sort example ","2016-11-24T13:40:01Z"
• When I compiled mxnet with the ssd example, I got erro : #3948,"When I compiled mxnet with the ssd example, I got erro :","2016-11-23T08:08:15Z"
• SSD example has missing files #3805,"SSD example has missing files","2016-11-12T16:51:41Z"
• multi-gpu support of joint training of rcnn example is broken #3639,"multi-gpu support of joint training of rcnn example is broken","2016-10-27T10:20:15Z"
• IndentationError in the file example/fcn-xs/image_segmentaion.py line 22 #3571,"IndentationError in the file example/fcn-xs/image_segmentaion.py line 22","2016-10-19T06:27:29Z"
• empty shapes returned in RNN example #3477,"empty shapes returned in RNN example","2016-10-08T08:28:07Z"
• the bug of the adversary example #3315,"the bug of the adversary example","2016-09-18T02:46:17Z"
• make memcost example error #3094,"make memcost example error","2016-08-22T05:20:52Z"
• link in the example/kaggle-ndsb2/readme.md that describing how to build MXNet on AWS is invalid  #3061,"link in the example/kaggle-ndsb2/readme.md that describing how to build MXNet on AWS is invalid ","2016-08-18T10:41:30Z"
• Cannot run C++ example on Win7. #2792,"Cannot run C++ example on Win7.","2016-07-21T08:26:14Z"
• missing script in example bi-lstm-sort #2783,"missing script in example bi-lstm-sort","2016-07-20T12:56:05Z"
• RuntimeError when running neural-style examples #2758,"RuntimeError when running neural-style examples","2016-07-19T03:32:40Z"
• error of "Incorrect shape" in example/cpp/image-classification-predict.cc  #2709,"error of ""Incorrect shape"" in example/cpp/image-classification-predict.cc ","2016-07-15T01:50:09Z"
• MxNet/example/rcnn/demo.py error #2694,"MxNet/example/rcnn/demo.py error","2016-07-14T04:10:56Z"
• MXnet Neural art example error #2658,"MXnet Neural art example error","2016-07-09T03:54:30Z"
• example bug？ #2271,"example bug？","2016-05-29T03:48:19Z"
• Inception V3 model is different from the Tensorflow example #2235,"Inception V3 model is different from the Tensorflow example","2016-05-25T08:27:53Z"
• now I run the "example/image-classification" with two machines,long time nothing respond  #2117,"now I run the ""example/image-classification"" with two machines,long time nothing respond ","2016-05-12T04:52:19Z"
• adversary generation example: no module named data #2048,"adversary generation example: no module named data","2016-05-06T07:01:56Z"
• why my own gray scale image can't work use train_mnisy.py as example #1912,"why my own gray scale image can't work use train_mnisy.py as example","2016-04-20T15:39:46Z"
• fcnxs example validation accuracy does not change #1815,"fcnxs example validation accuracy does not change","2016-04-10T19:52:49Z"
• Mxnet gets stuck when run example/image-classification/train_mnist.py #1468,"Mxnet gets stuck when run example/image-classification/train_mnist.py","2016-02-14T10:36:57Z"
• example error: typestring mismatch #1115,"example error: typestring mismatch","2015-12-30T17:47:01Z"

[Bumblebee1964]

One important reason seems to me that documentation is not available or out-dated.

For instance the info on C++ API on mxnet,io can be considered as obsolete. The older version of this API had at least a VS solution and a complete set of header files so you can see how to build a working sample. On the latest version of the C++ API there is no VS solution anymore so you have to find out everything by trial and error (oruse the old version and copy relevant settings).

For us, users, it is kind of free so I do not like to complain. But if there is time and/or budget for the maintainers, some more up-to-date and less cryptic documentation would at least attract more people from SME's. It is for me hard to defend it took weeks to get a bare working MxNet with C++ interface on a Win10 system with a Titan X Board .

[futurely]

Examples also need unit tests.

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/image_retraining/retrain_test.py

https://github.com/tensorflow/models/blob/master/slim/nets/inception_resnet_v2_test.py

[szha]

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!

[matt32106]

I think this issue is still alive...

[data4science]

Has anyone looked at example code compatibility with mxnet-mkl & Python 3.6? I'm willing to if not.

[eric-haibin-lin]

@data4science thanks that will be a great help!

[data4science]

@matt32106 @eric-haibin-lin sounds good. My daily driver is MacOS, so I'm building from source. Let me know if anyone wants the resulting build. Then I'll go through the example code with Linux (Ubuntu 16.04, unless someone has a different distribution request). Does it mater if the Linux run is local or on a VPS (prob. AWS).

Assuming the example code works, I'll benchmark with the provided benchmark examples. Then I'll make a CSV file with columns for standard BLAS, MKL BLAS, & % improvement where rows are individual images. Finally, I'll create a visualization (type determined by data).

Does this sound like a decent plan of attack, or is MacOS build a bit unnecessary (heavy loads would likely be on VPS or cluster)?

[Bumblebee1964]

Great of you can make time to do this effort. Personally I am interested in c++ examples getting to work. Once I finished my adjusted version of InceptionBN in C++ (changing last layers, showing how to predict images in batch without the predict interface) l will contribute this.

[eric-haibin-lin]

@Bumblebee1964 any update for the cpp examples? :)
Someone is asking about batch mode in cpp https://discuss.mxnet.io/t/c-api-batch-mode/253/1

[data4science]

@eric-haibin-lin Sorry I got side tracked from the MKL examples. I’ll re-focus on them.

[data4science]

Skipping the MacOS and going straight to VPS btw. @eric-haibin-lin should the image include gpu(s)? My thought is no since MKL is optimized for Intel CPUs. Please recommend VPS image specs for benchmarking.

[Bumblebee1964]

Thanks @eric-haibin-lin for notifying me about https://discuss.mxnet.io/t/c-api-batch-mode/253/1 I will answer this thread later this week.

I have not yet figured out how to publish a sample on github and last weeks busy figuring out how to compile mxnet 0.11 with cpp-package and GPU support, which I recently accomplished.