[Nightly test] v1.3.x failing with missing cmake

[jlcontreras]

The step InstallationGuide:GPU is failing in the nightly tests due to missing cmake. The problems started on Jan 3rd.

Example case:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTests/detail/v1.3.x/146/pipeline

make[1]: Leaving directory '/work/mxnet/incubator-mxnet/3rdparty/tvm/nnvm'
Downloaded and unpacked Intel(R) MKL small libraries to /work/mxnet/incubator-mxnet/3rdparty/mkldnn/external
cmake /work/mxnet/incubator-mxnet/3rdparty/mkldnn -DCMAKE_INSTALL_PREFIX=/work/mxnet/incubator-mxnet/3rdparty/mkldnn/build/install -B/work/mxnet/incubator-mxnet/3rdparty/mkldnn/build -DARCH_OPT_FLAGS="-mtune=generic" -DWITH_TEST=OFF -DWITH_EXAMPLE=OFF
/bin/sh: 1: cmake: not found
mkldnn.mk:38: recipe for target '/work/mxnet/incubator-mxnet/3rdparty/mkldnn/build/install/lib/libmkldnn.so.0' failed
make: *** [/work/mxnet/incubator-mxnet/3rdparty/mkldnn/build/install/lib/libmkldnn.so.0] Error 127
make: *** Waiting for unfinished jobs....
ar cr libdmlc.a line_split.o indexed_recordio_split.o recordio_split.o input_split_base.o io.o filesys.o local_filesys.o data.o recordio.o config.o
make[1]: Leaving directory '/work/mxnet/incubator-mxnet/3rdparty/dmlc-core'
build.py: 2019-01-06 19:51:59,687 Running of command in container failed (2):

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Test

[jlcontreras]

@TaoLv Do you think this could be related to the latest MKL updates?

[TaoLv]

Do you mean the v1.3.x branch? I notice the last commit on it is more than 1 month ago.

[TaoLv]

With v1.3.x branch, I don't think command line like make -j 32 USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 will trigger MKL-DNN build.

[jlcontreras]

Thanks for the reply :)
I did mean the v1.3.x branch, editing the title. Maybe #13681 somehow had something to do? Dates coincide, but that PR shouldn't impact v1.3.x

[TaoLv]

Yes, from that PR, we have MKL-DNN default on master branch. It definitely should not impact branch v1.3.x. So I guess the source code you built was not that one you checked out from v1.3.x.

[zachgk]

Thank you for submitting the issue! I'm labeling it so the MXNet community members can help resolve it.

@mxnet-label-bot add [CI]

[TaoLv]

@jlcontreras Is the problem still there? Could you double check the code base used for InstallationGuide: GPU. I notice the submodule update message in the log:

Submodule path '3rdparty/tvm': checked out '0f053c82a747b4dcdf49570ec87c17e0067b7439'

apache/tvm@0f053c8 is used on the master branch while on v1.3.x it's apache/tvm@90db723 .

[jlcontreras]

Yes, the problem still persists. I'll update this issue with any progress made. Thanks for the info about the submodule update!

[jlcontreras]

Found why we are checking out the wrong version of tvm. The InstallationGuide step in CI executes the commands from docs/install/index.md . One of the steps there is the following:

$ git clone --recursive https://github.com/apache/incubator-mxnet
$ cd incubator-mxnet
$ make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas

So we are actually testing master, even from this v1.3.x job.

[TaoLv]

@jlcontreras issue resolved?

[perdasilva]

@TaoLv not yet. The issue seems to come from the following command:

eval sudo apt-get 'update;' sudo apt-get install -y build-essential 'git;' sudo apt-get install -y libopenblas-dev 'liblapack-dev;' sudo apt-get install -y 'libopencv-dev;' git clone --recursive 'https://github.com/apache/incubator-mxnet;' cd 'incubator-mxnet;' make -j '$(nproc)' USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda 'USE_CUDNN=1;' sudo apt-get install -y python-dev python-setuptools python-pip 'libgfortran3;' cd 'python;' pip install -e '.;'

It seems we are testing whether the v1.3.x documentation has the instructions to compile the master branch. I guess there must have been some changes in the dependencies on master, which have updated the documentation (therefore the master build passes). We need to figure out a way to test the documentation instructions against the code in branch in which those instructions reside.

[piyushghai]

@jlcontreras I see a PR merged that seems to have fixed this issue.
Do you think this issue is good to close now ?

[perdasilva]

@piyushghai should be fine to close ^^

[lanking520]

Close this issue for now. Please feel free to reopen it if you are facing more problems with it.