flaky ARMv7 build

[zheng-da]

CMake Error at /usr/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:137 (message):

  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)

Call Stack (most recent call first):

  /usr/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)

  /usr/share/cmake-3.11/Modules/FindOpenMP.cmake:451 (find_package_handle_standard_args)

  CMakeLists.txt:372 (find_package)



-- Configuring incomplete, errors occurred!

See also "/work/build/CMakeFiles/CMakeOutput.log".

See also "/work/build/CMakeFiles/CMakeError.log".

build.py: 2018-08-20 19:38:20,056 Running of command in container failed (1):

docker\

	run\

	--rm\

	-t\

	--shm-size=500m\

	-v\

	/home/jenkins_slave/workspace/build-ARMv7:/work/mxnet\

	-v\

	/home/jenkins_slave/workspace/build-ARMv7/build:/work/build\

	-v\

	/efs-ccache:/work/ccache\

	-u\

	1001:1001\

	-e\

	CCACHE_MAXSIZE=500G\

	-e\

	CCACHE_TEMPDIR=/tmp/ccache\

	-e\

	CCACHE_DIR=/work/ccache\

	-e\

	CCACHE_LOGFILE=/tmp/ccache.log\

	mxnetci/build.armv7\

	/work/runtime_functions.sh\

	build_armv7


build.py: 2018-08-20 19:38:20,056 You can get into the container by adding the -i option

Traceback (most recent call last):

  File "ci/build.py", line 394, in <module>

    sys.exit(main())

  File "ci/build.py", line 323, in main

    local_ccache_dir=args.ccache_dir, interactive=args.interactive)

  File "ci/build.py", line 210, in container_run

    raise subprocess.CalledProcessError(ret, cmd)

subprocess.CalledProcessError: Command 'docker\

	run\

	--rm\

	-t\

	--shm-size=500m\

	-v\

	/home/jenkins_slave/workspace/build-ARMv7:/work/mxnet\

	-v\

	/home/jenkins_slave/workspace/build-ARMv7/build:/work/build\

	-v\

	/efs-ccache:/work/ccache\

	-u\

	1001:1001\

	-e\

	CCACHE_MAXSIZE=500G\

	-e\

	CCACHE_TEMPDIR=/tmp/ccache\

	-e\

	CCACHE_DIR=/work/ccache\

	-e\

	CCACHE_LOGFILE=/tmp/ccache.log\

	mxnetci/build.armv7\

	/work/runtime_functions.sh\

	build_armv7' returned non-zero exit status 1

script returned exit code 1```

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12209/4/pipeline

[haojin2]

@KellenSunderland @marcoabreu @larroy @lebeg Please take a look when you have time.
@mxnet-label-bot [Build, Bug]

[lanking520]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12256/1/pipeline

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12256/2/pipeline

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12256/3/pipeline

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12256/4/pipeline

Failed 4 times... Looks like not a flaky but a consistent crash

[zheng-da]

I also realize that. I tried multiple times and failed every time.

[zheng-da]

Tried an empty PR and failed again.
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12259/1/pipeline/116

@marcoabreu could you fix it?

[KellenSunderland]

I kicked off a PR that disables the test here: #12260.
If a committer could keep their eye on those tests and merge when everything passes it should unblock PRs.

Investigated the root cause, and while it must be some platform change that we depend on, I couldn't nail down the exact cause. We'll take a closer look tomorrow.

Edit: By the way if you need to urgently work on a PR you can cherry-pick that commit and it should allow you to test your PRs properly. Later when the commit is merged your PR will be able to be fast fwd'd without re-running tests. Should be something along the lines of:

git remote add temp_remote https://github.com/KellenSunderland/incubator-mxnet.git
git fetch --all
git cherry-pick 4be2325

[anirudhacharya]

one more instance - http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/incubator-mxnet/branches/PR-12230/runs/5/nodes/116/log/?start=0

[KellenSunderland]

All passed, can someone merge #12260

[haojin2]

@KellenSunderland it was merged just now.