MXNet GPU MKLDNN Build failure

[roywei]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-15082/2/pipeline

This is currently blocking all PRs in CI.

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Build

[jlcontreras]

Thanks for pointing it out! We are having a look to try and identify the problem. Any ideas what might have caused it?

[frankfliu]

@mxnet-label-bot add [mkldnn, build, CI]

[aaronmarkham]

Thanks for pointing it out! We are having a look to try and identify the problem. Any ideas what might have caused it?

If we look at the trend, there's a cut off point. Should these commits be rolled back?
http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/job/unix-gpu/job/master/

[roywei]

see detailed discussion in #15099
We need to keep this open until root cause found.

As next step maybe try remove some build flags and see if it helps.

[anirudh2290]

I tried again with

make DEV=1 ENABLE_TESTCOVERAGE=1 USE_CPP_PACKAGE=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 'CUDA_ARCH=-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70' USE_SIGNAL_HANDLER=1 -j$(nproc)

and now I am not able to reproduce the bug :). Would be really helpful to me if you can try the same @roywei .

In the meanwhile I am looking more into this, seeing if something in the archive utility could have caused this.

[anirudh2290]

This seems to be related to the archive utility ar bug: https://sourceware.org/bugzilla/show_bug.cgi?id=14625 . It has been fixed in a later version of archive utility in this patch : https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=e6cc316af931911da20249e19f9342e5cf8aeeff. I am trying to reproduce the issue and have also downloaded the binutils source for the version where the issue is fixed. It is very likely that the mxnet archive crossed the 4G limit. If I am able to confirm this, this will require the CI to use a newer version of the archive utility.

[anirudh2290]

Alright I was able to reproduce the issue with my commit and I was able to fix it on the commit when using ar version 2.27: GNU ar (GNU Binutils) 2.27. The CI needs to be upgraded to this AR version.
@marcoabreu @jlcontreras @lebeg @perdasilva @Chancebair Please help with the upgrade. This is blocking an important perf regression fix currently and can reoccur: #15033

[roywei]

Maybe we can add a script to upgrade the archive utility during the build stage of GPU: MKLDNN if this only happens for this set of build flags? This will ensure all auto-scaled machines to be upgraded.

[anirudh2290]

@roywei yes good point. This will be required in all stages sooner or later, but right now gpu stages should be enough I think.

[Chancebair]

Our CI instances use Ubuntu 16.04 so the latest version of ar available is 2.26

sudo apt-get install binutils
Reading package lists... Done
Building dependency tree
Reading state information... Done
binutils is already the newest version (2.26.1-1ubuntu1~16.04.8).
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.

[larroy]

Would upgrading the containers to 18.04 alleviate this and help move forward our CI?
I think is better to not deviate from the distribution if possible.