Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

flaky test: test_operator.test_depthwise_convolution #14052

Closed
mseth10 opened this issue Feb 2, 2019 · 5 comments · May be fixed by #14452
Closed

flaky test: test_operator.test_depthwise_convolution #14052

mseth10 opened this issue Feb 2, 2019 · 5 comments · May be fixed by #14452

Comments

@mseth10
Copy link
Contributor

mseth10 commented Feb 2, 2019

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-14016/4/pipeline/

ERROR: test_operator.test_depthwise_convolution

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/work/mxnet/tests/python/unittest/common.py", line 173, in test_new
orig_test(*args, **kwargs)
File "/work/mxnet/tests/python/unittest/test_operator.py", line 1676, in test_depthwise_convolution
np.testing.assert_allclose(arr1.asnumpy(), arr2.asnumpy(), rtol=1e-3, atol=1e-3)
File "/work/mxnet/python/mxnet/ndarray/ndarray.py", line 1995, in asnumpy
ctypes.c_size_t(data.size)))
File "/work/mxnet/python/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
MXNetError: [01:40:38] src/operator/nn/mkldnn/mkldnn_base.cc:576: Check failed: similar

NOTE: The test fails on CPU when MKLDNN is enabled and passes when MKL-DNN is disabled. So, the issue is mostly due to MKL-DNN.

@mxnet-label-bot
Copy link
Contributor

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Test, Flaky

@vdantu
Copy link
Contributor

vdantu commented Feb 2, 2019

@mxnet-label-bot add [mkldnn, test, flaky]

@ZhennanQin
Copy link
Contributor

@mseth10, Thanks for reporting this. Confirmed it's caused by mkldnn bug. The fix is already in mkldnn master, but not available in its latest release. We will upgrade mkldnn when its next release comes out.

@mseth10
Copy link
Contributor Author

mseth10 commented Feb 25, 2019

@ZhennanQin Thanks for confirming it's an MKLDNN bug. Good to know that it's fixed. Can you also give an ETA on when the next release containing the fix be rolled out?

@pengzhao-intel
Copy link
Contributor

Resolved so closing.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants