[Flaky] test_mkldnn.test_activation

[reminisce]

======================================================================

FAIL: test_mkldnn.test_activation

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/mkl/../unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/mkl/test_mkldnn.py", line 350, in test_activation

    check_activation_training(stype)

  File "/work/mxnet/tests/python/mkl/test_mkldnn.py", line 346, in check_activation_training

    check_numeric_gradient(test, in_location, numeric_eps=1e-5, rtol=0.16, atol=1e-4)

  File "/work/mxnet/python/mxnet/test_utils.py", line 1015, in check_numeric_gradient

    ("NUMERICAL_%s"%name, "BACKWARD_%s"%name))

  File "/work/mxnet/python/mxnet/test_utils.py", line 533, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error 1.609589 exceeds tolerance rtol=0.160000, atol=0.000100.  Location of maximum error:(0, 1, 0, 0), a=0.523432, b=0.705208

 NUMERICAL_data: array([[[[0.1385808 , 0.        ],

         [0.923872  , 0.58710575]],

...

 BACKWARD_data: array([[[[0.13833651, 0.        ],

         [0.9269223 , 0.58610183]],

...

@PatricZhao Any idea? Thanks.

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended label(s): Test, Flaky

[pengzhao-intel]

@ElaineBao could you help take a look?

[zixuanweeei]

FYI @ElaineBao , using MXNET_TEST_SEED=851486559 could produce failed test of this. And I also tested it with MXNET_TEST_COUNT=100, MXNET_TEST_SEED=42, which results to the test passed.

[ElaineBao]

FYI @ElaineBao , using MXNET_TEST_SEED=851486559 could produce failed test of this. And I also tested it with MXNET_TEST_COUNT=100, MXNET_TEST_SEED=42, which results to the test passed.

@zixuanweeei Thank you for the test! I'll reproduce the case.

[ElaineBao]

Hi, @zixuanweeei , your case has been solved in #16133, it's caused by the discontinuous gradient at the origin.
Hi, @reminisce , your case seems not happen near the origin, but unfortunately I run for 10000 times and cannot reproduce the case. Can you please provide the MXNET_TEST_SEED of that case? Thank you.

[reminisce]

@ElaineBao Thanks for checking it. Unfortunately, the log was overwritten by the follow-up run. I will report it once I see it again.

[pengzhao-intel]

closing since we don't see the issue for a long time.
Feel free to reopen.