[Flaky] test_numpy_op.test_np_einsum

[ChaiBapchya]

Description

======================================================================

FAIL: test_numpy_op.test_np_einsum

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 315, in _with_np_shape

    return func(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 499, in _with_np_array

    return func(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_numpy_op.py", line 3524, in test_np_einsum

    assert_almost_equal(out_mx.asnumpy(), expected_np, rtol=rtol, atol=atol)

  File "/work/mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error 1.669922 exceeds tolerance rtol=1.000000e+00, atol=1.000000e-01 (mismatch 25.000000%).

Location of maximum error: (0, 1), a=0.24609375, b=0.02963257

 ACTUAL: array([[156.    ,   0.2461],

       [433.2   , -30.8   ]], dtype=float16)

 DESIRED: array([[156.     ,   0.02963],

       [433.2    , -30.72   ]], dtype=float16)

-------------------- >> begin captured stdout << ---------------------


*** Maximum errors for vector of size 4:  rtol=1.0, atol=0.1


  1: Error 1.669922  Location of error: (0, 1), a=0.24609375, b=0.02963257


--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

root: INFO: NumPy-shape semantics has been activated in your code. This is required for creating and manipulating scalar and zero-size tensors, which were not supported in MXNet before, as in the official NumPy library. Please DO NOT manually deactivate this semantics while using `mxnet.numpy` and `mxnet.numpy_extension` modules.

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1120100586 to reproduce.

--------------------- >> end captured logging << ---------------------

Occurrences

Unrelated PR - #16599

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-16599/5/pipeline

What have you tried to solve it?

Nothing

[ChaiBapchya]

Introduced in #15911

[nickguletskii]

Also happened here: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-gpu/detail/PR-16639/1/pipeline
and here: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-16639/1/pipeline

[hzfan]

Temporarily disable the flaky tests in #16643 .

[ChaiBapchya]

@mxnet-label-bot update [Flaky, Numpy]

[TaoLv]

It's still flaky in #16555: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-16555/18/pipeline

FAIL: test_numpy_op.test_np_einsum

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 315, in _with_np_shape

    return func(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 499, in _with_np_array

    return func(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_numpy_op.py", line 3585, in test_np_einsum

    assert_almost_equal(grad[0][iop], grad[1][iop], rtol=rtol, atol=atol)

  File "/work/mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error 1.018927 exceeds tolerance rtol=1.000000e-03, atol=1.000000e-05 (mismatch 20.000000%).

Location of maximum error: (0, 1), a=-0.03388485, b=-0.03384018

 ACTUAL: array([[  97.04612   ,   -0.03388485, -148.55832   ,  -86.593506  ,

         -22.488245  ],

       [  97.04612   ,   -0.03388485, -148.55832   ,  -86.593506  ,...

 DESIRED: array([[  97.04615   ,   -0.03384018, -148.5583    ,  -86.5935    ,

         -22.48825   ],

       [  97.04615   ,   -0.03384018, -148.5583    ,  -86.5935    ,...

-------------------- >> begin captured stdout << ---------------------



*** Maximum errors for vector of size 25:  rtol=0.001, atol=1e-05



  1: Error 1.018927  Location of error: (0, 1), a=-0.03388485, b=-0.03384018

  2: Error 1.018927  Location of error: (1, 1), a=-0.03388485, b=-0.03384018

  3: Error 1.018927  Location of error: (2, 1), a=-0.03388485, b=-0.03384018

  4: Error 1.018927  Location of error: (3, 1), a=-0.03388485, b=-0.03384018

  5: Error 1.018927  Location of error: (4, 1), a=-0.03388485, b=-0.03384018