[Flaky] test_operator_gpu.test_dropout

[junrushao]

http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/unix-gpu/branches/PR-14270/runs/5/nodes/271/steps/624/log/?start=0

======================================================================
FAIL: test_operator_gpu.test_dropout
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 173, in test_new
    orig_test(*args, **kwargs)
  File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 6107, in test_dropout
    check_dropout_ratio(1.0, shape, cudnn_off=False)
  File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 6040, in check_dropout_ratio
    check_correctness(exe, exe.arg_arrays[0].asnumpy(), ratio)
  File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 6005, in check_correctness
    assert output_zeroes == len(input)
AssertionError: 
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=623337751 to reproduce.
--------------------- >> end captured logging << ---------------------

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Test, Flaky

[junrushao]

See also

http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/unix-gpu/branches/PR-14270/runs/5/nodes/271/steps/624/log/?start=0

[junrushao]

@mxnet-label-bot add [Test, Flaky]

[perdasilva]

Seeing it again: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/master/650/pipeline

Making a PR to disable test until it is fixed.

[ChaiBapchya]

Was able to reproduce this issue.
With cudnn_off=True,

check_dropout_ratio(1.0, shape)

Works correctly
assert output_zeroes == len(input) is correct (both values 10000)

However, With cudnn_off=False i.e. with cudnn

check_dropout_ratio(1.0, shape)

Fails at the assertion because the values are
output_zeroes 9999
input 10000

[eric-haibin-lin]

@DickJC123 @ptrendx it looks like a bug in the cudnn dropout implementation on the ratio=1 case