[WIP] Fix precision issue of test case test_rnnrelu_bidirectional and test_rnnrelu_sym

[lihaofd]

Description

In this PR, it fixed precision issue of test case #11410 test_rnnrelu_bidirectional and test_rnnrelu_sym. @pengzhao-intel, @TaoLv

Feature changes

New features

• Fixed issue of precision issue of test case Flaky test: test_operator_gpu.test_rnnrelu_bidirectional #11410 test_rnnrelu_bidirectional and test_rnnrelu_sym. Sometimes, the data comparison will fail because of tolerance of precision, like 0.043388 and 0.04405. In our testing environment, the biggest testing result deviation is around difference between 0.027... and 0.037... sometimes. Adjust the tolerance to fix the issue.

Checklist

• Passed code style checking (make lint).
• All changes have test coverage.
• Code is well-documented.

[vandanavk]

@lihaofd any reason for closing this PR?

[lihaofd]

@vandanavk. We are still looking into the issue and check if there is other way to fix it. @TaoLv @pengzhao-intel

[vandanavk]

@lihaofd
Any pointers so far?

The issue is always reproducible with seed 272273578, FusedRNN with 2 layers and bidirectional=True. I tried out a few experiments - the issue doesn't occur with 3 layers in test_rnnrelu_bidirectional or when the FusedRNN is unfused.

Also, any reason for
T, N, I, H = 5, 20, 200, 200 in the case of test_rnnrelu_bidirectional and test_rnnrelu_sym instead of T, N, I, H = 5, 20, 800, 800 for test_rnnrelu_bidirectional

[vandanavk]

@lihaofd do you think we could temporarily submit this PR and in parallel, debug to find the fix?

[pengzhao-intel]

@vandanavk, agree, we can submit this PR first and debug in the same time.

@lihaofd please reopen this PR.

[lihaofd]

The PR can't be reopened as my local branch has been recreated. Submit new one on #12099. Thanks!