[Unity][UnitTest] Increase atol to resolve flaky CI failure (#16340)

Lunderberg · web-flow · commit 6f2fe457c285 · 2024-01-04T04:19:33.000+09:00
The `tests/python/relax/test_frontend_onnx.py::test_attention` unit test currently has sporadic failures (5/200 executions), which can cause failures for unrelated changes (e.g. [PR#16304](#16304), [CI link](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm-unity/detail/PR-16304/3/pipeline)). This commit specifies a tolerance to use for comparisons, to avoid these spurious CI failures.
diff --git a/tests/python/relax/test_frontend_onnx.py b/tests/python/relax/test_frontend_onnx.py
@@ -1226,6 +1226,8 @@ def verify_attention(
                 "mask_index": mask_index,
                 "relative_position_bias": relative_position_bias,
             },
+            # Maximum observed delta from 500 iterations was 2e-4.
+            atol=1e-3,
         )
         # "present" output should be nullptr when the "past" input isn't included,
         # but ort requires an output shape to be specified?

Original file line number	Diff line number	Diff line change
`@@ -1226,6 +1226,8 @@ def verify_attention(`
`1226`	`1226`	`"mask_index": mask_index,`
`1227`	`1227`	`"relative_position_bias": relative_position_bias,`
`1228`	`1228`	`},`
	`1229`	`+ # Maximum observed delta from 500 iterations was 2e-4.`
	`1230`	`+ atol=1e-3,`
`1229`	`1231`	`)`
`1230`	`1232`	`# "present" output should be nullptr when the "past" input isn't included,`
`1231`	`1233`	`# but ort requires an output shape to be specified?`