Flaky and pytest-check don't fail when used together

[antoche]

Hi,

I've also filed this under okken/pytest-check#25, but since the issue arises only when using both plugins together, I'm not sure where the responsibilities lie, so flagging it with both projects to be sure.

This simple case fails as expected:

from pytest_check import check
import pytest

# @pytest.mark.flaky
def test_foo():
    with check:
        assert 11 == 10

But uncomment the @pytest.mark.flaky and the test now "passes".

The "Flaky Test Report" in the console prints test_foo passed 1 out of the required 1 times. Success!, whereas the pytest final line says no tests ran instead of either the red 1 failed (expected) or the green 1 passed (which would be more consistent with the flaky report).

The exit code from pytest is 0 (i.e., tests passing) instead of either 1 (expected) or 5 (which would be more consistent with the no tests ran printout).

This was using python 3.7.3, pytest 4.6.5, pytest-check 0.3.5 and flaky 3.6.1.

Regards,
A.