Flaky and pytest-check don't fail when used together

[antoche]

Hi,

I've also filed this under box/flaky#162, but since the issue arises only when using both plugins together, I'm not sure where the responsibilities lie, so flagging it with both projects to be sure.

This simple case fails as expected:

from pytest_check import check
import pytest

@pytest.mark.flaky
def test_foo():
    # with check:
        assert 11 == 10

But uncomment the with check and the test now "passes".

The "Failed Checks: 1" output disappears.

The "Flaky Test Report" in the console prints test_foo passed 1 out of the required 1 times. Success!, whereas the pytest final line says no tests ran instead of either the red 1 failed (expected) or the green 1 passed (which would be more consistent with the flaky report).

The exit code from pytest is 0 (i.e., tests passing) instead of either 1 (expected) or 5 (which would be more consistent with the no tests ran printout).

This was using python 3.7.3, pytest 4.6.5, pytest-check 0.3.5 and flaky 3.6.1.

Regards,
A.

[joaonc]

@antoche did you try with the @flaky decorator instead of @pytest.mark.flaky ?

Edit: tested with the @flaky decorator and the behavior is the same.

[joaonc]

This seems like a fundamental incompatibility between these two plugins.
pytest-check marks the test as failed in here and I think flaky makes it so pytest_runtest_makereport doesn't end up running the way it's expected as flaky may re-run tests before that's called (I may be wrong on the details, but it's something along those lines).

[joaonc]

Found a similar plugin that works well with pytest-check: pytest-rerunfailures

Adding the flaky mark w/ no extra parameters re-runs once, or you can specify reruns and reruns_delay on how many times it should re-run the flaky tests and how long to way between re-runs.

Tests:

@pytest.mark.flaky
def test_check_fail_flaky_default():
    check.equal(1, 2)


@pytest.mark.flaky(reruns=5)
def test_check_fail_flaky_5():
    check.equal(1, 2)

Results:

RERUN                       [ 64%]
test_check.py::test_check_fail_flaky_default FAILED                      [ 64%]
tests/test_check.py:78 (test_check_fail_flaky_default)
FAILURE: 
assert 1 == 2
test_check.py:81 in test_check_fail_flaky_default() -> check.equal(1, 2)
------------------------------------------------------------
Failed Checks: 1

RERUN                             [ 68%]
test_check.py::test_check_fail_flaky_5 RERUN                             [ 68%]
test_check.py::test_check_fail_flaky_5 RERUN                             [ 68%]
test_check.py::test_check_fail_flaky_5 RERUN                             [ 68%]
test_check.py::test_check_fail_flaky_5 RERUN                             [ 68%]
test_check.py::test_check_fail_flaky_5 FAILED                            [ 68%]
tests/test_check.py:83 (test_check_fail_flaky_5)
FAILURE: 
assert 1 == 2
test_check.py:86 in test_check_fail_flaky_5() -> check.equal(1, 2)
------------------------------------------------------------
Failed Checks: 1

[okken]

This is something I'd like to fix, just not sure how to.
PRs accepted. :)
Seriously though, I'll take a look at this (at some point), as flaky is one of the listed items in https://docs.pytest.org/en/latest/flaky.html and I'd like to make sure all of those work with pytest-check.

[joaonc]

TBH, I don't think this is fixable.. you can fix compatibility with the flaky plugin by using another pytest hook function, but then you'd probably break compatibility with some other plugin.
Case in point: pytest-check is compatible with pytest-timeout and not with flaky. pytest-rerunfailures is compatible with pytest-check but not with pytest-timeout (details here).

My solution (and what I recommend to others) is to not use a plugin to re-run failures and rather use pytest's options via CLI.

1. Mark flaky tests with @pytest.mark.flaky
2. pytest <file_or_dir> <options> || pytest <file_or_dir> <options> --last-failed -m flaky

The || is important b/c that portion will not be run if there are no failues. If it is run when there are no failures, pytest's exit code is 5 (as opposed to 0 success), which means the script would return an error.

@okken I would close this issue as won't fix, simply b/c the fix would simply shift the problem to another place.

[antoche]

Could this be fixed by changing pytest itself so that its hooks are more compatible with both "flaky tests" and "checks"?

…

On Thu, 19 Dec 2019 at 06:30, Joao Coelho ***@***.***> wrote: TBH, I don't think this is fixable.. you can fix compatibility with the flaky plugin by using another pytest hook function, but then you'd probably break compatibility with some other plugin. Case in point: pytest-check is compatible with pytest-timeout and not with flaky. pytest-rerunfailures is compatible with flaky but not with pytest-timeout (details here <pytest-dev/pytest-rerunfailures#99>). My solution (and what I recommend to others) is to *not* use a plugin to re-run failures and rather use pytest's options via CLI. 1. Mark flaky tests with @pytest.mark.flaky 2. pytest <file_or_dir> <options> || pytest <file_or_dir> <options> --last-failed -m flaky The || is important b/c that portion will not be run if there are no failues. If it is run when there are no failures, pytest's exit code is 5 (as opposed to 0 success), which means the script would return an error. @okken <https://github.com/okken> I would close this issue as won't fix, simply b/c the fix would simply shift the problem to another place. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25?email_source=notifications&email_token=AAMNJ6HJSGTK7LQNLU4X7ZTQZJM2PA5CNFSM4JO22VOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHG4DTA#issuecomment-567132620>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMNJ6HZA7ZAXBGO6SPRBXTQZJM2PANCNFSM4JO22VOA> .

[joaonc]

Unclear, but don't think so. pytest exposes those hooks with clear design on how they work and plugins are allowed to make potential breaking changes and that's what's going on.

Basically, when you have multiple plugins changing how the execution flow works, you're bound to have incompatibilities.

[antoche]

Having had a closer look at pytest-check, I think this is a design issue in pytest-check. Pytest-check swallows and logs the exceptions raised during a test, then flags the tests as failed at makereport() time. So any part of the system expecting to act on a failed test will not see any failure. It seems to me it would be more robust to raise the failure around runtest_call() or runtest_teardown(), i.e., immediately after the test is finished. This is how I implemented a similar feature in a nose plugin, and it looks like this is also what pytest-assume does ( https://github.com/astraw38/pytest-assume/blob/master/pytest_assume/plugin.py). At first glance, pytest-assume would probably be compatible with most other plugins including flaky.

…

On Thu, 19 Dec 2019 at 11:43, Joao Coelho ***@***.***> wrote: Unclear, but don't think so. pytest exposes those hooks with clear design on how they work and plugins are allowed to make potential breaking changes and that's what's going on. Basically, when you have multiple plugins changing how the execution flow works, you're bound to have incompatibilities. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25?email_source=notifications&email_token=AAMNJ6FNSXNK5EXCLFV4UI3QZKRSXA5CNFSM4JO22VOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHXYMA#issuecomment-567245872>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMNJ6HFC7NZPGBQPPV2C7TQZKRSXANCNFSM4JO22VOA> .

[okken]

pytest-rerunfailures is a reasonable workaround.
Also pytest-rerunfailures is way more explicit and has a more clear output than flaky.
So, I'm good with that