pytest >= 8.2 is not yet supported (not only tests fail)

[icemac]

See https://github.com/pytest-dev/pytest-rerunfailures/actions/runs/8409152908/job/23026172136

This can be a temporary problem or a is a hint that we have to change something here to make it compatible again.

[mgorny]

This is now affecting pytest 8.2.0.

[Chris-Peterson444]

This looks like a repeat of what 0ab54f0 tried to address (the teardown fixtures are not being called correctly). I spent some time debugging this this afternoon and the logic still looks correct, although it is effectively the same error. I'm not too familiar with the pytest internals but I wonder if the fix is somewhere along the lines of keeping a reference to the original setupstate instead of the depending on the final tests setupstate object?

[dkuchynskyi]

Hi guys, I think we've faced the same problem using pytest-rerunfailures with pytest 8.2.2 and pytest-xdist 3.6.1. we run tests with xdist loadscope and rerunfailures and on the step of collecting tests it fails:

platform linux
pytest-8.2.2
configfile: pytest.ini
plugins: rerunfailures-14.0, xdist-3.6.1
created: 3/3 workers
3 workers [43 items]

scheduling tests via LoadScopeScheduling
INTERNALERROR> def worker_internal_error(
INTERNALERROR>         self, node: WorkerController, formatted_error: str
INTERNALERROR>     ) -> None:
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error

We had to rollback to pytest 7.4.4 but left rerunfailures-14.0, xdist-3.6.1 and now it works properly.
Could you look at it please
thanks

[mkmoisen]

With pytest==8.2.2 and pytest-xdist==3.6.1, and pytest-rerunfailures==14.0, all tests that undergo a rerun result in the following AssertionError:

    def runtest(self) -> None:
        from _pytest.debugging import maybe_wrap_pytest_function_for_tracing
    
        testcase = self.instance
>       assert testcase is not None
E       AssertionError

It works fine for me on pytest==8.2.1 and pytest-cdist==3.5.0 and pytest-rerunfailures==14.0.

[SachinKSingh28]

It seems reruns are not working with pytest==8.2.2 and pytest-rerunfailures==14.0, anyone aware of any workarounds?

[icemac]

The currently only known workaround is to use an older version of pytest.

[jakkdl]

@asottile-sentry had teardown issues and bisected it to pytest-dev/pytest#11833 (which I authored). pytest-dev/pytest#12135 could also be the cause of some of the errors.

I'm not familiar with how pytest-rerunfailures messes with teardowns, but maybe looking through those PRs can help you debug.

[filiplajszczak]

Thanks @jakkdl I can confirm that pytest-dev/pytest@70c11582aaed is the exact commit that causes the fail of five tests in pytest-rerunfailures test suite:

• test_run_session_teardown_once_after_reruns
• test_exception_matches_rerun_except_query
• test_exception_not_match_rerun_except_query
• test_exception_matches_only_rerun_query
• test_exception_match_only_rerun_in_dual_query

[jakkdl]

reopening this as pytest==8.2 is still broken: #278 (comment)

[asottile-sentry]

it looks like this was fixed in 8.2.x after 8.2.2 regressed but 8.2.3 was never released -- I believe you can use !=8.2.2 and 8.2.1 should work (and in theory if pytest-dev/pytest#12438 is ever released then it will "just work" in 8.2.3 -- cc @nicoddemus @bluetech )