Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest unittest cases are destroyed after first teardown #212

Open
Chris-Peterson444 opened this issue Jun 6, 2024 · 0 comments
Open

pytest unittest cases are destroyed after first teardown #212

Chris-Peterson444 opened this issue Jun 6, 2024 · 0 comments

Comments

@Chris-Peterson444
Copy link

The following test from test/test_pytest/test_pytest_example.py fails:

@flaky
class TestExampleFlakyTestCase(TestCase):
    _threshold = -1

    @staticmethod
    def test_flaky_thing_that_fails_then_succeeds():
        """
        Flaky will run this test twice.
        It will fail once and then succeed.
        """
        TestExampleFlakyTestCase._threshold += 1
        assert TestExampleFlakyTestCase._threshold >= 1
========================================================================================================================== FAILURES ==========================================================================================================================
_____________________________________________________________________________________________ TestExampleFlakyTestCase.test_flaky_thing_that_fails_then_succeeds _____________________________________________________________________________________________

>       lambda: ihook(item=item, **kwds),  when=when, reraise=reraise
    )
E   AssertionError

flaky/flaky_pytest_plugin.py:148: AssertionError
===Flaky Test Report===

test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTestCase._threshold
        [<TracebackEntry /usr/lib/python3.12/unittest/case.py:58>, <TracebackEntry /usr/lib/python3.12/unittest/case.py:634>, <TracebackEntry /usr/lib/python3.12/unittest/case.py:589>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/test/test_pytest/test_pytest_example.py:101>]
test_flaky_thing_that_fails_then_succeeds failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>

        [<TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:342>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/flaky/flaky_pytest_plugin.py:148>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_hooks.py:513>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_manager.py:120>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:139>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/logging.py:850>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/logging.py:833>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/capture.py:878>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/skipping.py:257>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:103>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:184>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:174>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unittest.py:319>]

===End Flaky Test Report===
================================================================================================================== short test summary info ===================================================================================================================
FAILED test/test_pytest/test_pytest_example.py::TestExampleFlakyTestCase::test_flaky_thing_that_fails_then_succeeds - AssertionError
===================================================================================================================== 1 failed in 0.04s ======================================================================================================================

The first iteration fails (expected) and then each subsequent iteration fails due to an assertion error in pytest:

    def runtest(self) -> None:
        from _pytest.debugging import maybe_wrap_pytest_function_for_tracing
    
        testcase = self.instance
>       assert testcase is not None
E       AssertionError

This appears to be due to a bug in pytest itself pytest-dev/pytest#12424

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant