Double reporting teardown failure

[gruberma]

When the teardown fails, the last test case gets reported twice, both with it's own verdict, and with the error of the teardown.
This also happens in the junitxml, but only if the last test case fails (example 2).
My main question is: is this behavior intended (especially the double reporting within the junit-xml where the same test case shows up twice)?

There is a related issue reporting that the test number is wrong if the teardown fails, which also shows up here.

Example 1: Last test case passes

def test_fail():
    assert False

def test_pass():
    pass

def teardown():
    assert 1 == 0

python3 -m pytest -v --junit-xml junit.xml test_double_outcome.py
===================== test session starts =====================
platform linux -- Python 3.9.2, pytest-7.1.2, pluggy-1.0.0 -- REMOVED/venv/bin/python3
cachedir: .pytest_cache
rootdir: REMOVED
collected 2 items                                             

test_double_outcome.py::test_fail FAILED                [ 50%]
test_double_outcome.py::test_pass PASSED                [100%]
test_double_outcome.py::test_pass ERROR                 [100%]

=========================== ERRORS ============================
_______________ ERROR at teardown of test_pass ________________

    def teardown():
>       assert 1 == 0
E       assert 1 == 0

test_double_outcome.py:8: AssertionError
========================== FAILURES ===========================
__________________________ test_fail __________________________

    def test_fail():
>       assert False
E       assert False

test_double_outcome.py:2: AssertionError
- generated xml file: REMOVED/junit.xml -
=================== short test summary info ===================
FAILED test_double_outcome.py::test_fail - assert False
ERROR test_double_outcome.py::test_pass - assert 1 == 0
============ 1 failed, 1 passed, 1 error in 0.23s =============

In the junit-xml, the test shows up only once.

<?xml version="1.0" ?>
<testsuites>
▸       <testsuite name="pytest" errors="1" failures="1" skipped="0" tests="3" time="0.245" timestamp="2022-05-02T17:54:49.261096"             hostname="db3">
▸       ▸       <testcase classname="test_double_outcome" name="test_fail" time="0.002">
▸       ▸       ▸       <failure message="assert False">def test_fail():
&gt;       assert False
E       assert False

test_double_outcome.py:2: AssertionError</failure>
▸       ▸       </testcase>
▸       ▸       <testcase classname="test_double_outcome" name="test_pass" time="0.001">
▸       ▸       ▸       <error message="failed on teardown with &quot;assert 1 == 0&quot;">def teardown():
&gt;       assert 1 == 0
E       assert 1 == 0

test_double_outcome.py:8: AssertionError</error>
▸       ▸       </testcase>
▸       </testsuite>
</testsuites>

Example 2: Last test case fails

If we switch the order of the test cases, the double reporting also shows up in the junit-xml.

def test_pass():
    pass

def test_fail():
    assert False

def teardown():
    assert 1 == 0

> python3 -m pytest -v --junit-xml junit.xml test_double_outcome.py
===================== test session starts =====================
platform linux -- Python 3.9.2, pytest-7.1.2, pluggy-1.0.0 -- REMOVED/venv/bin/python3
cachedir: .pytest_cache
rootdir: REMOVED
collected 2 items                                             

test_double_outcome.py::test_pass PASSED                [ 50%]
test_double_outcome.py::test_fail FAILED                [100%]
test_double_outcome.py::test_fail ERROR                 [100%]

=========================== ERRORS ============================
_______________ ERROR at teardown of test_fail ________________

    def teardown():
>       assert 1 == 0
E       assert 1 == 0

test_double_outcome.py:8: AssertionError
========================== FAILURES ===========================
__________________________ test_fail __________________________

    def test_fail():
>       assert False
E       assert False

test_double_outcome.py:5: AssertionError
- generated xml file: REMOVED/junit.xml -
=================== short test summary info ===================
FAILED test_double_outcome.py::test_fail - assert False
ERROR test_double_outcome.py::test_fail - assert 1 == 0
============ 1 failed, 1 passed, 1 error in 0.25s =============

<?xml version="1.0" ?>
<testsuites>
▸       <testsuite name="pytest" errors="1" failures="1" skipped="0" tests="2" time="0.044" timestamp="2022-05-02T17:52:52.371978"             hostname="db3">
▸       ▸       <testcase classname="test_double_outcome" name="test_pass" time="0.001"/>
▸       ▸       <testcase classname="test_double_outcome" name="test_fail" time="0.001">
▸       ▸       ▸       <failure message="assert False">def test_fail():
&gt;       assert False
E       assert False

test_double_outcome.py:5: AssertionError</failure>
▸       ▸       </testcase>
▸       ▸       <testcase classname="test_double_outcome" name="test_fail" time="0.000">
▸       ▸       ▸       <error message="failed on teardown with &quot;assert 1 == 0&quot;">def teardown():
&gt;       assert 1 == 0
E       assert 1 == 0

test_double_outcome.py:8: AssertionError</error>
▸       ▸       </testcase>
▸       </testsuite>
</testsuites>

My setup

> pip list          
Package       Version
------------- -------
attrs         21.4.0
iniconfig     1.1.1
packaging     21.3
pip           20.3.4
pkg-resources 0.0.0
pluggy        1.0.0
py            1.11.0
pyparsing     3.0.8
pytest        7.1.2
setuptools    44.1.1
tomli         2.0.1
wheel         0.34.2


> python3 -m pytest --version
pytest 7.1.2

[asottile]

teardowns are always associated with the most recent test which ran

in some cases this results in multiple statuses for a single test

everything shown in the issue is expected

[gruberma]

perfect, thanks for the quick response :-)

[RonnyPfannschmidt]

@asottile as this one is about sometimes doublse, sometimes single entries in the junit report,
i do wonder if we should take it as a bug, but its not clear how to map this sanely to junitxml if we ensure report collapsing

[asottile]

@RonnyPfannschmidt there isn't a sometimes doubling if you look at the xml -- op was just confused by which test the teardown failure was associated with

[gruberma]

@RonnyPfannschmidt there isn't a sometimes doubling if you look at the xml -- op was just confused by which test the teardown failure was associated with

Hmm, I think the doubling doesn't always occur. In example 1,test_pass only occurs once in the junit-xml, but in example 2, test_fail occurs twice.

[RonnyPfannschmidt]

@asottile as im reading this again on a computer, its clearly doing something wrong

it should report the teardown errors in the junit and it shouldnt have double reports blindly
my understanding is that plain teardown should happen for every test, not every module

so either we are missing a report or something else is completely amiss

[storenth]

Yeah, same issue I got! Test results duplicated in case of any teardown failure. Take a look at allure example to figure out the idea! I can't mark the test properly;( So, results looks like: 38 deselected, 1 failed, 1 xpassed in 1.41s, but expected to be 38 deselected, 1 xfailed in 1.41s

[petr-balogh]

Hello @RonnyPfannschmidt , I think we hit the related issue to this (see my comment here) and then our report which we are uploading to Report Portal is missing some of the properties for duplicate case. Not sure how to W/A or fix this different way to at least have our properties reported in all of the occurrences of failures in junit XML report? Thanks

[RonnyPfannschmidt]

@petr-balogh a workaround could be to preprocess the report to merge the items manually, I haven't investigated this any further so far

[Sillocan]

This is still an issue with the pytest 8.3.3. What's really odd is that the XML has an extra test entry versus the output of pytest

[storenth]

Looks like we need to deal with test design principles because setup and teardown are steps to instrument your system but not test itself. I suggest move the bug to Discuss section.