Performance regression with skipped tests since pytest 8.0.0

[henriquegemignani]

There's a significant CPU overhead when processing the results of a skipped test, depending on stack size when you call pytest.skip .

The following code takes 0.78s for pytest 7.4.4 and 23.93s for pytest 8.2.2

import pytest


def get_data():
    pytest.skip("Skipped")


@pytest.fixture(scope="session")
def data():
    return get_data()


@pytest.mark.parametrize("index", list(range(1000)))
def test_do_nothing(data, index):
    assert index >= 0

pip list:

python -m pip list
Package   Version
--------- -------
colorama  0.4.6
iniconfig 2.0.0
packaging 24.1
pip       24.0
pluggy    1.5.0
pytest    7.4.4

Environment for tests: Windows 11 with Python 3.13.0b2
Bug was originally identified on GHA Linux, Python 3.10, 3.11 and 3.12, then confirmed on Windows + Python 3.11.

[The-Compiler]

Thanks for the clear reproducer! Bisected to:

commit 9d0ddb462517fc6c4f6dc7e0f1e14af2e08aeed9
Author: Ran Benita <[email protected]>
Date:   Fri Jul 14 10:35:18 2023 +0300

    fixtures: change `FixtureDef.cached_result[2]` from exception triplet to exception
    
    Fix #11208.

• fixtures: change FixtureDef.cached_result[2] from exception triplet to exception #11208
• fixtures: some tweaks & improvements #11209

I think there might be another commit after that were it got even slower, but that commit made it go from 0.5s to >6s on my machine.

cc @bluetech

[bluetech]

I'm not near a computer, but is it also slow on main? I will take a look when I get a chance, should be an easy fix (though I think we already fixed it)

On Sun, Jul 14, 2024, at 17:04, Florian Bruhin wrote: Thanks for the clear reproducer! Bisected to: `commit 9d0ddb4 Author: Ran Benita ***@***.***> Date: Fri Jul 14 10:35:18 2023 +0300 fixtures: change `FixtureDef.cached_result[2]` from exception triplet to exception Fix #11208.

`

…

• #11208 <#11208> • #11209 <#11209> I think there might be another commit after that were it got even slower, but that commit made it go from 0.5s to >6s on my machine. cc @bluetech <https://github.com/bluetech> — Reply to this email directly, view it on GitHub <#12608 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJKW7UKWGETWK4HLC3BOF3ZMKAQJAVCNFSM6AAAAABK3FG4BGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXGM3DCMBUGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[The-Compiler]

Ah, indeed I didn't try that.

Already fixed here indeed:

commit 0b91d5e3e869d00c01c614d827745a9ed4cddb43 (HEAD)
Author: Ran Benita <[email protected]>
Date:   Sun Apr 28 11:44:55 2024 +0300

    fixtures: fix tracebacks for higher-scoped failed fixtures getting longer and longer
    
    Fix #12204.

• ExceptionGroup traceback of session fixture unlimitedly grows from test to test #12204
• fixtures,runner: fix tracebacks getting longer and longer #12264