VS Code Python Extension Compatibility

[eleanorjboyd]

Hello! I am a maintainer for the python for VS Code extension. I was investigating getting pytest-subtest to work correctly in the extension as outline in this issue and had some questions which I was looking for some help on.

here's the example Im working with

def test(subtests):
    with subtests.test(msg="First subtest"):
        assert 1 == 1
    with subtests.test(msg="Second subtest"):
        assert 2 == 1

output / outcome on running from the terminal:

test_subtest.py::test [First subtest] SUBPASS                                                                                                [100%]
test_subtest.py::test [Second subtest] SUBFAIL                                                                                               [100%]
test_subtest.py::test PASSED                                                                                                                 [100%]

1. Do subtests get their own callable test ids? Currently I allow the "run" functionality in the sidebar of VS Code seen in this screenshot by calling a test by its specific id but wasn't able to find if a subtest has a specific one that was unique from its parent

2. Why does the parent test test_subtest.py::test pass if one of the subtests fails?

3. Finally just curious what your thoughts are on ideal behavior. In my mind it would mirror subtests in unittest as it works now in vscode which is the parent node test_subtest.py::test is displayed in the tree on test discovery, and the children nodes are added to the tree after run (since this is when they are "known").

after discovery:

after run:

Thanks!

[nicoddemus]

Hi @eleanorjboyd,

Awesome that you are working on this integration with VS Code!

Do subtests get their own callable test ids? Currently I allow the "run" functionality in the sidebar of VS Code seen in this screenshot by calling a test by its specific id but wasn't able to find if a subtest has a specific one that was unique from its parent

They do not, and is not really possible to only execute a subtest individually - you need to execute the parent test, and all the subtests will run as result.

Why does the parent test test_subtest.py::test pass if one of the subtests fails?

That's kind of quirk on how the hooks work today -- while a subtest might fail, the "rest" of the parent test might still pass, consider:

def test(subtests):
    with subtests.test(msg="First subtest"):
        assert 2 == 1
    assert 1 == 1

If the subtest fails but test passes, means the asserts outside the subtests.test blocks passed.

It is not ideal, perhaps we can change this in the future (there's related discussion in #11).

Finally just curious what your thoughts are on ideal behavior.

Yes that looks great!

[phihung]

In the mean time, I am using the following trick to avoid tests to fail silently in vscode

# tests/conftest.py
@pytest.fixture
def subtests(subtests):
    if "VSCODE_PID" in os.environ:
        subtests.test = noop_ctx
    yield subtests

@contextmanager
def noop_ctx(*args, **kwargs):
    yield