Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More difficulties in 0.23 with async fixtures #718

Closed
bmerry opened this issue Dec 12, 2023 · 8 comments
Closed

More difficulties in 0.23 with async fixtures #718

bmerry opened this issue Dec 12, 2023 · 8 comments
Labels
Milestone

Comments

@bmerry
Copy link

bmerry commented Dec 12, 2023

This might have the same underlying cause as #705 / #706, but I didn't see the specific error message I'm running into so I thought I should open a separate bug.

To reproduce:

conftest.py (copied verbatim from https://pytest-asyncio.readthedocs.io/en/latest/how-to-guides/run_session_tests_in_same_loop.html):

import pytest

from pytest_asyncio import is_async_test


def pytest_collection_modifyitems(items):
    pytest_asyncio_tests = (item for item in items if is_async_test(item))
    session_scope_marker = pytest.mark.asyncio(scope="session")
    for async_test in pytest_asyncio_tests:
        async_test.add_marker(session_scope_marker)

foo/__init__.py: empty

foo/test_foo.py:

import pytest_asyncio


@pytest_asyncio.fixture(scope="package")
async def bar() -> int:
    return 3

async def test_stuff(bar: int) -> None:
    pass

Gives this error:

==================================== ERRORS ====================================
_________________________ ERROR at setup of test_stuff _________________________
file /home/bmerry/work/sdp/bugs/pytest-asyncio-session/foo/test_foo.py, line 8
  async def test_stuff(bar: int) -> None:
      pass
file /home/bmerry/work/sdp/bugs/pytest-asyncio-session/foo/test_foo.py, line 4
  @pytest_asyncio.fixture(scope="package")
  async def bar() -> int:
      return 3
E       fixture 'foo/__init__.py::<event_loop>' not found
>       available fixtures: __pytest_repeat_step_number, _session_event_loop, anyio_backend, anyio_backend_name, anyio_backend_options, bar, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, check, class_mocker, cov, doctest_namespace, event_loop, event_loop_policy, foo/test_foo.py::<event_loop>, mocker, module_mocker, monkeypatch, no_cover, package_mocker, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, session_mocker, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, unused_tcp_port, unused_tcp_port_factory, unused_udp_port, unused_udp_port_factory, worker_id
>       use 'pytest --fixtures [testpath]' for help on them.

Python 3.10.12, pytest-asyncio 0.23.2.

@bmerry bmerry changed the title More breakage in 0.23 with async fixtures More difficulties in 0.23 with async fixtures Dec 13, 2023
@bmerry
Copy link
Author

bmerry commented Dec 13, 2023

I forgot to include the pytest.ini:

[pytest]
asyncio_mode = auto

bmerry added a commit to ska-sa/katgpucbf that referenced this issue Dec 13, 2023
pytest-asyncio 0.23 changed the way event loops work and caused a number
of issues, including pytest-dev/asyncio#705 and
pytest-dev/pytest-asyncio#718. I wasn't able to find an easy way around
these for the qualification test, so for now I've pinned pytest-asyncio
to <0.22 (the 0.22 release was a half-way point to 0.23 and has been
yanked).
@seifertm seifertm added the bug label Dec 17, 2023
@seifertm seifertm added this to the v0.23.3 milestone Dec 17, 2023
@seifertm
Copy link
Contributor

Thanks for the reproducer. Good call for opening a separate issue!

@nzeid
Copy link

nzeid commented Jan 6, 2024

Let me know if this isn't the best place to post this, but I have a gut feeling it's very closely related to the sloppiness of fixture scopes and event loop initialization.

I was busting my head all day trying to refactor my code to get rid of the def event_loop override per version 0.23. I could not for the life of me get function-scoped asyncio fixtures to work on session-scoped asyncio test functions with simple markers/decorators. I would almost always end up with the dreaded MultipleEventLoopsRequestedError. I can understand why multiple event loops are prohibited, but I can't understand why fixtures renewing at different cadences condemns them to live in a separate event loop. This is absolutely a must-have.

For anyone who happens upon this, the workaround for now is as follows:

# This allows all fixtures to access the session-scoped event loop:
@pytest_asyncio.fixture(scope="session")
async def global_event_loop():
    return asyncio.get_running_loop()


# The function of the async fixture you want to create, but without the marker:
async def clear_db():
    async with Transaction() as tx:
        await tx.execute(sql.text("DELETE FROM some_table"))


# Use this non-async fixture to mimic the behavior of the async fixture you need.
# Run the async code using global_event_loop.
@pytest.fixture(autouse=True, scope="function")
def clear_db_wrapper(global_event_loop):
    yield
    global_event_loop.run_until_complete(clear_db())

(Edit: Fixed and clarified code.)

@seifertm
Copy link
Contributor

@nzeid I'm sorry that you had to go through so much trouble. Pytest-asyncio v0.23 supports running tests in different event loops. There's a loop for each level of the test suite (session, package, module, class, function). However, pytest-asyncio wrongly assumes that the scope of an async fixture is the same as the scope of the event loop in which the fixture runs. This essentially makes it impossible to have a session-scoped fixture that runs in the same loop as a function-scoped test. See #706.

@av223119
Copy link

av223119 commented May 2, 2024

I'm experiencing the similar issue, which I believe is another manifestation of #706.

Namely, the error message fixture 'test_something.py::<event_loop>' not found which, unexpectedly, is generated when tests from test_something_else.py being run. The only thing these two modules have in common is that they use one shared module-scoped fixture from conftest.py. It appears that pytest-asyncio tried to use the event loop from the first module while running the second one. It looks awfully similar to #829 which, in turn, I believe is just another manifestation of #706

@iamWing
Copy link

iamWing commented May 5, 2024

@seifertm Hi, I'm trying to use the async fixtures at class scope and I'm having issues running the test module when there're multiple classes in that module. Here some of my test code

@pytest_asyncio.fixture(scope="class")
async def grpc_server() -> AsyncGenerator[(grpc.aio.Server, int), None]:
    global _KEYSTORE, _API_KEY_MANAGER_SERVICER

    server = grpc.aio.server()
    # add service

    port = server.add_insecure_port("[::]:0")

    await server.start()

    yield (server, port)

    await server.stop(None)

   # more clean up actions


@pytest_asyncio.fixture(scope="class")
async def grpc_stub(
    grpc_server: tuple[grpc.aio.Server, int],
):
    server, port = grpc_server

    channel = grpc.aio.insecure_channel(f"localhost:{port}")

    await channel.channel_ready()

    stub = api_key_manager_pb2_grpc.ApiKeyManagerStub(channel)

    yield stub

    await channel.close()


class TestExportServiceConfig: # non async test class
    _expected_json_str = """{
  "free_tier_daily_usage_limit": 1,
  "premium_tier_daily_usage_limit": -1
}"""

    @pytest.fixture()
    def json_path(self, tmp_path: pathlib.Path) -> str:
        return tmp_path / "service-config.json"

    def test_export_success(self, json_path: pathlib.Path):
        with does_not_raise():
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(json_path)

        with open(json_path, "r") as file:
            assert file.read() == self._expected_json_str

    def test_export_fail(self, tmp_path: pathlib.Path):
        with pytest.raises(OSError):
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(
                tmp_path / "nonexistent" / "file.json"
            )


@pytest.mark.asyncio(scope="class")
class TestGetServiceConfig: # async test class
    async def test_get_service_config(
        self, grpc_stub: api_key_manager_pb2_grpc.ApiKeyManagerStub
    ):
        expected_response = api_key_manager_pb2.GetServiceConfigResponse(
            config=api_key_manager_pb2.ApiKeyManagerServiceConfig(
                free_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "free_tier_daily_usage_limit"
                ],
                premium_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "premium_tier_daily_usage_limit"
                ],
            )
        )

        call: grpc.aio.Call = grpc_stub.GetServiceConfig(empty_pb2.Empty())

        response = await call

        assert response == expected_response
        assert await call.code() == grpc.StatusCode.OK

...

It works when I run pytest with just one class/specify a class to test, but when I run it as a module where there're multiple classes, async or not, I get the error of fixture 'something.py::<event_loop>' not found.

I'm not entirely sure if my issue is related to this thread but it seems to be the case. Tried downgrade to 0.21 directly without changing the code but that doesn't work for me. I guess the pytest.mark.asyncio(scope="value") syntax works differently between 0.21 & 0.23?

Is there any workaround I can implement to make my use case workable? Right now I can run the test on individual class but it makes it not possible for the test module to pass the CI/CD pipeline I've setup. Thanks!

Edit: Had a quick look through on the open issues and seems like my issue if also related to #829. Will come back and see if it works after I tried putting those class scope fixtures in the classes.

@iamWing
Copy link

iamWing commented May 5, 2024

@seifertm Hi, I'm trying to use the async fixtures at class scope and I'm having issues running the test module when there're multiple classes in that module. Here some of my test code

@pytest_asyncio.fixture(scope="class")
async def grpc_server() -> AsyncGenerator[(grpc.aio.Server, int), None]:
    global _KEYSTORE, _API_KEY_MANAGER_SERVICER

    server = grpc.aio.server()
    # add service

    port = server.add_insecure_port("[::]:0")

    await server.start()

    yield (server, port)

    await server.stop(None)

   # more clean up actions


@pytest_asyncio.fixture(scope="class")
async def grpc_stub(
    grpc_server: tuple[grpc.aio.Server, int],
):
    server, port = grpc_server

    channel = grpc.aio.insecure_channel(f"localhost:{port}")

    await channel.channel_ready()

    stub = api_key_manager_pb2_grpc.ApiKeyManagerStub(channel)

    yield stub

    await channel.close()


class TestExportServiceConfig: # non async test class
    _expected_json_str = """{
  "free_tier_daily_usage_limit": 1,
  "premium_tier_daily_usage_limit": -1
}"""

    @pytest.fixture()
    def json_path(self, tmp_path: pathlib.Path) -> str:
        return tmp_path / "service-config.json"

    def test_export_success(self, json_path: pathlib.Path):
        with does_not_raise():
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(json_path)

        with open(json_path, "r") as file:
            assert file.read() == self._expected_json_str

    def test_export_fail(self, tmp_path: pathlib.Path):
        with pytest.raises(OSError):
            _API_KEY_MANAGER_SERVICER.export_service_config_to_json(
                tmp_path / "nonexistent" / "file.json"
            )


@pytest.mark.asyncio(scope="class")
class TestGetServiceConfig: # async test class
    async def test_get_service_config(
        self, grpc_stub: api_key_manager_pb2_grpc.ApiKeyManagerStub
    ):
        expected_response = api_key_manager_pb2.GetServiceConfigResponse(
            config=api_key_manager_pb2.ApiKeyManagerServiceConfig(
                free_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "free_tier_daily_usage_limit"
                ],
                premium_tier_daily_usage_limit=_TEST_API_KEY_MANAGER_SERVICE_CONFIG[
                    "premium_tier_daily_usage_limit"
                ],
            )
        )

        call: grpc.aio.Call = grpc_stub.GetServiceConfig(empty_pb2.Empty())

        response = await call

        assert response == expected_response
        assert await call.code() == grpc.StatusCode.OK

...

It works when I run pytest with just one class/specify a class to test, but when I run it as a module where there're multiple classes, async or not, I get the error of fixture 'something.py::<event_loop>' not found.

I'm not entirely sure if my issue is related to this thread but it seems to be the case. Tried downgrade to 0.21 directly without changing the code but that doesn't work for me. I guess the pytest.mark.asyncio(scope="value") syntax works differently between 0.21 & 0.23?

Is there any workaround I can implement to make my use case workable? Right now I can run the test on individual class but it makes it not possible for the test module to pass the CI/CD pipeline I've setup. Thanks!

Edit: Had a quick look through on the open issues and seems like my issue if also related to #829. Will come back and see if it works after I tried putting those class scope fixtures in the classes.

Ended up commented out the scope=class lines and use the fixtures on function scope instead. Not ideal but it minimised the effort needed for refactoring

@seifertm
Copy link
Contributor

The issue title is very general. I think different issues have come in the discussion:

Given the generic nature of this issue and the fact that all comments are already tracked or accounted for, I'll close this issue.

@seifertm seifertm closed this as not planned Won't fix, can't repro, duplicate, stale Jul 13, 2024
paravoid added a commit to paravoid/ircstream that referenced this issue Sep 30, 2024
pytest-asyncio 0.22, 0.23 and 0.24 are broken in various ways, cf.
* pytest-dev/pytest-asyncio#670
* pytest-dev/pytest-asyncio#718
* pytest-dev/pytest-asyncio#587

Additionally, pytest 8 cannot work with pytest-asyncio 0.21, so we need
to pin pytest to 7.x too.

Perhaps there is a way to make this work, but pin it to earlier versions
to do this deliberately at some point.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants