Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customize the fixture ordering on the same scope level #3393

Open
gavincyi opened this issue Apr 12, 2018 · 11 comments
Open

Customize the fixture ordering on the same scope level #3393

gavincyi opened this issue Apr 12, 2018 · 11 comments
Labels
topic: parametrize related to @pytest.mark.parametrize type: enhancement new feature or API change, should be merged into features branch type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature

Comments

@gavincyi
Copy link

gavincyi commented Apr 12, 2018

Could we have one more parameter, e.g. priority (between 0 and 100), for the users the customize the cost of setup/teardown fixture? For example,

@pytest.fixture(scope="session", params=[1, 2, 3], priority=100)
def f1(request):
    pass

@pytest.fixture(scope="session", params=['a', 'b', 'c'], priority=50)
def f2(request):
    pass

def test_f(f1, f2):
    pass

should give

test_order.py::test[1-a] PASSED
test_order.py::test[1-b] PASSED
test_order.py::test[1-c] PASSED
test_order.py::test[2-a] PASSED
test_order.py::test[2-b] PASSED
test_order.py::test[2-c] PASSED
test_order.py::test[3-a] PASSED
test_order.py::test[3-b] PASSED
test_order.py::test[3-c] PASSED

pytest now assumes an equal weighted cost of setup/teardown fixtures. So it reorders the product of fixture combination to optimize the count of setup/teardown time. For the code given before, the current behavior is

pytest_example.py::test_f[1-a] PASSED
pytest_example.py::test_f[1-b] PASSED
pytest_example.py::test_f[2-b] PASSED
pytest_example.py::test_f[2-a] PASSED
pytest_example.py::test_f[2-c] PASSED
pytest_example.py::test_f[1-c] PASSED
pytest_example.py::test_f[3-c] PASSED
pytest_example.py::test_f[3-b] PASSED
pytest_example.py::test_f[3-a] PASSED

The assumption is not pragmatic when

  1. The setup/teardown cost of a fixture is much heavier than that of other fixtures
  2. The current reordering can have more than 1 solution, while the user cannot determine which ordering will be used (e.g. the cost of ['1-a', '1-b', '2-b', '2-a'] is same as that of ['1-a', '2-a', '2-b', '1-b']).

This enhancement will allow the users to customize the ordering if they want and resolve #2846.

@pytestbot pytestbot added topic: parametrize related to @pytest.mark.parametrize type: bug problem that needs to be addressed labels Apr 12, 2018
@pytestbot
Copy link
Contributor

GitMate.io thinks possibly related issues are #538 (Fixture scope documentation), #805 (Fixture execution order ), #668 (autouse fixtures break scope rules), and #2846 (Unexpected order of tests using parameterized fixtures).

@nicoddemus nicoddemus added type: enhancement new feature or API change, should be merged into features branch and removed type: bug problem that needs to be addressed labels Apr 12, 2018
@nicoddemus
Copy link
Member

@gavincyi thanks for opening up this issue. 👍

Seems doable, although I'm not sure adding a parameter to @pytest.fixture which would be used solely by session-scoped-parametrized-fixtures is a good idea, it might be confusing. Perhaps adding a mark instead, something like @pytest.mark.session_reorder_priority(100) or @pytest.mark.session_reorder_weight(100)? That mark then can be used by the reorder algorithm to reduce fixture instantiations of fixtures with higher priority/weight.

@nicoddemus nicoddemus added the type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature label Apr 12, 2018
@nicoddemus
Copy link
Member

cc @ceridwen and @cheezman34 because of #3161.

@RazerM
Copy link

RazerM commented Sep 5, 2018

I may be thinking about this wrong, but wouldn't this also be useful for non-session fixtures?

In my case what I'm trying is different function scoped fixtures (e.g. fix_pre1, fix_pre2, ...) that I always want to happen before a certain other fixture (e.g. fix_main). The problem is that fix_main can't depend on fix_pre because it's up to the tests to choose which fix_pre fixtures are in use.

I thought that the parameter order might help determine the fixture order, but that's only the case when I try a simple example like this:

import pytest

@pytest.fixture
def fix_main():
    pass

@pytest.fixture
def fix_pre():
    pass

def test_abc(fix_pre, fix_main):
    pass

In my real test suite, fix_main is always setup before fix_pre, and I can't see a way to change that. fix_main is provided by a plugin, so it's difficult to replace it if I wanted to make a different version for each combination of fix_preN that my tests use.

@Sup3rGeo
Copy link
Member

Sup3rGeo commented May 6, 2019

Wow it took me a while but I finally found other people with the same problem as I am having now! Definitely not all fixtures for a same scope have equal weight.

Seems doable, although I'm not sure adding a parameter to @pytest.fixture which would be used solely by session-scoped-parametrized-fixtures is a good idea, it might be confusing.

I think this is valid for all the scopes higher than function - then the priority would select which fixtures should be preserved among all other fixtures at the same scope level.

Is there any workarounds that can be implemented locally to overcome this problem? I thought about implementing my own algorithm in pytest_collection_modifyitems but I am not sure if this would be too hard.

@Sup3rGeo
Copy link
Member

Sup3rGeo commented May 6, 2019

So my proposal would be as follows:

  1. Add a weight parameter to fixtures (even negative, considering the default priority could be 0)
  2. The ordering would be based on fixture weight groups:
  • higher weight fixtures should be preserved in detriment of more instantiations lower weight fixtures
  • the current algorithm being applied for fixtures with the same weight.

@nicoddemus
Copy link
Member

nicoddemus commented May 6, 2019

Is there any workarounds that can be implemented locally to overcome this problem?

One workaround, if you have access to the code of the fixtures, is to make one fixture depend on the other. For example:

@pytest.fixture
def foo():
    ...

@pytest.fixture
def bar():
    ...

If you want to make sure that bar always executes before foo, make foo depend on bar:

@pytest.fixture
def foo(bar):
    ...

@pytest.fixture
def bar():
    ...

About the original proposal, I'm still not entirely sure if we should really add a priority parameter to fixtures... perhaps we can reorganize the code a bit to allow for a plugin to change the priority instead? 🤔

@Sup3rGeo
Copy link
Member

Sup3rGeo commented May 19, 2019

I have a working implementation of the sorting algorithm that takes weights/priorities into account, now I just need to figure the best way to define it in tests so that the algorithm can retrieve the weights for each parameter.

@nicoddemus you mentioned the idea of

Perhaps adding a mark instead, something like @pytest.mark.session_reorder_priority(100) or @pytest.mark.session_reorder_weight(100)? That mark then can be used by the reorder algorithm to reduce fixture instantiations of fixtures with higher priority/weight.

How could I access that mark? I basically want to access that mark in this function (or, to be more precise, my plugin's own implementation of that):

def get_parametrized_fixture_keys(item, scopenum):
""" return list of keys for all parametrized arguments which match
the specified scope. """
assert scopenum < scopenum_function # function
try:
cs = item.callspec
except AttributeError:
pass
else:
# cs.indices.items() is random order of argnames. Need to
# sort this so that different calls to
# get_parametrized_fixture_keys will be deterministic.
for argname, param_index in sorted(cs.indices.items()):
if cs._arg2scopenum[argname] != scopenum:
continue
if scopenum == 0: # session
key = (argname, param_index)
elif scopenum == 1: # package
key = (argname, param_index, item.fspath.dirpath())
elif scopenum == 2: # module
key = (argname, param_index, item.fspath)
elif scopenum == 3: # class
key = (argname, param_index, item.fspath, item.cls)
yield key

And I see we could access the FixtureDef through cs.metafunc._args2fixturedefs[argname]

@Sup3rGeo
Copy link
Member

Sup3rGeo commented May 20, 2019

People interested in this, I made a plugin that implements a @parameter_priority decorator that can use to set strict order for parameters. The good thing is that this still uses the original algorithm, but priorities are not defined just by scope, but by (scope, priority).

Check how to use it in https://github.com/Sup3rGeo/pytest-param-priority and please provide feedback :) I did not test with classes yet.

The implementation possibly can be improved, and it would be quite easier if we could just add it to FixtureDefs somehow directly (check https://github.com/Sup3rGeo/pytest-param-priority/blob/master/pytest_param_priority.py)

@smarie
Copy link
Contributor

smarie commented May 20, 2019

Two comments on this:

  • it seems to me that the ordering problem also applies on module-scope fixtures
  • would it be useful also to enable users to declare that a session-scoped or module-scoped fixture should be unique (= setup and torn down only once per parameter in a given session/module)? For example @fixture(unique_in_scope=True). False would obviously be the default value to preserve the current behaviour.

@forestnew
Copy link

After so many years, this question remains relevant.
I ran into this problem and was also looking for a solution, I found this discussion.

People interested in this, I made a plugin that implements a @parameter_priority decorator that can use to set strict order for parameters. The good thing is that this still uses the original algorithm, but priorities are not defined just by scope, but by (scope, priority).

Check how to use it in https://github.com/Sup3rGeo/pytest-param-priority and please provide feedback :) I did not test with classes yet.

The implementation possibly can be improved, and it would be quite easier if we could just add it to FixtureDefs somehow directly (check https://github.com/Sup3rGeo/pytest-param-priority/blob/master/pytest_param_priority.py)

Your solution seems interesting to me. It's amazing that after all this time it still causes problems

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: parametrize related to @pytest.mark.parametrize type: enhancement new feature or API change, should be merged into features branch type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature
Projects
None yet
Development

No branches or pull requests

7 participants