Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel testing support (xdist) #22

Closed
dex6 opened this issue Aug 29, 2017 · 3 comments
Closed

Parallel testing support (xdist) #22

dex6 opened this issue Aug 29, 2017 · 3 comments

Comments

@dex6
Copy link

dex6 commented Aug 29, 2017

Thanks for great plugin. However, there's one problem for me: my test suite uses xdist to run tests on multiple CPU cores, and it completely breaks with pytest-random-order.

Here's some fake tests to demonstrate the problem:

# cat /tmp/some_tests.py 
def test_1():
        pass
def test_2():
        pass
def test_3():
        pass
def test_4():
        pass

And here it is executed in two parallel processes:

# pytest /tmp/some_tests.py -n2
============================ test session starts ============================
platform linux2 -- Python 2.7.5, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
Using --randomly-seed=1503989180
Using --random-order-bucket=module
Using --random-order-seed=249480

rootdir: /tmp, inifile:
plugins: xdist-1.20.0, timeout-1.2.0, repeat-0.4.1, randomly-1.2.1, random-order-0.5.4, raisesregexp-2.1, pythonpath-0.7.1, mock-1.6.2, logging-2015.11.4, instafail-0.3.0, forked-0.2, faker-2.0.0, cov-2.5.1, browser-0.2.0
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors                                                
================================== ERRORS ===================================
___________________________ ERROR collecting gw1 ____________________________
Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -1,4 +1,4 @@

 some_tests.py::test_3
-some_tests.py::test_4
 some_tests.py::test_2
 some_tests.py::test_1
+some_tests.py::test_4
========================== 1 error in 0.51 seconds ==========================

It gets worse the more tests and CPU cores you have. I suppose that every subprocess generates a different seed and executes other test cases than it was expected to, because when a seed is specified using --random-order-seed, it always seem to work.

@adamchainz
Copy link
Member

@jbasko jbasko mentioned this issue Mar 19, 2018
jbasko added a commit that referenced this issue Mar 19, 2018
Make it work with pytest-xdist as requested in #22 - generate random order seed during command line argument initialisation so that all processes get the same default value.
@jbasko
Copy link
Collaborator

jbasko commented Mar 19, 2018

Fixed.

@jbasko jbasko closed this as completed Mar 19, 2018
@Alexander-Serov
Copy link

I know it's an old and closed issue, but I seem to be having the same problem with
pytest-random-order==1.0.4
pytest-xdist==2.5.0.


gw1:None (gw1)
Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -1,276 +1,276 @@
...

Should I open a new issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants