exit code 1 even though all tests are passing

[qci-amos]

I have a set of pytests that I want to run from a Github Action. I've configured my workflow to include this:

    - name: Test with pytest
      run: |
        pytest -ra test/test_utils.py test/test_client.py

and set it to run in a matrix:

    strategy:
      matrix:
        python-version: [3.8]
        os: ["ubuntu-latest", "windows-latest"]

The tests pass on ubuntu. In Windows though, I get:

============================= test session starts =============================
platform win32 -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
***dir: D:\a\<snip>
plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, mock-3.6.1, timeout-2.0.1
collected 32 items

test\test_utils.py ..                                                    [  6%]
test\test_client.py ......ss..ss..sss............s                       [100%]

============================== warnings summary ===============================
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\stringcase-1.2.0-py3.8.egg\stringcase.py:247
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\stringcase-1.2.0-py3.8.egg\stringcase.py:247
  c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\stringcase-1.2.0-py3.8.egg\stringcase.py:247: DeprecationWarning: invalid escape sequence \W

<snip>

-- Docs: https://docs.pytest.org/en/stable/warnings.html
===Flaky Test Report===

test_cancel[Signals.CTRL_C_EVENT] passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
=========================== short test summary info ===========================
SKIPPED [8] test\conftest.py:146: Set DUMMY_JCS to the name of an available JCS to test
============ 24 passed, 8 skipped, 6 warnings in 447.29s (0:07:27) ============
Error: Process completed with exit code 1.

How is this even possible -- that everything would pass (and ubuntu passes) but when run on Windows has a non-zero exit code?

Unfortunately, given that this is in a remote server, it's not straight forward to prepare a pip list.

[nicoddemus]

Hi @qci-amos,

Hmm that's strange.

pytest returns 1 at this point:

pytest/src/_pytest/main.py

Lines 324 to 325 in cd49075

	if session.testsfailed:
	return ExitCode.TESTS_FAILED

And is increased only here, when a test reports a failure:

pytest/src/_pytest/main.py

Lines 520 to 521 in cd49075

	if report.failed and not hasattr(report, "wasxfail"):
	self.testsfailed += 1

However we don't see anything in the output to indicate that.

Any chance that flaky plugin might be changing session.testsfailed?

Unfortunately, given that this is in a remote server, it's not straight forward to prepare a pip list.

You can just run pip list before your pytest command:

    - name: Test with pytest
      run: |
        pip list
        pytest -ra test/test_utils.py test/test_client.py

[qci-amos]

Thank you @nicoddemus . I've confirmed that if I skip the (otherwise passing) flaky test, my overall pytest exit code is 0.

[qci-amos]

box/flaky#185