pytest: ERROR at teardown of TestUnicodeDocstring.test_with_docstring_1_v_l_

[koobs]

While QA'ing parameterized 0.7.5 FreeBSD Port update, the following test failure is seen:

ERROR at teardown of TestUnicodeDocstring.test_with_docstring_1_v_l_
...
E AssertionError: Lists differ: ["test_instance_method('foo0', bar=None)",[461 chars]o')"] != []

Test suite / environment summary:

platform freebsd13 -- Python 3.9.1, pytest-4.5.0, py-1.9.0, pluggy-0.13.1
rootdir: /var/tmp/tmpfs0/usr/home/koobs/repos/freebsd/ports/devel/py-parameterized/work-py39/parameterized-0.7.5
collected 57 items

parameterized/test.py .........xx......x......xx....................x.s..xxxx..E

...<snip>...

46 passed, 1 skipped, 10 xfailed, 10 warnings, 1 error in 1.14 seconds

Identical test failure is seen with Python 3.6.12, 3.7.9, 3.8.7, 3.9.1

Unsure if this is a pytest 4.x specific or related issue, reporting nevertheless. May be related to test invocation commands (which use pythonX.Y -m <foo> rather than test command scripts directly:

/usr/local/bin/python3.9 -m nose2 || rc=1;
/usr/local/bin/python3.9 -m pytest parameterized/test.py || rc=1; 
/usr/local/bin/python3.9 -m unittest parameterized.test || rc=1;  
/usr/local/bin/python3.9 -m unittest2 parameterized.test || rc=1;
exit $rc

Full Traceback incl. Invocation

cd /var/tmp/tmpfs0/usr/home/koobs/repos/freebsd/ports/devel/py-parameterized/work-py39/parameterized-0.7.5; rc=0;  /usr/local/bin/python3.9 -m nose || rc=1;  /usr/local/bin/python3.9 -m nose2 || rc=1;  /usr/local/bin/python3.9 -m pytest parameterized/test.py || rc=1;  /usr/local/bin/python3.9 -m unittest pa[45/9975]
ed.test || rc=1;  /usr/local/bin/python3.9 -m unittest2 parameterized.test || rc=1;  exit $rc
...........................................................................................S
----------------------------------------------------------------------
Ran 92 tests in 0.179s

OK (SKIP=1)
....s.......................................................................................
----------------------------------------------------------------------
Ran 92 tests in 0.059s

OK (skipped=1)
==================================================================================================================================================== test session starts ====================================================================================================================================================
platform freebsd13 -- Python 3.9.1, pytest-4.5.0, py-1.9.0, pluggy-0.13.1
rootdir: /var/tmp/tmpfs0/usr/home/koobs/repos/freebsd/ports/devel/py-parameterized/work-py39/parameterized-0.7.5
collected 57 items

parameterized/test.py .........xx......x......xx....................x.s..xxxx..E                                                                                                                                                                                                                                      [100%]

========================================================================================================================================================== ERRORS ===========================================================================================================================================================
___________________________________________________________________________________________________________________________ ERROR at teardown of TestUnicodeDocstring.test_with_docstring_1_v_l_ ____________________________________________________________________________________________________________________________

tp = <class 'AssertionError'>, value = None, tb = None

    def reraise(tp, value, tb=None):
        try:
            if value is None:
                value = tp()
            if value.__traceback__ is not tb:
                raise value.with_traceback(tb)
>           raise value

/usr/local/lib/python3.9/site-packages/six.py:703:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/six.py:703: in reraise
    raise value
/usr/local/lib/python3.9/site-packages/six.py:703: in reraise
    raise value
parameterized/test.py:371: in tearDownModule
    assert_equal(missing, [])
/usr/local/lib/python3.9/unittest/case.py:831: in assertEqual
    assertion_func(first, second, msg=msg)
/usr/local/lib/python3.9/unittest/case.py:1037: in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
/usr/local/lib/python3.9/unittest/case.py:1019: in assertSequenceEqual
    self.fail(msg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nose.tools.trivial.Dummy testMethod=nop>, msg = 'Lists differ: ["test_instance_method(\'foo0\', bar=None)",[461 chars]o\')"] != []\n\nFirst list contains 12 additiona...1, \'umask\', \'getpid\')",\n-  "test_patch_no_expand(42, 51, \'umask\')",\n-  "test_wrapped_iterable_input(\'foo\')"]'

    def fail(self, msg=None):
        """Fail immediately, with the given message."""
>       raise self.failureException(msg)
E       AssertionError: Lists differ: ["test_instance_method('foo0', bar=None)",[461 chars]o')"] != []
E
E       First list contains 12 additional elements.
E       First extra element 0:
E       "test_instance_method('foo0', bar=None)"
E
E       + []
E       - ["test_instance_method('foo0', bar=None)",
E       -  "test_instance_method('foo1', bar=None)",
E       -  "test_instance_method('foo2', bar=42)",
E       -  'test_instance_method(42, bar=None)',
E       -  "test_mock_patch_standalone_function(42, 'umask')",
E       -  "test_naked_function('foo0', bar=None)",
E       -  "test_naked_function('foo1', bar=None)",
E       -  "test_naked_function('foo2', bar=42)",
E       -  'test_naked_function(42, bar=None)',
E       -  "test_patch_class_no_expand(42, 51, 'umask', 'getpid')",
E       -  "test_patch_no_expand(42, 51, 'umask')",
E       -  "test_wrapped_iterable_input('foo')"]

/usr/local/lib/python3.9/unittest/case.py:670: AssertionError
/usr/local/lib/python3.9/unittest/case.py:670: AssertionError
===================================================================================================================================================== warnings summary ======================================================================================================================================================
parameterized/test.py:55
  parameterized/test.py:55: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_naked_function will be ignored
    @parameterized(test_params)

parameterized/test.py:201
  parameterized/test.py:201: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_mock_patch_standalone_function will be ignored
    @parameterized([(42, )])

parameterized/test.py:356
  parameterized/test.py:356: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_wrapped_iterable_input will be ignored
    @parameterized(lambda: iter(["foo"]))

parameterized/test.py:401
  parameterized/test.py:401: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_parameterized_argument_value_pairs will be ignored
    @parameterized([

parameterized/test.py:424
  parameterized/test.py:424: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_short_repr will be ignored
    @parameterized([

parameterized/test.py:433
  parameterized/test.py:433: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_with_docstring will be ignored
    @parameterized([

parameterized/test.py:443
  parameterized/test.py:443: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_cases_over_10 will be ignored
    @parameterized(cases_over_10)

parameterized/test.py:68
  parameterized/test.py:68: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_instance_method will be ignored
    @parameterized(test_params)

parameterized/test.py:146
  parameterized/test.py:146: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_patch_class_no_expand will be ignored
    @parameterized([(42, 51)])

parameterized/test.py:190
  parameterized/test.py:190: PytestCollectionWarning: yield tests were removed in pytest 4.0 - test_patch_no_expand will be ignored
    @parameterized([(42, 51)])

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================================================================================================================== 46 passed, 1 skipped, 10 xfailed, 10 warnings, 1 error in 1.14 seconds ===========================================================================================================================
....................................
----------------------------------------------------------------------
Ran 36 tests in 0.026s

OK
....................................
----------------------------------------------------------------------
Ran 36 tests in 0.027s

OK

[wolever]

Hey! It looks like this issue is coming up because there were some breaking changes in pytest4 which I haven't had a chance to address.

See: #116

Note: it may make your life easier to run tests with tox, which will automatically run tests against the matrix of python versions + test runners that are supported :)

[wolever]

Closing this, as this will be trackd by #34 and #116

[koobs]

Hey! It looks like this issue is coming up because there were some breaking changes in pytest4 which I haven't had a chance to address.

See: #116

Note: it may make your life easier to run tests with tox, which will automatically run tests against the matrix of python versions + test runners that are supported :)

Thanks. I'm aware running with tox will allow one to pin test dependencies, but this does not allow OS packagers or downstream users to test / QA within the context of their system Python environment (which we need to do), hence running pytest directly.

Thanks for letting us know its a pytest 4.x related issue, I'll exclude the test for now with an upstream reference to this issue

[wolever]

Ah that makes sense!

Is there anything I can do to make your life easier there?

Also, I'll have the fix for pytest4 out in a ~week, in case it's easier to just wait for that :)