Document way to customize abbreviation (elipsis) on AssertionError

[ssbarnea]

I keep getting errors that I cannot debug because the printed AssertionError is censored so much that makes impossible to see what was the difference.

Adding -vvvv did not help at all changing the way exceptions are rendered.

      assert l in out

E AssertionError: assert 'instance docker ansible default false false' in 'Instance Name Driver Name Provisioner Name Scenario Name Created Converged\n--------------- ---------...i-node false false\ninstance-2 docker ansible multi-node false false\n'

I did try to research the subject a lot online but I was not able to find any solution that would allow me to disable or customize when abbreviation happens, to make the limit upwards.

I raised this to PyTest because I am looking for a solution that can enable this at the entire test-suite and not on a specific file with tests, being able to do it globally is key for big projects.

Some resources I found:

• https://stackoverflow.com/questions/50608443/how-do-i-get-pycharm-to-show-entire-error-diffs-from-pytest -- I was not able to find a way to hack DEFAULT_MAX_CHARS
• https://stackoverflow.com/questions/38000993/how-can-i-get-my-assertions-in-pytest-to-stop-being-abbreviated-with-ellipsis

[blueyed]

Try env CI=true …, but it does not help here.
(It is an annoying thing, I know - there are some issues/ideas in that regard open already)
Related: blueyed#63
The problem here is that the internal pytest_assertrepr_compare hook does nothing here (for `op = "in"), and the assertion rewrite having it truncated already anyway.
Somehow related docs: https://github.com/blueyed/pytest/blob/49f6aaf725610acf67087b0a38dc69e636c7402f/doc/en/assert.rst#defining-your-own-explanation-for-failed-assertions

[ssbarnea]

One year later and I am facing again the same issue, a bug caused by an incomplete feature, one that cannot be disabled, the truncation.

Can we please do something about it? For some projects this is testing PITA as the comparisons are almost always bigger than the limit and the user is left clueless to face the what is behind those ellipses? questions.

[nicoddemus]

Indeed this is frustrating, thanks @ssbarnea for bringing it to light again.

I've opened #8391 as an attempt to fix this, feedback welcome.

[ssbarnea]

I think it was #8391 -- going to test it now.

[nicoddemus]

Oops yes, fixed my comment.

[ssbarnea]

@nicoddemus I think that we need to reopen this issue because current behavior is still not ok. I found multiple similar unanswered question online such https://stackoverflow.com/questions/38000993/how-can-i-get-my-assertions-in-pytest-to-stop-being-abbreviated-with-ellipsis

IMHO, I think that we need to make these values configurable regardless of the verbosity level, especially as we know that changing general verbosity level has other side effects.

My opinion is that most people would want to run in minimal verbosity level (especially) on ci but have verbose errors for failed tests.

If I run in verbose by default displaying the console log in the browser can be problematic, true for github actions and also most CI/CD system I know (none behaves nice with very long console logs).

[bluetech]

@ssbarnea Check out https://docs.pytest.org/en/stable/reference/reference.html#confval-verbosity_assertions, added in pytest 8.

[ssbarnea]

@bluetech That issue seems to never want to be go away... :p --- I tried, still same output with ellipses, in fact even calling pytest -vvvv does ellipsis....

[tool.pytest.ini_options]
verbosity_assertions = 4
verbosity_test_cases = 4

Output:

>       assert len(results) == len(expected), results
E       AssertionError: [[no-tabs] (Most files should not contain tabs.) matched examples/playbooks/rule-no-tabs.yml:10 Task/Handler: Foo, [no...hould not contain tabs.) matched examples/playbooks/rule-no-tabs.yml:37 Task/Handler: Should not trigger no-tabs rules]
E       assert 3 == 2
E        +  where 3 = len([[no-tabs] (Most files should not contain tabs.) matched examples/playbooks/rule-no-tabs.yml:10 Task/Handler: Foo, [no-tabs] (Most files should not contain tabs.) matched examples/playbooks/rule-no-tabs.yml:13 Task/Handler: Key has a tab, [no-tabs] (Most files should not contain tabs.) matched examples/playbooks/rule-no-tabs.yml:37 Task/Handler: Should not trigger no-tabs rules])
E        +  and   2 = len((13, 'Most files should not contain tabs.'))

Using latest 8.2.0. I put some breakpoints and tried to find what happens. Apparently the DEFAULT_REPR_MAX_SIZE = 240 and saferepr() is causing it to happen anyway:

  /Users/ssbarnea/c/os/pytest/src/_pytest/python.py(162)pytest_pyfunc_call()
    161     testargs = {arg: funcargs[arg] for arg in pyfuncitem._fixtureinfo.argnames}
--> 162     result = testfunction(**testargs)
    163     if hasattr(result, "__await__") or hasattr(result, "__aiter__"):

  /Users/ssbarnea/c/a/ansible-lint/src/ansiblelint/rules/no_tabs.py(92)test_no_tabs_rule()
     90             assert results[i].lineno == expected[0]
     91             assert results[i].message == expected[1]
---> 92         assert len(results) == len(expected), results

  /Users/ssbarnea/c/os/pytest/src/_pytest/assertion/rewrite.py(454)_format_assertmsg()
    453     if not isinstance(obj, str):
--> 454         obj = saferepr(obj)
    455         replaces.append(("\\n", "\n~"))

  /Users/ssbarnea/c/os/pytest/src/_pytest/_io/saferepr.py(112)saferepr()
    111     """
--> 112     return SafeRepr(maxsize, use_ascii).repr(obj)
    113 

  /Users/ssbarnea/c/os/pytest/src/_pytest/_io/saferepr.py(62)repr()
     61             else:
---> 62                 s = super().repr(x)
     63 

  /Users/ssbarnea/.asdf/installs/python/3.12.2/lib/python3.12/reprlib.py(59)repr()
     58     def repr(self, x):
---> 59         return self.repr1(x, self.maxlevel)
     60 

  /Users/ssbarnea/.asdf/installs/python/3.12.2/lib/python3.12/reprlib.py(67)repr1()
     66         if hasattr(self, 'repr_' + typename):
---> 67             return getattr(self, 'repr_' + typename)(x, level)
     68         else:

  /Users/ssbarnea/.asdf/installs/python/3.12.2/lib/python3.12/reprlib.py(110)repr_list()
    109     def repr_list(self, x, level):
--> 110         return self._repr_iterable(x, level, '[', ']', self.maxlist)
    111 

  /Users/ssbarnea/.asdf/installs/python/3.12.2/lib/python3.12/reprlib.py(98)_repr_iterable()
     97             repr1 = self.repr1
---> 98             pieces = [repr1(elem, newlevel) for elem in islice(x, maxiter)]
     99             if n > maxiter:

  /Users/ssbarnea/.asdf/installs/python/3.12.2/lib/python3.12/reprlib.py(69)repr1()
     68         else:
---> 69             return self.repr_instance(x, level)
     70 

  /Users/ssbarnea/c/os/pytest/src/_pytest/_io/saferepr.py(80)repr_instance()
     79         if self.maxsize is not None:
---> 80             s = _ellipsize(s, self.maxsize)
     81         return s

> /Users/ssbarnea/c/os/pytest/src/_pytest/_io/saferepr.py(29)_ellipsize()
     28     breakpoint()
---> 29     if len(s) > maxsize:
     30         i = max(0, (maxsize - 3) // 2)

[ncoghlan]

I became sufficiently frustrated with this limitation that I'm currently porting some tests over to unittest specifically to reliably obtain comprehensive diffs when the tests fail (the tests are comparing metadata structs, so they're not useful if the diffs get truncated)

One of the many issues I encountered while trying to find a definitive answer on how to get native pytest tests to reliably print full diffs (akin to unittest's maxDiff=None without dialling up the entire test suite execution to "spam me now" with -vv had suggested that assert_truncate_level=0 should work, but it turned out that was just a suggestion in a PR rather than a change that actually landed (the explicit warning about an unknown config setting was very useful in that regard, so thank you for that feature!).

It also isn't clear whether any of the suggested workarounds affect the error summary report at the end of the test suite execution (that is getting truncated even when the tests are implemented in unittest)

[nicoddemus]

Sorry folks, seems we dropped the ball on this one. Opened #12662 with a proposed fix.

[ncoghlan]

Thanks @nicoddemus!

I agree that combined with the previous reference to https://docs.pytest.org/en/stable/reference/reference.html#confval-verbosity_assertions that change should cover both this issue and #12307