pytest.warns hides type errors in warnings.warn calls

[zanieb]

• a detailed description of the bug or problem you are having
• [ ] output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

When using pytest.warns, incorrect values sent to warnings.warn are not reported obscuring bugs:

import pytest
import warnings


def test_example_one():
    with pytest.warns(UserWarning):
        warnings.warn(1)

    # This test passes but should raise a `TypeError` as demonstrated in `example_two`


def test_example_two():
    with pytest.raises(TypeError):
        warnings.warn(1)

macOS 13.2.1
pytest 7.2.2
python 3.11

Related to #9288

[nicoddemus]

Hmm that's unfortunate, thanks for the report!

Should be simple to fix.

[zanieb]

You're welcome! Certainly confused me for a minute :)

I'm happy to contribute a fix — want to just raise a type error if it's not a string? Looks like CPython allows Warning types too https://github.com/python/cpython/blob/f513d5c80672c76acbdaf7d5b601f4bbe9fae56a/Lib/warnings.py#L299-L301

[zanieb]

While trying to find the logic where this TypeError is raised I've gotten a bit confused. Weirdly this is fine

❯ python -c "import warnings; warnings.warn(1)"
<string>:1: UserWarning: 1

but the TypeError is definitely raised in pytest i.e. if I remove the pytest.raises:

    def test_example_two():
>       warnings.warn(1)
E       TypeError: expected string or bytes-like object, got 'int'

I feel like this caused me an error outside of pytest as well though

[nicoddemus]

Are you sure? This works for me:

import warnings
def test_example_two():
    warnings.warn(1)

Which Python and pytest versions are you using?

[zanieb]

The Python and pytest versions are included in the original post, but consider me very confused — I did not reproduce this with the minimal example and a test matrix at https://github.com/madkinsz/pytest-10865/actions/runs/4714285419/jobs/8360551859

It does however reproduce quite clearly when I'm working in PrefectHQ/prefect. Here's the example running in our test suite PrefectHQ/prefect#9231

[zanieb]

The issue appears to be related to filterwarnings, for example:

import warnings

# Does not raise a type error
warnings.warn(1)

# Raises a type error
with warnings.catch_warnings():
    warnings.filterwarnings("ignore", "test")
    warnings.warn(1)

with the addition of a warnings filter to pytest zanieb/pytest-10865@1c3e4e3 the type error is raised on all Python and 7.x pytest versions https://github.com/madkinsz/pytest-10865/actions/runs/4714336351

[zanieb]

I've opened a CPython issue as this doesn't really seem like pytest's fault python/cpython#103577 but pytest probably shouldn't hide this error still.

[zanieb]

@nicoddemus I don't expect there to be changes upstream for this. Should we validate that argument is of the correct type in pytest?

[RonnyPfannschmidt]

as far as im concerned, type metadata in typeshed is provided, and my understandig is, that it would point to the error

[zanieb]

@RonnyPfannschmidt the problem here is that when using filterwarnings the TypeError is raised outside of pytest but not raised inside pytest.warns blocks. This means that an error that can occur at runtime for users is hidden during execution of the test suite. I agree that this mistake would also be caught by a type-checker, but many people do not use static analysis tools and it feels rough to obscure the error that the standard library would raise.

[Zac-HD]

Retagging as bug based on upstream discussion, where consensus looks like "this should always error" and CPython may change accordingly.

[whysage]

Hi, please review
#11804

[whysage]

Hi, any comments on pull request?
#11804
@zanieb @Zac-HD