pytest==8.3.1 collects tests from Python dependencies within conda environments (regression from 8.2.2)

[joshuacwnewton]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
  • NB: Before and after failing CI runs: https://www.diffchecker.com/SReU8wCR/
• pytest and operating system versions
  • 8.3.1, multiple different GitHub Actions runners (windows-2019, ubuntu-2022, etc.)
• minimal example if possible

---

We have the following setup.cfg file:

[tool:pytest]
addopts = --verbose --show-capture=stderr --tb=native
python_files = testing/**/test_*.py
# NB: norecursedirs is *not* set

When running pytest (with no additional options) in the root of our repository, we see the following behavior:

• pytest==8.2.2: Only our tests are collected.
• pytest==8.3.1: Our tests and the tests of our dependencies are collected, leading to many errors.
  • collecting ... collected 367457 items / 408 errors / 100 skipped (Woof!)

---

Changing our setup.cfg to the following didn't fix the problem either:

[tool:pytest]
addopts = --verbose --show-capture=stderr --tb=native
testpaths = testing
python_files = test_*.py

The fix I used was adding norecursedirs = {name_of_venv}

---

Since this involves collection in virtual environments, the reason for the regression might be related to changes in:

• Support venv detection on Windows with mingw Python #12545

Notably, we use a conda env with pip packages installed inside, which may subvert the expectations of the changes in the above PR.

[bluetech]

Does the venv directory contain a pyvenv.cfg file?

[joshuacwnewton]

Yes!

# In root of project folder (where tests are located at `./testing`)
master|joshua@monarch:~/repos/spinalcordtoolbox/$ find . -name pyvenv.cfg
./python/envs/venv_sct/pyvenv.cfg

[bluetech]

Interesting. Any chance you can poke the _in_venv function https://github.com/pytest-dev/pytest/blob/8.3.1/src/_pytest/main.py#L370 to see why it fails for this path ./python/envs/venv_sct/?

[joshuacwnewton]

In a Python console:

>>> from pathlib import Path
>>> def _in_venv(path: Path) -> bool:
    """Attempt to detect if ``path`` is the root of a Virtual Environment by
    checking for the existence of the pyvenv.cfg file.
    [https://peps.python.org/pep-0405/]"""
    try:
        return path.joinpath("pyvenv.cfg").is_file()
    except OSError:
        return False
    
>>> import os
>>> os.getcwd()
'/home/joshua/repos/spinalcordtoolbox'
>>> _in_venv(Path("./python/envs/venv_sct/"))
True

Exploring more on my end too... 🤔

[joshuacwnewton]

Hmm. On my local Linux machine, I get fewer errors. Windows specifically reports many more errors... so I might have to test on a Windows machine. See:

• CI is failing with pytest-related import errors during the collection stage spinalcordtoolbox/spinalcordtoolbox#4570

[Hemario]

Just chiming in. We run automated tested in a docker container and on that container, we load the conda prefix inside the project folder. We could probably have done this better, but it was working for us in the past.

/projects/projectX: the git repository location
/projects/projectX/conda: the conda prefix

Since version 8.3.1 we notice the same issue as described above. Tests from included packages are being discovered.

With pytest 8.2.2: collected 190 items / 1 deselected / 189 selected
With pytest 8.3.1: collected 85611 items / 157 errors / 1 deselected / 29 skipped / 85610 selected

We solved it by adding to pytest.ini

addopts = 
   --ignore=conda

[RonnyPfannschmidt]

it seems like checking only for modern venv left us wanting for conda - is there a similar easy check we could do for conda prefixes/envs -- i'll reach out to conda on fosstodon to get input on fixing this, we might have to undo the steamlining for now

[jezdez]

For the record, I'm responding to @RonnyPfannschmidt reaching out on Mastodon (https://fosstodon.org/@ossronny/112837662322781627), we'll take a look at this ASAP.

[jezdez]

@RonnyPfannschmidt Can you provide more details other than what was said here, which feature was added in 8.3.1 that triggered this?

[jezdez]

I've filed conda/conda#14069

[jezdez]

Looks like #12544/#12545 is the culprit that reduces virtualenv detection to look for pyvenv.cfg only and effectively removes conda environment detection.

Given that we can't easily retrofit existing conda environments out there, and a significant number of users are using pytest within conda environments, I'd appreciate it, if you could revert the patch, or at least retrofit it with the previous detection mechanism.

[RonnyPfannschmidt]

as per #12545 (comment) - i'd slightly prefer adding explicit conda detection over reverting - in case thats not easy/feasible quickly we go with a revert

[RonnyPfannschmidt]

@jezdez so the key question - is there a easy foolproof way to check for conda explicitly - even if a pyvenv.cfg was not yet retrofitted

[jezdez]

@jezdez is there a good way to check for conda prior to that change? im happy to roll back for now, but if a consistent bugfix that ensures compatibility with older conda is easy, i'd prefer landing that over a revert dance

The best course of action would probably be checking for path.joinpath("conda-meta", "history").is_file() for now, as that's required to exist for conda envs to use the conda environment revision system (conda list --revisions and conda install --revision REV).

[RonnyPfannschmidt]

working on a bugfix now, venv detection will be expanded to ("conda-meta", "history")

[RonnyPfannschmidt]

released