OSError: [Errno 39] Directory not empty when doing cache.mkdir() from session fixture + pytest-xdist

[amezin]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

cache.mkdir() in a session-level fixture, in combination with xdist, sometimes results in OSError: [Errno 39] Directory not empty when pytest tries to rename the cache directory.

This is a regression in 8.2.0, most likely introduced by #12168 / #12167

On 8.1.x cache.mkdir() works fine in the same case.

In conftest.py:

@pytest.fixture(scope='session')
def my_fixture(request):
    return request.config.cache.mkdir('testdir')

In test_tests.py:

@pytest.mark.parametrize('n', list((n,) for n in range(42)))
def test_test(my_fixture, n):
    pass

Running with pytest -n auto results in:

self = PosixPath('/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/pytest-cache-files-8ulls0nm')
target = PosixPath('/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/.pytest_cache')

    def rename(self, target):
        """
        Rename this path to the target path.
    
        The target path may be absolute or relative. Relative paths are
        interpreted relative to the current working directory, *not* the
        directory of the Path object.
    
        Returns the new Path instance pointing to the target path.
        """
>       self._accessor.rename(self, target)
E       OSError: [Errno 39] Directory not empty: '/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/pytest-cache-files-8ulls0nm' -> '/home/runner/work/pytest-cache-race/pytest-cache-race/.tox/python/.pytest_cache'

Operating System: Ubuntu 22.04.4 LTS
Python 3.10.12, pytest-8.2.1
Virtualenv content: exceptiongroup==1.2.1,execnet==2.1.1,iniconfig==2.0.0,packaging==24.0,pluggy==1.5.0,pytest==8.2.1,pytest-xdist==3.6.1,tomli==2.0.1

https://github.com/amezin/pytest-cache-race

Reproduced in GitHub Actions: https://github.com/amezin/pytest-cache-race/actions/runs/9261012859/job/25475719651

[bluetech]

Hmm, so now we have a different race. I think we should be able to ignore this error, i.e. if it happens just continue (while removing the directory). This is because we can assume that the first worker correctly created the directory atomically.