Tests failing on PyPy

[jaraco]

These tests passed on pypy in #4630, but failed on the release job (https://github.com/pypa/setuptools/actions/runs/10692324985).

Originally posted by @jaraco in #4630 (comment)

[jaraco]

I also note that PyPy went from 7.3.16 to 7.3.17 between those builds.

[jaraco]

What's even more weird is that those tests are run with SETUPTOOLS_USE_DISTUTILS=stdlib, so the failing test should be skipped.

I'm attempting to replicate the issue using act --job test --matrix python:pypy3.10 --matrix platform:ubuntu-latest --matrix distutils:stdlib --container-architecture linux/amd64

[jaraco]

The first attempt to repro ran into an issue with some of the workers failing then it just hung:

4.1, subprocess-1.5.2, checkdocs-2.13.0
| Warning: cannot find your CPU L2 cache size in /proc/cpuinfo
created: 8/8 workersWarning: cannot find your CPU L2 cache size in /proc/cpuinfo
initialized: 1/8 workersWarning: cannot find your CPU L2 cache size in /proc/cpuinfo
initialized: 2/8 workersWarning: cannot find your CPU L2 cache size in /proc/cpuinfo
initialized: 3/8 workersWarning: cannot find your CPU L2 cache size in /proc/cpuinfo
initialized: 4/8 workersWarning: cannot find your CPU L2 cache size in /proc/cpuinfo
initialized: 5/8 workersWarning: cannot find your CPU L2 cache size in /proc/cpuinfo
initialized: 6/8 workersWarning: cannot find your CPU L2 cache size in /proc/cpuinfo
8 workers [1525 items]  
| ss.s.................................................................... [  4%]
| ........................................................................ [  9%]
| .................................................................x...... [ 14%]
| .....................................................F.................. [ 18%]
| ....................................................................s... [ 23%]
| ..............s.................................X....................... [ 28%]
| ....X........................................s................s.s.s.sss. [ 32%]
| .ss.ss.s................................................................ [ 37%]
| ........................................................................ [ 42%]
| ............x...................x.........x....x........................ [ 47%]
| ............................................................x........x.. [ 51%]
| ..........x[gw4] node down: Not properly terminated
| F
| replacing crashed worker gw4
| Warning: cannot find your CPU L2 cache size in /proc/cpuinfo
collecting: 8/9 workers.[gw6] node down: Not properly terminated
| F
| replacing crashed worker gw6
| Warning: cannot find your CPU L2 cache size in /proc/cpuinfo
10 workers [1525 items] .[gw2] node down: Not properly terminated.s..x.............
| F
| replacing crashed worker gw2
| Warning: cannot find your CPU L2 cache size in /proc/cpuinfo
collecting: 10/11 workers..[gw3] node down: Not properly terminated
| F
| replacing crashed worker gw3
| Warning: cannot find your CPU L2 cache size in /proc/cpuinfo
collecting: 11/12 workers[gw1] node down: Not properly terminated
| F
| replacing crashed worker gw1
| Warning: cannot find your CPU L2 cache size in /proc/cpuinfo
^C[tests/test]   ❌  Failure - Main Run

Not idea what that's all about. Maybe my system went to sleep when I walked away.

[jaraco]

It hung again a second time, this time with only two but again, I walked away during the run. It takes a very long time. I'll try again.

Ugh. Tried again and this time it failed early. I think it may be running low on resources, but I have no idea which.

[jaraco]

In https://github.com/pypa/setuptools/actions/runs/10710763767/job/29698171478, I've added some debugging and confirmed that SETUPTOOLS_USE_DISTUTILS=local, so something must be munging the environment.

[jaraco]

When I run the test in isolation, the test is skipped as expected. So there is indeed some interference from other tests.

[jaraco]

In https://github.com/pypa/setuptools/actions/runs/10710992997/job/29698838140, I added a fixture to check that os.environ['SETUPTOOLS_USE_DISTUTILS'] doesn't change for any given test, and none of the tests failed that check, so how is that possible?

[jaraco]

The failure doesn't happen for me locally, even with the same PyPy version.

[jaraco]

In https://github.com/pypa/setuptools/actions/runs/10712709409/job/29703544680, I've had some success detecting when the env var changes. It looks like it's one of the tests in test_integration is altering it. That sort-of makes sense, as it's running code from outside the repo within the environment, so it can mutate the environment. But why PyPy only, and why CI only?

[jaraco]

In 84a0079, I ran the tests sequentially so that the offending test can be identified. The job output implicates test_novaclient (or maybe test_pbr):

setuptools/tests/test_integration.py::test_pbr PASSED                    [ 83%]
setuptools/tests/test_integration.py::test_python_novaclient XFAIL       [ 83%]
setuptools/tests/test_integration.py::test_python_novaclient XFAIL       [ 83%]
setuptools/tests/test_integration.py::test_pyuri ERROR                   [ 83%]

[jaraco]

Something else that's odd and interesting - that novaclient test is run twice. Why would it run twice?

[jaraco]

In https://github.com/pypa/setuptools/actions/runs/10713178150/job/29704803966, I've proven that it's the test_python_novaclient that's triggering the issue:

py: commands[0]> pytest -p no:xdist -v -k 'pyuri or novaclient'
============================= test session starts ==============================
platform linux -- Python 3.10.14[pypy-7.3.17-final], pytest-8.3.2, pluggy-1.5.0 -- /home/runner/work/setuptools/setuptools/.tox/py/bin/python
cachedir: .tox/py/.pytest_cache
rootdir: /home/runner/work/setuptools/setuptools
configfile: pytest.ini
plugins: checkdocs-2.13.0, ruff-0.4.1, cov-5.0.0, jaraco.vcs-2.4.0, perf-0.15.0, home-0.6.0, xdist-3.6.1, subprocess-1.5.2, jaraco.test-5.4.0, enabler-3.2.0, mypy-0.10.3, timeout-2.3.1
collecting ... collected 1525 items / 1523 deselected / 2 selected
setuptools/tests/test_integration.py::test_python_novaclient XFAIL       [ 50%]
setuptools/tests/test_integration.py::test_python_novaclient XFAIL       [ 50%]
setuptools/tests/test_integration.py::test_pyuri ERROR                   [100%]
==================================== ERRORS ====================================
_________________________ ERROR at setup of test_pyuri _________________________
    @pytest.fixture(autouse=True)
    def check_setuptools_use_distutils():
        import os
    
        orig = os.environ.get('SETUPTOOLS_USE_DISTUTILS')
>       assert orig == 'stdlib'
E       AssertionError: assert 'local' == 'stdlib'
E         
E         - stdlib
E         + local
conftest.py:84: AssertionError

[jaraco]

Okay. That test is marked as xfail, and if that tests fails twice (normally, then an error in teardown), then it will be reported twice (as xfail). If I run with --runxfail, the causes are clearer:

FAILED setuptools/tests/test_integration.py::test_python_novaclient - DeprecationWarning: The 'wheel' package is no longer the canonical location...
ERROR setuptools/tests/test_integration.py::test_python_novaclient - RuntimeError: SETUPTOOLS_USE_DISTUTILS changed from stdlib to local.
ERROR setuptools/tests/test_integration.py::test_pyuri - AssertionError: assert 'local' == 'stdlib'

That's why I didn't see the "changed from stdlib to local" sooner - the xfail marker was masking it.

I scanned the novaclient code for the environment variable, but I see nothing referencing it, but I'm seeing now there are other packages involved (dependencies, presumably).

[jaraco]

In 99e58a8, I've confirmed that pyyaml is implicated, and indeed pyyaml sets the variable in its setup.py (ugh). But that still doesn't explain why the issue doesn't occur on other platforms. Maybe it has something to do with wheels and caching?

In any case, I'm mostly uninterested in fixing this test. It's been marked as xfail for some time and is testing a deprecated code path (easy_install), so let's just get rid of it (and probably the other integration tests for easy_install).

[jaraco]

For posterity, I pushed the commits for this issue to refs/archive/triage/4632