Several tests related to Python distributions fail on Fedora Linux on non-x86_64 architectures in Hatch 1.12.0

[musicinmybrain]

I’m testing an update from python-hatchling-1.21.1 to python-hatchling-1.25.0 and hatch-1.9.7 to hatch-1.12.0 in Fedora Rawhide. (This has been stalled for a few months by the need to package uv, but that’s done now.)

All tests pass on x86_64.

On ppc64le:

FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''

On aarch64:

FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='aarch64' abi='gnu' variant=''

On s390x:

FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''

(We don’t build noarch packages on i386/i686 anymore, so I didn’t test it.)

This resembles #1145.

[musicinmybrain]

We do have an older python-virtualenv-20.21.1 with some backported patches, but currently without pypa/virtualenv#2709 or pypa/virtualenv#2712 (#1395 (comment)). I’m testing with the lower bound on the version of virtualenv removed. I didn’t think that should be affecting this, but I thought it was worth mentioning. To be certain, I tried reproducing this in a Fedora Rawhide chroot on ppc64le using qemu-user-static emulation. The packages gcc cargo python3-devel openssl-devel openssl-devel-engine are needed because cryptography still has to be built from source on Python 3.13. I can’t use hatch test, or install xdist[psutil], because psutil also still has to be built from source, and it looks for gcc in the wrong place (/usr/bin/powerpc64le-linux-gnu-gcc), as if it were being cross-compiled.

$ mock -r fedora-rawhide-ppc64le --clean
$ mock -r fedora-rawhide-ppc64le -i  git-core cargo python3-devel openssl-devel openssl-devel-engine
$ mock -r fedora-rawhide-ppc64le --shell --enable-network
# git clone https://github.com/pypa/hatch.git
# cd hatch
# git checkout hatch-v1.12.0
# python3 -m venv _e
# . _e/bin/activate
(_e) # python --version
Python 3.13.0rc1
(_e) # pip install -e .
(_e) # pip install pytest pytest-mock pytest-randomly pytest-rerunfailures pytest-xdist editables
(_e) # python -m pytest -n auto
[…]
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/cli/test/test_test.py::TestCustomScripts::test_single - AssertionError: assert [call('test h..., shell=True)] == [call('test h..., shell=True)]
FAILED tests/cli/run/test_run.py::TestScriptRunner::test_python_version_constraint - AssertionError: assert 'Creating env...ependencies\n' == 'Creating env...ependencies\n'
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
ERROR tests/utils/test_auth.py::test_pypirc

This is slightly more failures than I reported – some, like TestCustomScripts::test_single, appear to be artifacts of how I’m running the tests here, and some perhaps are because this test had network access, unlike Fedora package builds – but it’s still safe to say that this basically reproduces the issue. Since this test used an unpatched hatch/hatchling and dependencies from PyPI and respected all version bounds, it seems safe to say that Fedora’s python-virtualenv-20.21.1 is not involved.

[musicinmybrain]

Although Fedora 40 won’t get a Hatch update, I repeated the experiment from #1670 (comment) in a Fedora 40 ppc64le chroot, where the system Python is 3.12. I still have to compile cryptography from source. I thought that would have been a Python 3.13 thing, but instead it turns out that cryptography only releases binary wheels for x86_64 and aarch64.

$ mock -r fedora-40-ppc64le --clean
$ mock -r fedora-40-ppc64le -i git-core cargo python3-devel openssl-devel
$ mock -r fedora-40-ppc64le --shell --enable-network
# git clone https://github.com/pypa/hatch.git
# cd hatch
# git checkout hatch-v1.12.0
# python3 -m venv _e
# . _e/bin/activate
(_e) # python --version
Python 3.12.4
(_e) # pip install -e .
(_e) # hatch test
[…]
=========================================================================================== short test summary info ============================================================================================
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
=========================================================================== 12 failed, 1996 passed, 69 skipped in 582.37s (0:09:42) ============================================================================

[musicinmybrain]

Repeating the Fedora 40 / Python 3.12 test from #1670 (comment) for s390x gives:

=========================================================================================== short test summary info ============================================================================================
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
=========================================================================== 12 failed, 1996 passed, 69 skipped in 634.55s (0:10:34) ============================================================================

…and aarch64 gives:

=========================================================================================== short test summary info ============================================================================================
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='aarch64' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='aarch64' abi='gnu' variant=''
=========================================================================== 2 failed, 2006 passed, 69 skipped in 4242.38s (1:10:42) ============================================================================