Hatch doesn’t find Python versions other than its own and `…/python{,3}` on Linux

[flying-sheep]

If I configure a project with requires_python '>=3.12', on my system I see this error:

hatch/src/hatch/env/virtual.py

Line 258 in 8648544

else 'no compatible Python distribution available'

I don’t understand what the code is supposed to do. It requires either an env variable or one of two functions to not return None:

1. _get_available_distribution seems to call hatch.python.resolve.get_compatible_distributions, and then it removes the “installed distribution”

   hatch/src/hatch/env/virtual.py

   Lines 374 to 376 in 8648544

   	compatible_distributions = get_compatible_distributions()
   	for installed_distribution in self.python_manager.get_installed():
   	compatible_distributions.pop(installed_distribution, None)

   get_compatible_distributions() returns everything hatch can download:

   $ ~/.local/pipx/venvs/hatch/bin/python -c '
   ❯ from hatch.python.resolve import get_compatible_distributions
   ❯ print(get_compatible_distributions().keys())'
   dict_keys(['3.7', '3.8', '3.9', '3.10', '3.11', '3.12', 'pypy2.7', 'pypy3.9', 'pypy3.10'])

   In my case self.python_manager.get_installed().keys() == {'3.12'}, so _get_available_distribution returns None

2. _find_existing_interpreter calls virtualenv.discovery.builtin.propose_interpreters, which yields this on my system:

   PythonInfo(spec=CPython3.11.8.final.0-64, system=/usr/bin/python3.11, exe=/home/phil/.local/pipx/venvs/hatch/bin/python, platform=linux, version='3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]', encoding_fs_io=utf-8-utf-8)
   PathPythonInfo(spec=CPython3.11.8.final.0-64, exe=/usr/bin/python, platform=linux, version='3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]', encoding_fs_io=utf-8-utf-8)

   so that function also returns None.

I can work around this using

pipx install --python=python3.12 hatch

but this should still be fixed. I’d be happy to help, but I don’t know what the intended semantics of _get_available_distribution and _find_existing_interpreter are: Which one of these is supposed to return /usr/bin/python3.12?

[flying-sheep]

When I run rm -rf /home/phil/.local/share/hatch/env/virtual/.pythons/, on the next hatch run, Hatch re-downloads Python 3.12, and it works until the next hatch prune.

I wonder why Hatch downloads Python 3.12 when /usr/bin/python3.12 exists and is in the PATH?

[ofek]

Is this issue present on master?

[flying-sheep]

yes, Hatch 1.9.2.dev93 (8648544) throws the same error (I didn’t check if the APIs return the same values as in 1.9.2)

[ofek]

Please show me:

from virtualenv.discovery.builtin import propose_interpreters
from virtualenv.discovery.py_spec import PythonSpec

def main():
    for data in propose_interpreters(PythonSpec.from_string_spec('3.12'), (), None):
        interpreter = data[0]
        version = '.'.join(map(str, interpreter.version_info[:3]))
        print(f'{version} -> {interpreter.executable}')

if __name__ == '__main__':
    main()

[flying-sheep]

3.11.8 -> /home/phil/.local/pipx/venvs/hatch/bin/python
3.12.2 -> /usr/bin/python3.12
3.11.8 -> /usr/bin/python3
3.11.8 -> /usr/bin/python

[ofek]

That conflicts with what happened in the opening description. Is this a new terminal session? Has PATH been updated since the first one which is why originally there was no 3.12?

[flying-sheep]

OK, I have now slept and executed it with the correct interpreter lol. Edited above with the correct output.

For context on why your script output differs from the hatch invocation: When running hatch run ..., _find_existing_interpreter is only called once, with python_version == ''.

So propose_interpreters is called with PythonSpec.from_string_spec('') instead of ...('3.12').

[ofek]

Can you please then show the output of the empty string change?

[flying-sheep]

When replacing '3.12' with '', it prints:

3.11.8 -> /home/phil/.local/pipx/venvs/hatch/bin/python
3.11.8 -> /usr/bin/python

I think the problem is that it doesn’t find all binaries on the $PATH:

$ ls /usr/bin/python* | where name =~ '\d$'
╭───┬─────────────────────┬─────────┬─────────┬───────────────╮
│ # │        name         │  type   │  size   │   modified    │
├───┼─────────────────────┼─────────┼─────────┼───────────────┤
│ 0 │ /usr/bin/python3    │ symlink │    10 B │ 2 months ago  │
│ 1 │ /usr/bin/python3.10 │ file    │ 14.3 KB │ 3 weeks ago   │
│ 2 │ /usr/bin/python3.11 │ file    │ 14.4 KB │ 2 months ago  │
│ 3 │ /usr/bin/python3.12 │ file    │ 14.2 KB │ 2 months ago  │
│ 4 │ /usr/bin/python3.7  │ file    │ 14.2 KB │ 10 months ago │
│ 5 │ /usr/bin/python3.8  │ file    │ 14.3 KB │ 3 weeks ago   │
│ 6 │ /usr/bin/python3.9  │ file    │ 14.3 KB │ 3 weeks ago   │
╰───┴─────────────────────┴─────────┴─────────┴───────────────╯

[ofek]

That's very interesting, I interpreted empty string as the way to get everything, which I suppose does work on Windows because there is never a version suffix! Oh my

[ofek]

what happens when you try just '3'?

[flying-sheep]

Still only 3.11 found

[flying-sheep]

@gaborbernat is started to extract this from virtualenv recently (https://github.com/tox-dev/py-discovery), maybe he’s immersed deeply enough into the code to help here.

[ofek]

Yes @gaborbernat what is the magic first argument spec I must pass to virtualenv.discovery.builtin.propose_interpreters so that it returns everything? So far it's not an empty string nor '3'. Here is the script:

from virtualenv.discovery.builtin import propose_interpreters
from virtualenv.discovery.py_spec import PythonSpec

def main():
    for data in propose_interpreters(PythonSpec.from_string_spec('3'), (), None):
        interpreter = data[0]
        version = '.'.join(map(str, interpreter.version_info[:3]))
        print(f'{version} -> {interpreter.executable}')

if __name__ == '__main__':
    main()

[flying-sheep]

I don’t think there is any.

• possible_specs(PythonSpec.from_string_spec('')) yields ('', False), ('python', True)
• possible_specs(PythonSpec.from_string_spec('3')) yields ('3', False), ('python3', True), ('python', True)

So these are searched in the PATH. Seems like virtualenv.discvery.builtin doesn’t support finding all Python interpreters on Linux.