Add a --python flag to allow installation into arbitrary Python interpreters

[charliermarsh]

Summary

This PR adds a --python flag that allows users to provide a specific Python interpreter into which uv should install packages. This would replace the VIRTUAL_ENV= workaround that folks have been using to install into arbitrary, system environments, while also actually being correct for installing into non-virtual environments, where the bin and site-packages paths can differ.

The approach taken here is to use sysconfig.get_paths() to get the correct paths from the interpreter, and then use those for determining the bin and site-packages directories, rather than constructing them based on hard-coded expectations for each platform.

Closes #1396.

Closes #1779.

Closes #1988.

Test Plan

• Verified that, on my Windows machine, I was able to install requests into a global environment with: cargo run pip install requests --python 'C:\\Users\\crmarsh\\AppData\\Local\\Programs\\Python\\Python3.12\\python.exe, then python and import requests.
• Verified that, on macOS, I was able to install requests into a global environment installed via Homebrew with: cargo run pip install requests --python $(which python3.8).

[charliermarsh]

Marking as draft, one important thing to fix here when creating environments for isolated builds (hence test failures). I see the problem.

[charliermarsh]

\cc @konstin - I think include is the same in a venv as not...

[konstin]

Odd, the docs make it look as if should also change.

[charliermarsh]

Which docs? Link please!

[konstin]

https://docs.python.org/3/library/sysconfig.html#installation-paths

But i don't disagree, include is also an outlier for me:

Paths: 
        data = "/home/konsti/projects/uv/.venv"
        include = "/home/konsti/.pyenv/versions/3.10.13/include/python3.10"
        platinclude = "/home/konsti/.pyenv/versions/3.10.13/include/python3.10"
        platlib = "/home/konsti/projects/uv/.venv/lib/python3.10/site-packages"
        platstdlib = "/home/konsti/projects/uv/.venv/lib/python3.10"
        purelib = "/home/konsti/projects/uv/.venv/lib/python3.10/site-packages"
        scripts = "/home/konsti/projects/uv/.venv/bin"
        stdlib = "/home/konsti/.pyenv/versions/3.10.13/lib/python3.10"

Same with the apt installed python as base:

Paths: 
        data = "/home/konsti/projects/uv/.venv"
        include = "/usr/include/python3.11"
        platinclude = "/usr/include/python3.11"
        platlib = "/home/konsti/projects/uv/.venv/lib/python3.11/site-packages"
        platstdlib = "/home/konsti/projects/uv/.venv/lib/python3.11"
        purelib = "/home/konsti/projects/uv/.venv/lib/python3.11/site-packages"
        scripts = "/home/konsti/projects/uv/.venv/bin"
        stdlib = "/usr/lib/python3.11"

[samypr100]

Depends on the build headers I think? There was an issue a while ago with deadsnakes changing them to /usr/local/include vs canonical /usr/include the headers had deadsnakes/issues#237

[charliermarsh]

Gonna put up some follow-up PRs. Merging for now.

[charliermarsh]

518e74b contains a test script and a GitHub Actions workflow to install on system python on macOS and Linux. (I tried to get it to work on Windows, but I wasted too much time trying to get GitHub Actions to work.)

[wimglenn]

Hi @charliermarsh , this is a very useful feature, thanks! I have a couple of questions on usage.

Q1: is it expected that using a relative path is not working (edit: on macOS - seems to work fine on Linux)?

$ uv pip install --dry-run --python .venv/bin/python six 
error: Failed to query Python interpreter at `.venv/bin/python`
  Caused by: No such file or directory (os error 2)

An absolute path to the same python exe is working:

$ pwd
/Users/wim
$ uv pip install --dry-run --python /Users/wim/.venv/bin/python six
Audited 1 package in 1ms
Would make no changes

edit: I can't reproduce this on 0.1.22. I checked again after downgrading back to 0.1.21 (where I originally saw the issue) and could no longer reproduce it there either... 🤷

Q2: how do you actually use this feature with uv pip compile? I couldn't figure it out:

$ cat reqs.in                                                            
psycopg2-binary; implementation_name == 'cpython'
psycopg2; implementation_name != 'cpython'
$ uv pip compile reqs.in --python-version /tmp/pypy/.venv/bin/python
error: invalid value '/tmp/pypy/.venv/bin/python' for '--python-version <PYTHON_VERSION>': expected version to start with a number, but no leading ASCII digits were found

For more information, try '--help'.
$ uv pip compile reqs.in --python /tmp/pypy/.venv/bin/python        
error: unexpected argument '--python' found

  tip: a similar argument exists: '--python-version'

Usage: uv pip compile --python-version <PYTHON_VERSION> <SRC_FILE>...

For more information, try '--help'.

It works with uv pip install as expected.

$ uv pip install --dry-run --python /tmp/cpython/.venv/bin/python -r reqs.in
Resolved 1 package in 3ms
Would download 1 package
Would install 1 package
 + psycopg2-binary==2.9.9
$ uv pip install --dry-run --python /tmp/pypy/.venv/bin/python -r reqs.in   
Resolved 1 package in 3ms
Would download 1 package
Would install 1 package
 + psycopg2==2.9.9