Revisit the installation strategy

[jaraco]

In #100 (comment), I describe the current pip-run installation strategy: pip-run is most useful when installed to each and every environment in which it might run. This approach has the benefit of simply allowing the command to run in the current interpreter and environment (e.g. sys.executable -m pip). This strategy mimics the strategy of pip, which typically needs to be present in the environment to be used in that environment.

With the advent of uv, however, there's a different paradigm, where a tool might exist that runs across different interpreters and environments. If pip-run followed this strategy, it could ease the adoption of a tool like uv but could also drastically simplify the (under-documented) expectation about how to use pip-run across multiple Python environments.

[jaraco]

In a related side-note, I've recently changed my personal installation strategy. Previously, I would install pip-run into the system site-packages for each of my Python interpreters so that I could do things like py -3.10 -m pip-run to run with dependencies under Python 3.10.

More recently, however, instead of installing pip-run to each environment, I've been installing it using pip install --python 3.8 -t ~/.local/pip-run and then setting PYTHONPATH=~/.local/pip-run. That is, I've created a single install and made it available to every interpreter. I had to use --python 3.8 to get all of the dependencies (assuming that the oldest python is the superset of all dependencies). This approach has been working, but it's messy and undocumented. It sure would be nicer to just pipx install pip-run and then be able to use it from there for any environment.

[jaraco]

In jaraco/multipy-tox@2a04a7b, I applied the same technique to the multipy-tox environment. And while this had some simplifying and unifying effects for the installation, it also had some undesired effects. In particular:

• Because PYTHONPATH is set and takes precedence over site-packages, the pip package as depended by pip-run is now found at ~/.local/pip-run, masking any system-level pip.
• Making pip-run available as a first-class command required altering the scripts after installing them.
• Because it's setting PYTHONPATH, it nicely is available in any Python (e.g. py -3.9 or python3.9 or .venv/bin/python), but that also means that one needs to disable it if they want a pristine environment (such as in Simplify vendoring and declare dependencies optionally pypa/setuptools#4457 (comment)).
• Because pip-run is being installed for Python 3.7, that means that a version of pip compatible with Python 3.7 will be installed, which may be older than what's available for the latest Pythons.

All of these undesired effects would be removed if pip-run could run as an application, a tool, and would operate on some target environment.

If only there was a spec or a standard and a reference implementation for tools to know on which environment to operate.

pip has a --python option to override the target environment. Unfortunately, the user ergonomics of that are clumsy. It requires a path to an executable, which draws in platform-specific behaviors and can't honor the python launchers and their interpreter discovery mechanisms.

Here's something pip-run could do:

• Allow providing a specific Python (like pip does, and probably honor PIP_PYTHON).
• If no Python is specified, and pip-run appears to be installed into a virtual environment, discover the environment to use thus:
  • If a Python launcher is available, use that to locate Python (e.g. py -c "import sys; print(sys.executable)").
  • Otherwise, re-implement the discovery that the Unix Python launcher uses.

[jaraco]

If only there was a spec or a standard and a reference implementation for tools to know on which environment to operate.

In #100, @pfmoore writes:

I'd like to see a standard behaviour for tools when they need to discover the appropriate Python environment to work on. I think uv has a fairly reasonable approach although I'm sure that if we tried to standardise this, people would have differing views 🙂 For Python-based tools, there's historically been a tendency to use "the interpreter I'm running with" as a default, but that (a) doesn't translate well to non-Python tools, and (b) pollutes the user's environment with the tool's code and dependencies.

[pfmoore]

My view, copied over from #100:

---

While I understand where you're coming from, I don't think it's the job of this tool to resolve the problem of how people invoke Python. That's very much the core developers issue, and if they don't resolve it, IMO we should work with existing conventions, no matter how much we may disagree with them, rather than fighting the ecosystem.

With that in mind, my view is:

1. Platform differences are inevitable, simply because the relevant PEPs (394 for Unix, 397 for Windows) mandate different behaviours across platforms.
2. That means that, in the absence of an activated virtual environment, the "system environment" for Windows users is whatever the py launcher picks, or if the launcher isn't available, whatever usable python command is available on PATH¹. For Unix, the system environment should be whatever python3 executable is on PATH, and if none is available, whatever python executable is present (I'd be willing to prefer python over python3 if people more familiar with Unix thought that was a reasonable thing to do nowadays).

If there is an activated virtual environment, that is what should be used. Detecting an activated virtual environment, given that pip-run could be running in its own, different, environment, could be problematic, though. A practical solution would be to check the VIRTUAL_ENV environment variable, as the standard activation scripts set that, and entry point wrappers, such as the ones that pipx uses, do not.

To make this work, we should always run the installer (whether pip or uv) with the --python option explicitly specifying the interpreter we want to use. We should never expose the installer's default behaviour.

We could consider extensions to this behaviour, such as uv's approach of automatically detecting a .venv directory and using it even when it's not activated. I'm not sure that's necessary for a tool like pip-run, which doesn't modify the environment except temporarily. But if it becomes a common and popular behaviour, we could adopt it at a later date.

Footnotes

1. There's a potential issue with this as Microsoft provide the python wrapper that opens the Windows Store. I'll ignore that for now, in the interests of simplicity. I'd be inclined to ask Steve Dower for advice on dealing with that, if necessary. ↩

[pfmoore]

... and yes, this soulds like it's in contradiction to my comment that you quoted, where I said I'd like to see a standard approach. But it's not really, as while I would like to see a standard, I just don't think it's this project's job to invent one.

[jaraco]

In #51, I'm exploring removing the support for augmenting existing environments. That removal will avoid one possible impediment to this effort.

Another possible impediment is the advertised support for being able to run pip-run in nested installs. If we take pip-run out of the installed packages for the subprocess, that feature will break. I quite like that feature as it provides an easy way to simulate other environments that might have multiple site packages with different versions of the same package present. I use it frequently to troubleshoot such environments (and provide repro steps for related issues).

If we're going to remove the assumption that python is run from the same environment where pip is installed, we may want to consider additionally injecting (probably appending) another PYTHONPATH entry for pathlib.Path(pip_run.__file__).parents[1] (and maybe also the same for PATH and bin/) so that expectation can be retained.

[pfmoore]

OK, if those use cases are important to you, I'll leave it to you to come up with a solution that addresses them. I don't personally think that pip-run should deliberately provide the ability to create broken environments as in your "nested install" case. And the situation in #51 is (as far as I can tell) a consequence of the way pip's --target option works. In my personal view, --target was only ever intended to be used to populate an empty directory with a complete set of requirements - so anything to do with "already installed" packages in the context of --target is explicitly unsupported. There's a pretty extensive discussion of all this, leading to uv's implementation of --target, at astral-sh/uv#1517.

My understanding is that pip-run uses pip install --target to install into a completely empty directory, so its usage is supported by pip. But what pip-run does after that to merge the target directory with an existing environment is a separate matter, and pip (and uv) won't have anything to say on the validity of that. Nor will pip be likely to add any features to support such usage.

If I'm 100% honest, I'd argue that pip-run should switch to using a virtual environment (creating one using the stdlib venv module without installing pip is extremely fast) and deprecate the "layering" behaviour that it currently has. But while that is sufficient my use cases for the tool, which never rely on the layering, I don't know how important the layering is for pip-run's user base as a whole.

[jaraco]

Today, I had another reason to use nested environments (install a broken or fixed build backend, then invoke pip-run to cause pip to install with that backend something but without mutating any environment).