Python environment "layering" Rye vs Uv

[bluss]

Here we are comparing rye run and uv run, how they behave w.r.t the python virtual environment they are started from.

Steps to reproduce: https://github.com/bluss/uv-issue-5459

We have a process tree like this

• rye run foo     This starts some bigger application foo
  • rye run bar     Somewhere inside foo another application bar is started.

Foo and bar live in different projects or in different packages. And we can do the same thing using uv run instead of rye run:

• uv run foo
  • uv run bar

.

• What happens in Rye is that it doesn't matter what virtualenv is active and so on, foo is executed in its project context with its own virtualenv. bar is executed the same way, with its own virtualenv.

• What happens in Uv is that bar is executed in a python environment that "layers" on top of foo. So what the python program in bar sees is that it can use a mix of packages from both foo and bar.

There's a different case for uv run --with, then we have:

• base project environment .venv
  • uv run --with's ephemeral environment

In this case the two environments also layer each other. I thought that was kind of cool, in this case. But not in the original case.

I think this strict project focus and separation is an important selling point of rye. It makes more robust applications possible.

I don't know why this happens, but one difference I can notice when comparing rye run python and uv run python in the same project, is that only uv sets PYTHONPATH.

[charliermarsh]

What happens in Uv is that bar is executed in a python environment that "layers" on top of foo. So what the python program in bar sees is that it can use a mix of packages from both foo and bar.

Hmm, I don't think it's intentional that uv run bar (is that being run from within bar?) is layered on top of foo. We should only be layering when --with is provided. Can you add an MRE?

[bluss]

Sure, came up with this repository which I hope shows what happens https://github.com/bluss/uv-issue-5459

As one can guess, my real life use case for this is that foo=jupyterlab and bar=notebook kernel, which live in separate projects. But you could imagine that you would run an editor (spyder?) as the "foo" and any other developer tool as "bar".

[bluss]

I would consider dropping setting PYTHONPATH entirely in uv run - (if this should be fixed! Don't know that..)

Nesting can possibly be done right using a path file when using ephemeral environments? I don't have experience with that, but it sounds right - I don't know what's been attempted already(!)

[charliermarsh]

Thank you! What do you see as the issue with modifying PYTHONPATH?

[charliermarsh]

This is a very good example, I think I consider it a bug.

[bluss]

PYTHONPATH seems to be the factor that causes this.

PYTHONPATH shouldn't need to be set for using a virtual environment in the normal way. I can see why the environment layering (base project + ephemeral) would reach for using it, however, but maybe that can be avoided too. (I'm sure someone somewhere in history has lost a lot of time to virtual environment chaining and I haven't and don't know what works well..)

One can compare uv vs rye sys.path outcomes and notice that PYTHONPATH usage also moves the venv site-packages to be earlier than stdlib for uv, while it is not for rye. Does this mean that execution of a program - if it depends on shadowing one way or the other - could be subtly different for uv run thescript and .venv/bin/thescript?

[charliermarsh]

Yeah, but we can fix the proximate issue by not "propagating" PYTHONPATH through nested calls:

diff --git a/crates/uv/src/commands/project/run.rs b/crates/uv/src/commands/project/run.rs
index 34348a8f0..97f8153d1 100644
--- a/crates/uv/src/commands/project/run.rs
+++ b/crates/uv/src/commands/project/run.rs
@@ -452,15 +452,11 @@ pub(crate) async fn run(
                     .flatten(),
             )
             .map(PathBuf::from)
-            .chain(
-                std::env::var_os("PYTHONPATH")
-                    .as_ref()
-                    .iter()
-                    .flat_map(std::env::split_paths),
-            ),
     )?;
     process.env("PYTHONPATH", new_python_path);

There may be other issues with it though. Gonna ask some more knowledge people.

[bluss]

It would be cool to try to drop PYTHONPATH entirely and try the path configuration file route. Do you know why pythonpath is used at all? Is it in case the user has existing settings for PYTHONPATH?

[charliermarsh]

I think I learned about it from pip-run and Jason knows a lot about this kind of manipulation.

[charliermarsh]

Maybe a .pth file would work, though.

[charliermarsh]

I talked to @carljm and he walked me through how we might do it with sitecustomize.py, so I'll give that a shot.

[carljm]

To be clear, the reason for sitecustomize.py over a pth file is to get processing of pth files in the added directory, so editable installs in the base environment are still importable.