pythonTerminalEnvVarActivation shadows pyenv PATH

[kbehlers]

Behaviour

When the pythonTerminalEnvVarActivation experiment is active, I can no longer use pyenv-virtualenv commands for activating and deactivating venvs in the VSCode terminal. Instead the interpreter selected in the Python extension always stays activated in the terminal.

The pyenv-virtualenv commands give the appearance of working (they change my zsh command prompt showing the active venv) but running python commands like python --version or pip freeze show that the python interpreter being used isn't the one shown in my command prompt but instead the one selected using the Python extension in VSCode.

It seems possibly related to how pythonTerminalEnvVarActivation prepends to PATH.

Steps to reproduce:

1. Make sure settings.json has

"python.experiments.optInto": ["pythonTerminalEnvVarActivation"]

2. Install pyenv and pyenv-virtualenv
3. Install two versions of python with pyenv (e.g. pyenv install 3.10 pyenv install 3.9) to make it easier to see when the wrong venv is active
4. Create a virtualenv for each version of python

pyenv virtualenv 3.9 first_example_env
pyenv virtualenv 3.10 second_example_env

5. Choose first_example_env as the python interpreter in VSCode using Python: Select Interpreter
6. Open a VSCode terminal
7. echo $PATH (this shows vscode specific paths when the experiment is enabled)
8. python --version (should say Python 3.9.*)
9. pyenv deactivate (doesn't end up deactivating as it should but gives the appearance of doing so by changing the command prompt)
10. pyenv activate second_example_env (changes the command prompt, doesn't actually change the venv though)
11. python --version (should say Python 3.10.* but still says Python 3.9.* because the environment didn't really switch)
12. Use Python: Select Interpreter and choose second_example_env
13. Reload the VSCode terminal
14. Repeat above steps, except this time second_example_env will be active no matter what

If I instead add:

"python.experiments.optOutFrom": ["pythonTerminalEnvVarActivation"]

to settings.json, I can activate and deactivate venvs using pyenv-virtualenv just as I would expect, where the venv actually switches in the terminal.

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2024-04-11 11:24:39.228 [info] Prepending environment variable PATH in collection with /Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin: {"applyAtShellIntegration":true,"applyAtProcessCreation":true}
2024-04-11 11:24:39.228 [info] Setting environment variable VIRTUAL_ENV in collection to /Users/username/.pyenv/versions/3.9.17/envs/first_example_env {"applyAtShellIntegration":true,"applyAtProcessCreation":true}
2024-04-11 11:24:39.228 [info] Prepending environment variable PS1 in collection with (first_example_env)  {"applyAtShellIntegration":true,"applyAtProcessCreation":false}
2024-04-11 11:24:39.228 [error] Failed to initialize deactivate script /bin/zsh [Error: "/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh/envVars.txt" file not created
	at Timeout.<anonymous> (/Users/username/.vscode/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:271175)
	at listOnTimeout (node:internal/timers:569:17)
	at process.processTimers (node:internal/timers:512:7)]

[anthonykim1]

hello @kbehlers Thanks for filing the issue. Is this only happening with pyenv?? Is it fine with .venv

[kbehlers]

@anthonykim1 Hi! This is only with pyenv, specifically pyenv-virtualenv.

Standard .venv doesn't have the issue, and looking into it that is probably because the source .venv/bin/activate command prepends the venv to PATH itself..

pyenv-virtualenv seems to be relying on the existing shim already in the PATH, rather than prepending the venv each time, so the shadowing occurs because the VS Code experiment seems to prepend the venv from the interpreter selected in the Python extension.

Exploring the behavior a bit more, if I first source deactivate instead of pyenv deactivate then that clears the venv from the interpreter selected in the Python extension from PATH and I can use pyenv-virtualenv normally again.

[anthonykim1]

Hello @kbehlers Thanks much for further looking into this.

I think it can be one or either of two problems:

1. I'm also thinking about the shadowing scenarios of prepending possbily causing shadowing of pyenv adding to existing shim already in PATH. I believe we prepend to PATH all the time. When you take a look at your $PATH after you try to activate pyenv and see pyenv venv prompt, does your PATH contain pyenv related code?

2. My second guess is the way we "deactivate" for pyenv. I will need to further look into deactivate scenario for pyenv. (Im going to try to figure out if python extension does sort of different deactivation for pyenv compared to traditional .venv and if that is correctly working.

Good to hear that there is at least some workaround if you source deactivate first.

I'll let you know if I get update on this.

[anthonykim1]

Related: #21906 Seems to be the PR where we started to use prepend to PATH. With its purpose of attempting to avoid replacing. since pyenv asks their user to manipulate PATH in init script.

[anthonykim1]

@kbehlers can you share what your PATH looks like?
We are trying to figure out which deactivation script is running when yousource deactivate vs. pyenv deactivate

[kbehlers]

@anthonykim1 Sure!

Opted out from the experiment

/opt/homebrew/Cellar/pyenv-virtualenv/1.2.1/shims:/Users/username/.pyenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Opted into the experiment

/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin:/opt/homebrew/Cellar/pyenv-virtualenv/1.2.1/shims:/Users/username/.pyenv/shims:/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin

You can see when opted-in it has some interesting prepend and append behavior

[anthonykim1]

Thanks much for providing PATH. Greatly appreciated!
The prepending is expected as: #21906 and #22905

Im not able to repro the behavior on my machine exactly but I do get "pyenv-virtualenv: no virtualenv has been activated." when I try pyenv deactivate when opting into the experiment. But then that still maintains Python from Pyenv virtualenv. Only when I source deactivate, it seems to bring back to "correct" Python.

Still lost on why prepending would "not allow" pyenv deactivate to work because your pyenv shim is still technically in your PATH even after Python extension prepend our deactivate script in the PATH.

[kbehlers]

@anthonykim1 Ahh, I see so if I am following #21906 and #22905 to it sounds like

PATH=<activated_full_path><activated_full_path><original_path>

might be expected then but I do want to highlight my path is actually ending up as

PATH=<activated_full_path><activated_full_path><original_path><activated_full_path>

with the two prepended activated paths and a third appended activated_path. Probably not relevant to the problem, still a little odd to me though.

---

Going back to why prepending would "not allow" pyenv deactivate to work, I've been digging into this a bit more.

Built-in venv deactivate manipulates the PATH.

pyenv-virtualenv deactivate manipulates some env vars like VIRTUAL_ENV and PYENV_VIRTUAL_ENV but importantly it does not manipulate path. You can verify this by echo $PATH before and after a pyenv activate first_example_env and a pyenv deactivate.

The way pyenv uses its shims, it expects to do all python routing internally. As with pyenv-virtualenv, pyenv also does not manipulate PATH, instead as part of the installation instructions you modify .zshrc to add the shim to PATH upfront.

It was really helpful for me to understand the behavior by repeating my Steps to Reproduce, but checking PATH, PYENV_VERSION, VIRTUAL_ENV, and PYENV_VIRTUAL_ENV after each step.

As I understand it this means by having the venv force prepended to the PATH, pyenv deactivate will never be able to remove that, only a source deactivate will. Every time a python command comes in it finds the matching executable in the venv that was force prepended and never gets to the pyenv-vritualenv or pyenv shim so they can do an internal search against their activated and/or known python versions.

Out of the two paths appended

/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh
:
/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin

it isn't the /Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh that is the problem, it is the /Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin, and that can be proven by removing only /Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin from the path while keeping /Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh in place.

Hopefully that is helpful in trying to decide how to adapt the experiment for compatibility with pyenv.

[dfuchs-dev]

I'm experiencing a related issue. The appended paths that @kbehlers pointed out are causing problems with conda environments too. Here's what I observed after selecting my_conda_env via Python: Select Interpreter:

1. While opted out of pythonTerminalEnvVarActivation, open a new terminal session, and echo the PATH:

$ echo $PATH
/opt/anaconda3/envs/my_conda_env/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/condabin

2. Now conda deactivate the session. The updated PATH does not contain any references to my_conda_env, and python is inaccessible, which is expected:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/condabin

$ which python
python not found

3. Opt back into pythonTerminalEnvVarActivation, open a new terminal session, and echo the PATH. Note the extra my_conda_env second from last:

$ echo $PATH
/opt/anaconda3/envs/my_conda_env/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/envs/my_conda_env/bin:/opt/anaconda3/condabin

4. Now conda deactivate the session. The updated PATH still contains my_conda_env and python is accessible when it should not be:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/envs/my_conda_env/bin:/opt/anaconda3/condabin

$ which python
/opt/anaconda3/envs/my_conda_env/bin/python

---

As a slight aside, even when opted out, the PATH in VS Code is still a bit different from what I get in a native terminal on my Mac.

• VS Code, conda deactivated: /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/condabin
• Native, conda deactivated: /opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Since it's just condabin being reordered, I think it's fine, though a bit odd.

[chrisrosner]

I might be experiencing problems related to this as well. I'm on osx with pyenv + pyenv virtualenv. pyenv/venv works fine in the OS terminal app. In vscode terminal, everything works fine if I disable vscode terminal integrations. If I source ~/.zshrc or ~/.zprofile in the vscode terminal, pyenv starts working again.

The problem I'm seeing is if I open a new terminal with default vscode settings, the wrong pyenv is entered and changing directories does not switch to the correct one.

which pip shows the wrong pyenv pip
pip list shows the wrong list of installed packages

When things are not working, 'which pip' shows:
/Users//.pyenv/versions/3.12.4/envs//bin/pip

When things are working, 'which pip' shows:
/Users//.pyenv/shims/pip

[anthonykim1]

If I source ~/.zshrc or ~/.zprofile in the vscode terminal, pyenv starts working again.

Correct, this is coming from lack of context in env variables. We are working API microsoft/vscode#227467 so extension has better context of env vars inside .zshrc and other shell init files.

[anthonykim1]

Closing this against #24567
We are planning to get rid of this experiment for better experience with environment variables in terminal for Python.