Virtual environments created by nox fail to inherit the correct version of Python

[barbaricyawps]

Current Behavior

Hi, I'm from the Salt Project. I've had multiple issues trying to use nox to build Sphinx documentation. Typically, when I run a command like nox -e docs or similar, I encounter an error like this (I'm using a WSL):

Error: python is not installed into the virtualenv, it is located at 
/mnt/c/Users/myusername/myprojectpath/salt-reference-docs/.venv/bin/python. 
Pass external=True into run() to explicitly allow this.

FYI, I'm working in the salt-reference-docs, which is where the project name comes from.

Running python --version shows the system-level Python (Python 10 in my case).

The error seems to be related to how nox interacts with virtual environments such as venv and pyenv. (I have problems with both equally when using nox.) My best guess is that nox activates these virtual environments incorrectly. When these virtual environments activate, they get added to the end of the PATH rather than to the beginning, so it finds the system Python first. That defeats the purpose of using a virtual environment where you can run different versions of Python.

Expected Behavior

The virtual environment used by nox should load the correct version of Python by inheriting it from the virtual environment.

Steps To Reproduce

I've recently experienced this on a new Salt documentation repo we set up using the same setup we use on most of our other projects and repos. To reproduce it:

1. Fork and clone our repo: https://gitlab.com/saltstack/open/docs/salt-reference-docs
2. Follow our contributing guide to set up your environment for that specific repo, specifically the steps listed for "Set up your local preview environment."
3. With the virtual environment (venv) activated, try running: nox -e 'docs-html(clean=False)' or nox -e docs.
4. The build fails because the virtual environment doesn't load Python.

Environment

- OS: Ubuntu 22.04 WSL (on Windows)
- Python: 10
- Nox: 2020.12.31
- Sphinx 4.4.0

Workarounds

One workaround seems to be to manually start the virtual environment activation process so that it loads into the PATH correctly.

Another less-than-ideal workaround seems to be to copy the bin file from an existing virtual environment from another project where it works and put it inside the virtual environment being used by the current project. This is less than ideal because it messes up any custom CSS I've been working with locally.

By the way, I have no idea how to Pass external=True into run() like the error message suggests. Perhaps a better-written, more actionable error message or some more supporting documentation would help. Is that something that needs to be passed in when running the nox command to build the docs or is it something in the nox configuration file?

[FollowTheProcess]

Hey @barbaricyawps thanks for raising an issue! Unfortunately I can't replicate the issue locally even after following the steps to the letter (see below)

I'm not sure what could be causing this to be honest, potentially some sort of quirk with WSL? I'm on a mac using python 3.10.4.

By the way, I have no idea how to Pass external=True into run() like the error message suggests

On this specifically, you want the following:

session.run("your", "command", "args", external=True)

The auto release recipe in the Nox cookbook shows some example usage

[barbaricyawps]

Ah, well. That's good to know. Thanks for trying! I'll close the issue.

[FollowTheProcess]

Ah, well. That's good to know. Thanks for trying! I'll close the issue.

More than happy to keep it open if it's genuinely a bug I'm just not sure as I don't have the setup to replicate locally.

On the external=True thing this is almost certainly not what you actually want to do to get around this, session.install should find the python for the session every time.

Is it anything to do with simply requiring python3?

@nox.session(python="3")
def docs(session) -> None:

Might be worth trying a specific version but again I'm not sure if that will actually help.

[barbaricyawps]

Hello! Just wanted to give an update that I tried setting up a brand new WSL and was able to get nox functioning correctly. I suspect this error message I got while installing nox might have something to do with the issues I was having:

I followed the directions in the error message to run pipx ensurepath and that seems to have worked. I also added this to my ~/.bashrc just in case:

# Adding path to access pipx and misc packages
export PATH="/root/.local/bin:$PATH"

[samueljsb]

I'm running up against a problem that looks very similar to this one: nox is alwasy using the system python rather than the one that should be installed for the session.

I have a minimal example that demonstrates this:

# noxfile.py
import nox


@nox.session(python=['3.8', '3.9', '3.10', '3.11', '3.12'])
def test(session):
    session.run('python', '--version')

$ nox
nox > Running session test-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/test-3-8
nox > python --version
Python 3.11.1
nox > Session test-3.8 was successful.
nox > Running session test-3.9
nox > Creating virtual environment (virtualenv) using python3.9 in .nox/test-3-9
nox > python --version
Python 3.11.1
nox > Session test-3.9 was successful.
nox > Running session test-3.10
nox > Creating virtual environment (virtualenv) using python3.10 in .nox/test-3-10
nox > python --version
Python 3.11.1
nox > Session test-3.10 was successful.
nox > Running session test-3.11
nox > Creating virtual environment (virtualenv) using python3.11 in .nox/test-3-11
nox > python --version
Python 3.11.1
nox > Session test-3.11 was successful.
nox > Running session test-3.12
nox > Missing interpreters will error by default on CI systems.
nox > Session test-3.12 skipped: Python interpreter 3.12 not found.
nox > Ran multiple sessions:
nox > * test-3.8: success
nox > * test-3.9: success
nox > * test-3.10: success
nox > * test-3.11: success
nox > * test-3.12: skipped

Every run there appears to be using Python 3.11.1, even when it should be a different minor version.

nox is correctly determining which versions of Python I have available:

$ python3.8 --version
Python 3.8.16
$ python3.9 --version
Python 3.9.16
$ python3.10 --version
Python 3.10.9
$ python3.11 --version
Python 3.11.1
$ python3.12 --version
zsh: command not found: python3.12

I'm on macOS 13.3.1, I have nox installed with pipx

$ python3 --version
Python 3.11.1
$ nox --version
2023.4.22
$ pipx list
...
   package nox 2023.4.22, installed using Python 3.11.1
    - nox
    - tox-to-nox

EDIT: the context of the .nox directory:

$ exa -TL3 .nox
.nox
├── test-3-8
│  ├── bin
│  │  ├── activate
│  │  ├── activate.csh
│  │  ├── activate.fish
│  │  ├── activate.nu
│  │  ├── activate.ps1
│  │  ├── activate_this.py
│  │  ├── pip
│  │  ├── pip-3.11
│  │  ├── pip3
│  │  ├── pip3.11
│  │  ├── python -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
│  │  ├── python3 -> python
│  │  ├── python3.11 -> python
│  │  ├── wheel
│  │  ├── wheel-3.11
│  │  ├── wheel3
│  │  └── wheel3.11
│  ├── lib
│  │  └── python3.11
│  └── pyvenv.cfg
├── test-3-9
│  ├── bin
│  │  ├── activate
│  │  ├── activate.csh
│  │  ├── activate.fish
│  │  ├── activate.nu
│  │  ├── activate.ps1
│  │  ├── activate_this.py
│  │  ├── pip
│  │  ├── pip-3.11
│  │  ├── pip3
│  │  ├── pip3.11
│  │  ├── python -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
│  │  ├── python3 -> python
│  │  ├── python3.11 -> python
│  │  ├── wheel
│  │  ├── wheel-3.11
│  │  ├── wheel3
│  │  └── wheel3.11
│  ├── lib
│  │  └── python3.11
│  └── pyvenv.cfg
├── test-3-10
│  ├── bin
│  │  ├── activate
│  │  ├── activate.csh
│  │  ├── activate.fish
│  │  ├── activate.nu
│  │  ├── activate.ps1
│  │  ├── activate_this.py
│  │  ├── pip
│  │  ├── pip-3.11
│  │  ├── pip3
│  │  ├── pip3.11
│  │  ├── python -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
│  │  ├── python3 -> python
│  │  ├── python3.11 -> python
│  │  ├── wheel
│  │  ├── wheel-3.11
│  │  ├── wheel3
│  │  └── wheel3.11
│  ├── lib
│  │  └── python3.11
│  └── pyvenv.cfg
└── test-3-11
   ├── bin
   │  ├── activate
   │  ├── activate.csh
   │  ├── activate.fish
   │  ├── activate.nu
   │  ├── activate.ps1
   │  ├── activate_this.py
   │  ├── pip
   │  ├── pip-3.11
   │  ├── pip3
   │  ├── pip3.11
   │  ├── python -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
   │  ├── python3 -> python
   │  ├── python3.11 -> python
   │  ├── wheel
   │  ├── wheel-3.11
   │  ├── wheel3
   │  └── wheel3.11
   ├── lib
   │  └── python3.11
   └── pyvenv.cfg

[samueljsb]

I have tracked down my problem: I have VIRTUALENV_PYTHON set in my environment and that takes precedence over the explicit options to virtualenv (see pypa/virtualenv#2285). My solution here is to run VIRTUALENV_PYTHON= nox instead (or get that issue fixed).

Sorry for the noise!