Skip to content

virtualisation/rosetta: Perserve ArgvZero to fix python virtualenv creation#463106

Merged
SuperSandro2000 merged 1 commit intoNixOS:masterfrom
dwt:fix-python-environment-creation
Nov 26, 2025
Merged

virtualisation/rosetta: Perserve ArgvZero to fix python virtualenv creation#463106
SuperSandro2000 merged 1 commit intoNixOS:masterfrom
dwt:fix-python-environment-creation

Conversation

@dwt
Copy link
Contributor

@dwt dwt commented Nov 19, 2025

Preserving ArgvZero is supported from MacOS 14 Sonoma forward, which is the current minimum supported version of nixpkgs. See https://developer.apple.com/documentation/virtualization/running-intel-binaries-in-linux-vms-with-rosetta#Mount-the-Shared-Directory-and-Register-Rosetta

The necessity of this fix came up with python virtual environments, because python uses the path of the executable (argv zero) to determine if it should load a virtual environment or not. In the past this was overriden with NIX_PYTHONEXECUTABLE, NIX_PYTHONPATH and NIX_PYTHONPREFIX, but we stopped doing that for the python interpreter so it works more like a normal python interpreter in nix.

Without those variables, in the rosetta emulated case, python was no longer able to determine where it was called from, and thus could not correctly activate virtual environments in that case.

Investigation and this suggestion for the fix happened here. #461884

This is related to, but not the same issue as was observed here: #461406

This problem was introduced here #442540 where virtual env creation was changed so the interpreter was special cased and no longer sets the above mentioned variables.

I'd like to ping some of the people that helped debug this: @qbisi @9999years @eclairevoyant @winterqt

Some open questions: should this be targeting NixOS:staging instead of master like #462090?

Things done

  • Built on platform:
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • Tested, as applicable:
  • Ran nixpkgs-review on this PR. See nixpkgs-review usage.
  • Tested basic functionality of all binary files, usually in ./result/bin/.
  • Nixpkgs Release Notes
    • Package update: when the change is major or breaking.
  • NixOS Release Notes
    • Module addition: when adding a new NixOS module.
    • Module update: when the change is significant.
  • Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

Add a 👍 reaction to pull requests you find important.

…eation

Preserving ArgvZero is supported from MacOS 14 Sonoma forward, which is
the current minimum supported version of nixpkgs. See
https://developer.apple.com/documentation/virtualization/running-intel-binaries-in-linux-vms-with-rosetta#Mount-the-Shared-Directory-and-Register-Rosetta

The necessity of this fix came up with python virtual environments,
because python uses the path of the executable (argv zero) to determine
if it should load a virtual environment or not. In the past this was
overriden with NIX_PYTHONEXECUTABLE, NIX_PYTHONPATH and
NIX_PYTHONPREFIX, but we stopped doing that for the python interpreter
so it works more like a normal python interpreter in nix.

Without those variables, in the rosetta emulated case, python was no
longer able to determine where it was called from, and thus could not
correctly activate virtual environments in that case.
@winterqt
Copy link
Member

master is fine, thanks!

Copy link
Member

@winterqt winterqt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a ton!

(Nit: I’m not sure if this can be considered virtualenv-specific, as it happens when using python.withPackages too, which I don’t think is a venv. Actual change LGTM, though.)

@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 12.approvals: 1 This PR was reviewed and approved by one person. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Nov 19, 2025
@winterqt
Copy link
Member

Ah, another thing: could you add a reference to the commit and/or PR that broke this to the commit message, please? Just to save future archeologists a headache 😉

@winterqt
Copy link
Member

I’m not sure if this can be considered virtualenv-specific…

Arguably this is fine given the other commits that remove the NIX_* stuff mention venvs.

@dwt
Copy link
Contributor Author

dwt commented Nov 19, 2025

@winterqt I added a link to the pull request that broke things - does that work for you?

Regarding venv: I would argue that python.withPackages actually does create a virtual env under the hood (or I may be missing something).

@dwt
Copy link
Contributor Author

dwt commented Nov 19, 2025

Nix-Review spits this out, not quite sure if this is actually helpful to judge this works:

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 463106
Commit: f5eb10b52cc8f54e1970176e26d2e6627bd240ca


x86_64-linux

⏩ 2 packages blacklisted:
  • nixos-install-tools
  • tests.nixos-functions.nixos-test

aarch64-linux

⏩ 2 packages blacklisted:
  • nixos-install-tools
  • tests.nixos-functions.nixos-test

@dwt
Copy link
Contributor Author

dwt commented Nov 20, 2025

Friendly ping @qbisi @9999years @eclairevoyant - could you perhaps be so kind as to take a look at this pull request? I would really like to get this into 25.11, as that would un-break the review workflow for a lot of people who rely on VMs to check linux x86 binaries from darwin.

@eclairevoyant
Copy link
Contributor

I'm not a committer nor have a darwin machine, don't have any input here.

@dwt
Copy link
Contributor Author

dwt commented Nov 20, 2025

@NixOS/darwin-core - is this something you should perhaps take a look at?

@9999years
Copy link
Contributor

This doesn't fix the issue with Python environments for me, but it doesn't hurt either.

With repro.nix:

let
  pkgs = import ./. { };
in
(pkgs.python3.withPackages (pyPkgs: [
  pyPkgs.pandas
])).env
$ nix develop -f repro.nix
$ python3 -c 'import pandas'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import pandas
ModuleNotFoundError: No module named 'pandas'

@9999years
Copy link
Contributor

9999years commented Nov 20, 2025

nixpkgs-review pr 463106 says "nothing to be built" for me, which is odd.

Edit: I'm on aarch64-darwin, not x86_64-darwin or Rosetta x86_64-darwin, so maybe that makes sense?

@winterqt
Copy link
Member

This doesn't fix the issue with Python environments for me

Expected, this just fixes that issue with Rosetta.

@winterqt
Copy link
Member

*with Linux Rosetta on aarch64-darwin

@winterqt winterqt added this pull request to the merge queue Nov 20, 2025
@winterqt winterqt removed this pull request from the merge queue due to a manual request Nov 20, 2025
@9999years
Copy link
Contributor

Thanks @winterqt, sorry for the confusion!

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/6088

@SuperSandro2000 SuperSandro2000 added the backport release-25.11 Backport PR automatically label Nov 26, 2025
@SuperSandro2000 SuperSandro2000 added this pull request to the merge queue Nov 26, 2025
Merged via the queue into NixOS:master with commit ad8221e Nov 26, 2025
37 checks passed
@nixpkgs-ci
Copy link
Contributor

nixpkgs-ci bot commented Nov 26, 2025

@dwt
Copy link
Contributor Author

dwt commented Nov 28, 2025

@SuperSandro2000 Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 8.has: port to stable This PR already has a backport to the stable release. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 12.approvals: 1 This PR was reviewed and approved by one person. backport release-25.11 Backport PR automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants