Cannot solve cross platform with a pypi dependency

[maartenbreddels]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

[project]
name = "pixi-test"
version = "0.1.0"
description = "Add a short description here"
authors = ["Maarten A. Breddels <[email protected]>"]
channels = ["conda-forge", "https://repo.mamba.pm/emscripten-forge"]
platforms = ["emscripten-wasm32", "osx-arm64"]

[tasks]

[target.emscripten-wasm32.pypi-dependencies]
solara = "*"

[target.osx-arm64.dependencies]
python = ">=3.11.3,<3.13"

[target.emscripten-wasm32.dependencies]
python = ">=3.11.3,<3.12"
scipy = ">=1.11.1,<1.12"

Issue description

I tried to solve an emscripten environment on osx-arm64.
When solving that, pixi tells me to get python installed (curious why).
But after I do that, I get:

$ pixi install
ERROR rattler_repodata_gateway::fetch: error=repodata not found
  × failed to solve the pypi requirements of 'default' 'emscripten-wasm32'
  ╰─▶ could not determine python environment markers for emscripten-wasm32

(I think that ERROR message is unrelated btw)

Expected behavior

I expect it to solve the environment for me :)

[ruben-arts]

The reason it gets python is to solve the pypi-dependencies as it needs to a python interpreter to solve (build sdists for their metadata).

I'm not that familiar with wasm so correct me if I'm wrong but you should be able to install wasm package on any platform right?

[maartenbreddels]

The reason it gets python is to solve the pypi-dependencies as it needs to a python interpreter to solve (build sdists for their metadata).

Could you not do without, by downloading the wheel(s)? I regularly pip install, and about 99% of it is a wheel. If a wheel is not available, the install usually fails (e.g. on osx arm with a old version of python for which a library maintainer is not publishing wheels anymore but forgot the limit the python version - so often a mistake).

To summarize, I think pixi can do pypi installs without using/installing python, which would probably be much faster. If you bail out because you need to build the sdist and then give this error message, that would be much better sane behavior I think.

I'm not that familiar with wasm so correct me if I'm wrong but you should be able to install wasm package on any platform right?

Not sure what you mean here?

[ruben-arts]

Could you not do without, by downloading the wheel(s)?

The initial version of pypi-dependencies in pixi was wheel only. Which is much faster and simpler but it is not sufficient because of that 99% you mention. There might be more sane solutions to speed up this installation process but it probably requires more custom options. @tdejager could probably elaborate more.

Not sure what you mean here?

Let me ask @wolfv offline, I need to get more up to speed with the wasm work.

[maartenbreddels]

But in the case of wheel only, is it correct to assume you can do without python? So you would only need python installed if there is no compatible wheel available?

[ruben-arts]

Yes, that is correct. The python interpreter is needed for the sdist building.

[tdejager]

If you want, we could add the option to constrain to using only wheels for the resolution. uv allows this, we would need to figure out the level to set this, e.g. it could be conflicting over solve-groups/environments, and we need to give it a try with the uv API.
Also we would need to account for it in the pixi code, having different code paths for these options. We would not need to install the conda prefix at all actually before the pypi resolution.
But it seems a common enough use-case to be interesting perhaps.

Anyone who reads this, feel free to make an issue.

To summarize, I think pixi can do pypi installs without using/installing python, which would probably be much faster. If you bail out because you need to build the sdist and then give this error message, that would be much better sane behavior I think.

@maartenbreddels I feel a bit hesitant on adding the behavior you described, because I think it's quite dependent on the use-case, and hence the libraries you work with. A lot of python users will encounter sdists almost instantly for any decently sized project they are using. I feel it would be better, to be able to opt-in to the behavior you want, that way you could even create wheel-only environments e.g.

If a wheel is not available, the install usually fails (e.g. on osx arm with a old version of python for which a library maintainer is not publishing wheels anymore but forgot the limit the python version - so often a mistake).

I've also seen examples where people still use old packaging standards, resulting in sdists. Or researchers that are not developers that follow any old python tutorial for publishing. Building no-arch packages with simple setup.py's seem well enough.

[maartenbreddels]

An opt-in sounds reasonable.

Yeah, I thought the pypi json API would provide dependencies by now, but it doesn't. And indeed, for sdist, you can have dependencies depend on the moon phase for all we know.

There should be a large database that stores/caches all the dependencies... 👀

Anyway, thanks for the explaination.

[jayqi]

I'm encountering this limitation as well. My use case is that I'm planning to run a Pixi project inside a Linux Docker container but I'd like to be able to be able to run locking in my local development environment, e.g., on macOS.

I understand that, because Python packages can in general run arbitrary code in setup.py and do all sorts of crazy things, the only way to guarantee a correct lockfile is to install a Python interpreter with the right version and on the right platform.

However, my expectation (I may be wrong) is that most of the time, packages are not doing something crazy and pathological and so relaxing this aspect would still lead to correct lockfiles in practice. uv will automatically try to find a Python interpreter when running uv pip compile and documents the caveat that the lockfile might not be portable. Is that something that Pixi could do? Issuing a warning or opt-in or whatever feels like it would set expectations reasonably.