Environment markers with different sources for a dependency

[joshuataylor]

• Poetry version: 1.4.0.dev0, commit ef89e90fc1305e07a62e9e715c91a66c8d7425f7, also happens on 1.3.1

• Python version: 3.10.9

• OS version and name: macOS 12.6 (21G115)

• pyproject.toml: https://github.com/kilnbuild/pythonbuild_examples/blob/main/poetry/pyproject.toml

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I'm trying to use two different sources in Poetry for a dependency using markers:

1. Apple Silicon markers="sys_platform == 'darwin' and platform_machine == 'arm64'"
2. Fallback (markers="sys_platform != 'darwin' and platform_machine != 'arm64'")

This is because I'm hosting wheels on a source for Apple Silicon, and want to fallback to PyPI for everything else. grpcio for example does not provide wheels for Apple Silicon, so I'm hosting them on a soon-to-be public source (though you can use it now if you want :-)).

If I have a custom mirror for a marker, let's call it "pythonbuild", then can I fall back to PyPI for the other marker, is this supported in Poetry?

Example:

grpcio = [
   {version="==1.47.2", markers="sys_platform != 'darwin' and platform_machine != 'arm64'", source="pypi"},
   {version="==1.47.2", markers="sys_platform == 'darwin' and platform_machine == 'arm64'", source="pythonbuild"},
]

Doing a poetry update -vvv on MacOS ARM64 (using Poetry from master, as of 2022-01-05, commit ef89e90fc1305e07a62e9e715c91a66c8d7425f7), I get the following output (Gist).

Doing this builds the following lock file: lockfile.

During the update process this works, as it looks at the markers and uses the correct source for the dependency, but when I try to install the lock file, it installs from pypi, as that's listed higher in the lock file. That's how it should work in theory, but there should be a way to tell the lockfile which source to use for a dependency.

You can see an example project here that you can use: https://github.com/kilnbuild/pythonbuild_examples/tree/main/poetry (use poetry install on MacOS ARM64 (Apple Silicon, M1/M2 whatever buzzword Apple is using now) to see the issue).

I've read through the following issues:

#5205
#6710

And this PR:
#6679

And my source looks like this:

[[tool.poetry.source]]
name = "pythonbuild"
url = "https://python.build/simple/"
# tried with true/false for secondary, same result.
default = false
secondary = false

Before each step, I ensure that I'm starting with a clean environment:

rm -rf ~/Library/Caches/pypoetry .venv poetry.lock

I can verify the wheel is being used when using poetry update:

cat .venv/lib/python3.10/site-packages/grpcio-1.47.2.dist-info/WHEEL

Shows

Wheel-Version: 1.0
Generator: bdist_wheel (0.38.4)
Root-Is-Purelib: false
Tag: cp310-cp310-macosx_11_0_arm64

So I should be able to run poetry install now, and have the same results, using the lock file..

rm -rf ~/Library/Caches/pypoetry .venv

I can see via network monitoring, it hits https://pypi.org/pypi/grpcio/1.47.2/json for both the update + install, but during the update it downloads from https://wheels.python.build/grpcio-1.47.2-cp310-cp310-macosx_11_0_arm64.whl, where as during the install it does not.

[neersighted]

PTAL @radoering

[dimbleby]

update and install aren't very different, it doesn't make a lot of sense that they would install from different places

during the update it downloads ... where as during the install it does not.

Are you sure you're not just seeing poetry using the cached wheel - as obtained during update - when it does the install?

You've made it rather hard to tell what's going on by making poetry choose betwen identical wheels in different places - I can't see a reason that anyone would do that in real life.

If you can construct a more plausible scenario in which the two cases give actually different results, then probably it would be clearer whether this is behaving correctly or not

[joshuataylor]

I'm removing the cache between runs. So if I was getting a cached wheel, wouldn't it be from the update?

It's not identical wheels. It's a wheel built for MacOS ARM64 on one source, and falling back to PyPI.

That's what this does:

grpcio = [
   {version="==1.47.2", markers="sys_platform != 'darwin' and platform_machine != 'arm64'", source="pypi"},
   {version="==1.47.2", markers="sys_platform == 'darwin' and platform_machine == 'arm64'", source="pythonbuild"},
]

If platform is darwin and machine is arm64, use source pythonbuild.
If platform is not darwin and platform is not arm64, use source pypi.

The reason this is useful is because compiling Python packages on MacOS is a painful experience, especially when they require different packages, or you have multiple projects and need to compile each time.

[radoering]

Maybe, I missed something, but I didn't read what happened during poetry install. IIUC, poetry update installs the expected wheel (verified by the content of the dist-info/WHEEL file). That specific wheel is only available on pythonbuild but not on pypi, isn't it? What happens when running poetry install (after removing the virtual environment)? Is another wheel installed? What's the content of the dist-info/WHEEL file? Or does the command fail?

By the way, "sys_platform != 'darwin' and platform_machine != 'arm64'" is not the inverse of "sys_platform == 'darwin' and platform_machine == 'arm64'". It should probably be "sys_platform != 'darwin' or platform_machine != 'arm64'". However, I don't expect this to be related to the issue.

[dimbleby]

was always hard to tell what was being reported here: python.build is now dead and so I suppose we'll likely never get anywhere with this.

recommend closing.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.