Wheel installation fails when local path dependency introduced #4868
Comments
stonelazy
added
kind/bug
|
I get this behavior as well - building a wheel with a local path dependency results in a wheel that can't be installed on other systems (even when wheels for that path dependency have been previously built and installed). Stacktrace: |
|
Currently setting up a monorepo with python services/libraries, and need to work around these current limitations. At the end of the day, poetry supports local paths for dependencies so I think there needs to be a sane outcome when running poetry build in this case. IMHO, it should most likely package all the local dependencies up into the built .whl file - that's the simplest solution. If something is local, and we're building a module to be installed somewhere, it makes sense to bundle any non-external dependencies with the build, because there is no other way to get them. I mentioned the same in #1168 |
|
@stonelazy I suspect your issue is related to pypa/pip#6658 (comment). For Also even in a case where the above is fixed, your code will need the following snippet in order for the wheel to include the package as a vendored dependency. include = [
{ path = "appname-pkg3", format = ["sdist", "wheel"] }
]This will ensure the source is vendored into your wheel when built. @michaelcredera for your issue, the relative paths are problematic. When wheels are being built under PEP-517, the source is expected to be moved into an isolated environment at build time. This means your relative paths will need to hold even when your project source is moved to an ephemeral build directory. Unlike the OP's case yours is trickier. The general question of "what should this be relative to?" is still an open question within the packaging group. See here for more on this. The alternative approach for this case is to build |
At the risk of sounding ignorant, I'm speaking from a user/"peasant" perspective of someone trying to make things work with the available tools. It just doesn't seem right to leave it up to pip to build the relative path dependencies - it would be a wrong level of abstraction no? When IMHO, it should be a special case, or at least there should be a commandline switch or configuration to do this with poetry all during the build step itself, and not unnecessarily postpone half of the process to a separate tool (i.e. pip). It just overcomplicates something that users intuitively expect. |
|
I wouldn't want my pathed dependencies to be vendored in, personally. I'd rather pathed dependencies be turned into package-dependencies, perhaps with a version-equality exactly matching the version-at-time-of-build of the pathed dependencies. My approach would be to So similar to what #4868 (comment) expected, that pre-installing the resulting depended-upon wheel would work correctly. |
|
Understood - I can agree with not vendoring in by default. However,
This is quite cumbersome and non-intuitive from a usability perspective. It means having a pathed dependency requires a host of conventions and scripts that come with it to make it work. I.e. if I have
Otherwise IMO the path dependency is "incomplete" as a feature because it has all these caveats attached to it - it's not useable. At the very least these should be expressed in the documentation to make sure users know what to expect. Hopefully it would be possible to do this without any of the extra steps. |
|
I was actually thinking that I agree that it's not a workflow for every occasion: It works well for "I run The flow I want to use also works for "I build each package and push it to our PyPI repo in parallel", particularly if there are consumers of the libraries (not the root packages) who're not in the same monorepo, and hence need a normally exported package anyway. This flow means that both the monorepo output and the external users are using the same wheels. That said, if vendoring into a single wheel is implemented and reliable, I'd probably use it in preference to the current approach, which is exposing all the source into the containers and I'll also note that one of my considered approaches was to only use dev-dependencies in pathed dependencies, and then use normal versioned package dependencies in the non-dev dependencies. In this case, I'd want In this flow, it's required (and expected) that people are maintaining the versioned-dependency (dot point 2), and intentionally so, as that version is an ABI dependency, and should be being bumped when ABIs are changed/added/removed, so you can't successfully install mismatching packages. However, Poetry has some other limitations that currently prevent this, i.e. the resolver merges dependencies and dev-dependencies too early to make this work, if I recall correctly. (There's a ticket elsewhere here tracking this use-case, so it's not specifically-relevant here, I just wanted to share some context on my approach here.) |
@abn Pip seems to still have trouble with relative paths - for me this resulted in the same "Invalid URL" error: Process:
Edit: Maybe I'm misunderstanding your suggestion. Perhaps you mean to change And having every module reference other modules via their wheelhouse? As TBBle pointed out, this seems like it would require a decent amount of automation / scripting to manually manage dependency build order, wheel staging, and then installation order in a remote machine (as opposed to a simple That being said, I probably just don't understand what you're suggesting :) |
|
@michaelcredera what I am suggesting is closer to what @TBBle is already doing I reckon. Here is an example project. poetry new foo
cd foo
poetry new bar
poetry add -D ./bar
poetry add bar@*And then build it as show in this FROM docker.io/python:3.10 AS build
COPY . /build
WORKDIR /build
RUN pip install --quiet build
RUN mkdir wheelhouse
RUN python -m build --wheel --outdir ./wheelhouse ./bar
RUN python -m build --wheel --outdir ./wheelhouse .
FROM docker.io/python:3.10-slim AS runtime
COPY --from=build /build/wheelhouse /wheelhouse
RUN python -m venv .venv
RUN python -m pip install --find-links /wheelhouse foo
ENTRYPOINT ["/.venv/bin/python"]The "wheelhouse" is simply a directory with wheels for both This is more a workaround for now. Issues #936 and #2270 might be of interest. Additionally, the include = [
{ path = "bar", format = "sdist" }
] |
|
The root cause of the issue has been addressed in #4868 (comment). As there is no standard mechanism in wheels to handle relative path dependencies, and as any resolution in pip and/or wheel spec should address the issue, I am closing this. Regarding the broader use cases for monorepos/subprojects, see #936 and #2270. Regarding local path dependency support, see #1168. If you would like help regarding your project and/or guidence around workarounds, I suggest a discussion be started. |
|
So keep having issues with the wheel dependency for pysha3 Building wheel for pysha3 (setup.py) ... error × python setup.py bdist_wheel did not run successfully. note: This error originates from a subprocess, and is likely not a problem with pip. × python setup.py bdist_wheel did not run successfully. note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pysha3 I have used $pip install " " --no-cache-dir. it works sometimes. But I would like to solve the issue. The old python was made obsolete and the new has no id |
|
@Jack1082013 Your problem does not appear to have any connection to this issue, or in fact any connection with Poetry at all. I guess that you've hit Tierion/pymerkletools#20 and should use the approach described there, and follow up with zeronet (the thing you appear to be trying to install, that is pulling in pymerketools, that is pulling in pysha3) if you still can't get it to work. |
|
The issue has started coming up when I try to install anything on my
terminal it's not just with zero net but I will look into your suggested
fix and get back to you with a reply
…On Thu, Oct 19, 2023, 5:04 PM TBBle ***@***.***> wrote:
@Jack1082013 <https://github.com/Jack1082013> Your problem does not
appear to have any connection to this issue, or in fact any connection with
Poetry at all.
I guess that you've hit Tierion/pymerkletools#20
<Tierion/pymerkletools#20> and should use the
approach described there, and follow up with zeronet (the thing you appear
to be trying to install, that is pulling in pymerketools, that is pulling
in pysha3) if you still can't get it to work.
—
Reply to this email directly, view it on GitHub
<#4868 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BCTBUN5XVYK2EFLPYXFA3ITYAGWWHAVCNFSM5JQQNIG2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXGE4DCOJQGA4Q>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).OS version and name: MAC OS BigSur
Poetry version: Poetry version 1.1.12
Reproducible code: https://github.com/stonelazy/poetry-path-issue
Issue
.whl file generated using poetry build throws error when trying to install.
Steps to reproduce this issue
Expected result
- whl gets installed without any error.
Issue faced
- Exception trace thrown when invoking pip install of the .whl file
Issue Description
- This issue occurs only when a another local path dependency is kept. Suppose the line L13 is removed, then there is no issue.
- Including of this local path is correct according to https://python-poetry.org/docs/dependency-specification/#path-dependencies
Expected Result:
Why should this be a poetry issue and not something wrong with the code ?
The text was updated successfully, but these errors were encountered: