Support publishing git dependencies to PyPI using vendoring

[makew0rld]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

I noticed that when trying to upload a package with a Git dep to PyPI, I get an error:

HTTP Error 400: Invalid value for requires_dist. Error: Can't have direct dependency: 'example @ git+https://github.com/example/[email protected]'

My research has told me that this is because PyPI does not support non-package dependencies.

I was thinking, could Poetry support publishing packages with Git deps by copying the the contents of the git repository at the provided ref into the source distribution, and rewriting imports if needed to make sure the package was imported from a local directory instead? Then the wheel could be built from that.

This way the Git dep would be bundled with the package's code, and PyPI wouldn't complain. I think this could be a great feature that would set Poetry apart.

[marcosfelt]

Can confirm that it would be great to at least have some documentation on recommendations around this problem.

[abn]

My research has told me that this is because PyPI does not support non-package dependencies.

This is correct. The issue is a policy decision taken by PyPI. This is allowed since the PEPs make an explicit exception for public indices to disallow direct references in wheels.

I was thinking, could Poetry support publishing packages with Git deps by copying the the contents of the git repository at the provided ref into the source distribution, and rewriting imports if needed to make sure the package was imported from a local directory instead? Then the wheel could be built from that.

Unfortunately, this is not trivial. You'll have to either build and vendor the library correctly; or embed the built wheel of the dependency into the project wheel. And then there is the matter extracting the right dependencies for the environment in which the wheel is built etc and list them as first level dependencies.

Not to mention, manipulation of import paths can cause some unexpected side-effects.

This way the Git dep would be bundled with the package's code, and PyPI wouldn't complain. I think this could be a great feature that would set Poetry apart.

Personally, it would be great to have a vendor = true option for dependencies. But side-effects and complexity, makes me think twice.

[makew0rld]

I think having vendor = true could be a good solution, it'd be good to be able to do this per-dep.

vendor the library correctly

Any idea what that would look like?

Thanks for taking the time to respond.

[abn]

Any idea what that would look like?

There will be a few things to be considered when providing a solution.

First of which would be to identify if this is infact a "core" requirement or a plugin wrapping around something like https://github.com/pradyunsg/vendoring.

Either way, we will have to determine how vendoring happens. Pre-pending site directories? Or embedding in-package eg: foobar.vendors.x?. The latter would mean the code itself will need to explicitly call the embedded package - which is safer. However, this means, you cannot simply swap the vendored library for a non vendored one without having to update the imports (note vendoring already can handle this, and is already the default - the reverse is not pssible afaict). I am more inclined to be on the safe side with these things, because - you vendoring a package should not break some other package as a rule, exception being you are building an application (eg: poetry-core does this, and so does poetry's get-poetry.py installer - latter being a consequence of gaps in python's application distribtution ecosystem).

Then there is evaluating the consequence of doing this for package resolution. Does ithis mean that, if your package depends on A and B (vendored, lets call it B'), and A depends on B (non vendored) the depenency resolver treats B and B' as separate packages? or the same but different versions? Both have consequences that can be problematic.

These are the first things that come to my mind.

To be clear, I am not proposing we do this. Just discussing the possibilities. I am always on the fence whether vendoring is a good thing or a bad thing and if poetry should promote it or discourage it. :)

EDIT: Oh, also there is the conversation around how does a PEP 517 build handle this? Or is the scope of this limited to wheels built by poetry itself and not using poetry-core build backend?

[tigerhawkvok]

Consider this Enterprise use case:

We have several internal packages that reference one another, and now we want to upload a big orchestrating package to a cloud platform to run this. The cloud platform has no access to our internal artifactory, and only gets the exported wheel from us.

Vendoring contents tidily solves this issue.

The next-best thing I can see is using the packages directive to point to project/project in a git subtree, but that is kludgier and relies on split-up tooling (you have to keep track of your subtree dependency version in git instead of alongside the rest of your dependencies in pyproject.toml)

[neersighted]

Closing as a specialized variant of #4583 as submodules + the feature proposed there make more sense. Poetry's use of VCS/Git dependencies is of the PEP 508/standard-to-the-ecosystem style. If you want to include arbitrary code vendored in to your install/built artifacts and control the paths, that issue tracks it.

I don't think we'll ever overload the way Poetry defines Git dependencies to allow for the implicit vendoring requested in this issue -- instead, submodules and #4583 are the best way to achieve this, in a way that is more explicit and flexible.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.