How to create a whl which includes another b.whl, and Then, when installing a.whl, also install b.whl automatically(b.whl is not released to pypi))

[ydh10002023]

setuptools version

any

Python version

python 3.9

OS

macOS

Additional environment information

NA

Description

I want to push a whl package, and it includes another whl file, for example, b.whl is packaged in to a.whl. when install the a.whl, Can b.whl be installed automatically too?

I don't want release another b.whl to pypi.

for example, two whl files(a.whl and b.whl), b.whl in the package_data of a, but when installing a.whl, the b.whl is only put the installation path, how to trigger it's installation automatically?

thanks

Expected behavior

Can work

How to Reproduce

I want to push a whl package, and it includes another whl file, for example, b.whl is packaged in to a.whl. when install the a.whl, Can b.whl be installed automatically too?

I don't want release another b.whl to pypi.

for example, two whl files(a.whl and b.whl), b.whl in the package_data of a, but when installing a.whl, the b.whl is only put the installation path, how to trigger it's installation automatically?

thanks

Output

NA

[abravalheri]

Hi @yudonghai.

Currently, setuptools is not involved in the process of installation of a Python package. This is out of the scope of the project.

You can use setuptools to create a wheel.

If you really want, you can use setuptools to create a wheel that contains a wheel (which would be very weird, but still possible...).

But once your installer (e.g. pip) starts to install that file, there is nothing you can do in setuptools to customise that process.

If you are looking for an alternative, you can try something like:

• https://pypi.org/project/vendoring/
• https://pypi.org/project/vendorize/
• https://docs.python.org/3/library/zipimport.html

[ydh10002023]

@abravalheri thanks.
I checked these methods, It seems that it is not appropriate. It seems that the user needs some import changes.

I think I can download and decompress b.whl, then build the whl to include a and b to one whl. then we can create a.whl to include two pachages(a, and b). when installing this whl, we can see the top level will be the following. Maybe it's a better way?

a
b

[webknjaz]

That'd be manual. Wheels usually don't include the source packaging configuration. So if you do wheel unpack, you'd have to figure out how to reassemble that.
FWIW for this case you'd better go for building from sdist. But again, you'd have to come up with a new packaging configuration that would include both sources side-by-side.

[webknjaz]

Also, the best place to learn about different packaging approaches is https://packaging.python.org.

[abravalheri]

when installing this whl, we can see the top level will be the following. Maybe it's a better way?

Probably not, (on top of everything webknjaz pointed out) if you include the b folder at the same level as a, you are under the risk of overwriting a different standalone installation of b. They might have different versions and clash with each other.

In general, we call what you want to do "vendoring". In my opinion the best you can do when vendoring, is indeed implement import changes to minimize the risk of clashes with global installations, eg:

from ._vendor import b
# instead of
import b

This will allow the package b to exist simultaneously inside and outside of package a, without having the risk of version clashes and dependency hell.

But well, everyone has different constraints or objectives, so if it works for you, why not?

[webknjaz]

if you include the b folder at the same level as a, you are under the risk of overwriting a different standalone installation of b.

Not just simply overwriting but replacing bits of it partially and then having fun debugging marvelous side effects...

[ydh10002023]

yes, it is not a good method

[ydh10002023]

thanks @abravalheri and @webknjaz

If use this from ._vendor import b, it seems like I need to change the import in many places. and I need to change the code of b. b(whl or gz) is another inner product, but it doesn't release to pypi, I want to build it to my whl package directly, otherwise this depedency can't be downloaded from outside. If b is taken as package_data of a, it seems that there is no post-processing step for this PIP installation to install b directly?

[webknjaz]

@webknjaz I'm doing a.whl products, it needs to depend on b.whl and c.whl, but b.whl and c.whl is other internal products, and they cannot be published to pypi. However, I need to publish our product a.whl to pypi. Do you have a better way to package b.whl and c.whl to a.whl? thanks very much

Publish those into an additional additional private index like devpi. Or use a "wheelhouse" — copy those private wheels into a directory during deployment, then point pip there like pip install -f ./wheelhouse-dir project-a.

[ydh10002023]

@webknjaz If so, it seems that the user needs additional steps to install, and he also needs access to private repo. It seems that it will not be convenient for customers, Am i right?

[webknjaz]

Well, if you don't want that code to be public, that's the way.

[ydh10002023]

thanks @webknjaz. Now my method is to decompress b.whl, then cythonize them, and add them to the subdirectory of project a, and then package a to a.whl. It will not affect the separately installed b.wh, if it exists. This is also convenient for users, and is no different for installing with other whls. Do you think it is good?

[webknjaz]

I don't see a point not to publish "b", then. Since you're already exposing it. Plus, wheels aren't source dists so treating them like source is something that was never a thing.