ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels #7997
Conversation
|
@github-actions crossbow submit wheel-osx-* |
kou
changed the title
|
Revision: ad77b8f Submitted crossbow builds: ursa-labs/crossbow @ actions-493
|
|
So this is failing because it's trying to build the bundled aws-sdk-cpp external project, but that's not implemented: https://travis-ci.org/github/ursa-labs/crossbow/builds/719137336#L1231-L1232 You could try adding |
|
I don't remember the specifics, but IIRC the AWS SDK build procedure always takes up the system version of OpenSSL, while in many cases (especially packaging-related) we want to use a dedicated OpenSSL version (such as Conda-provided version when building a conda package, etc.). |
|
That was with an older AWS SDK version some time ago, though, so you could try to enable it and see what happens now. |
|
@github-actions crossbow submit wheel-osx-* |
|
Revision: e2359f5 Submitted crossbow builds: ursa-labs/crossbow @ actions-496
|
|
FWIW the latest Travis failures do look like aws/aws-sdk-cpp#1309. I don't think Homebrew is a viable source for aws-sdk-cpp for us. |
|
@kou I pushed some changes which optimistically re-enable building of AWS C++ sdk from source. @pitrou As said before, I'm not super familiar with conda but a cursory glance at the docs implies it might be possible to build wheels from conda? I wonder if that might be a good approach for the macOS wheels. |
|
The approach we use for manylinux wheels is to build the AWS SDK from source separately, and it works. Why would it be different on macOS? |
I'm not sure if it would be possible to use conda to build manylinux-compatible wheels, but in general it seems that the conda builds of pyarrow are more supported than the wheels. I wonder if using conda-build to generate wheels would reduce the lag time for new functionality to make it into the wheels and reduce overhead for maintainers. |
|
I can't answer definitely, but I'm extremely skeptical that conda-build will be able to proper usable Python wheels in anything other than the most simple cases, and Arrow is really at the other end of the complexity spectrum when it comes to dependencies and packaging. To understand why, you have to understand that Python wheels are not able to express non-Python dependencies, while conda packages are. So any C++ dependency that's expressed naturally in conda (as a dependency on just another package) has to become statically bundled in a Python wheel. |
|
So I maintain that building AWS SDK from source during the wheel build process is probably the most reasonable way forward here. |
Bundling those dependencies is exactly what I was hoping conda would do when building a wheel. I guess that's probably being a bit optimistic. |
|
@github-actions crossbow submit wheel-osx-* FYI: Anybody can run jobs to build wheels for macOS by writing the above comment. |
|
Revision: 0b3365d Submitted crossbow builds: ursa-labs/crossbow @ actions-497
|
|
@kou Ah, interesting, good to know! I could not seem to get @pitrou It looks like there is something called conda-press that may be closer to what I was imagining, but it doesn't seem very mature yet. |
|
@github-actions crossbow submit wheel-osx-* |
|
@kou seems crossbow builds won't run for me, alas. 😿 |
|
@github-actions crossbow submit wheel-osx-* Oh, sorry. I didn't know that we have the https://github.com/apache/arrow/blob/master/dev/archery/archery/bot.py#L180-L183 check. |
|
We experimented a bit with conda-press a while back but it yielded poor results for us (wheels that were much larger than our current ones). I expect we are going to be fighting to keep our wheels at an acceptable size for the foreseeable future (at some point we need to try to break up the Python project into multiple interdependent wheels to enable modular installations) |
|
@github-actions crossbow submit wheel-osx-* |
|
Revision: e7948be Submitted crossbow builds: ursa-labs/crossbow @ actions-498
|
|
If you're going to try to get the bundled
|
|
We'll complete this by #8315. |
5 participants
This PR updates the osx-build.sh script to enable S3 when building the C++ dependencies and the python wheels, and updates the corresponding travis job definition to include the requisite aws-sdk-cpp dependency via homebrew.