ARROW-10068: [C++] Add bundled external project for aws-sdk-cpp #8304

kszucs · 2020-09-29T20:42:19Z

No description provided.

cpp/cmake_modules/ThirdpartyToolchain.cmake

github-actions · 2020-09-29T20:47:39Z

https://issues.apache.org/jira/browse/ARROW-10068

kszucs · 2020-09-29T20:50:19Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

I'm not sure how feasible is this, perhaps we can turn off the curl dependency of AWS. I assume linking against the system curl should be fine rather than compiling curl as well.

cpp/cmake_modules/ThirdpartyToolchain.cmake

cpp/thirdparty/versions.txt

kszucs · 2020-09-29T21:39:03Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

Note that AWS::aws-cpp-sdk-config wasn't required although we look for it find_package components.

nealrichardson · 2020-09-30T22:16:41Z

@kou since you're looking at this now, wanted to let you know I'm debugging locally how on the rstudio/r-base:3.6-centos7 image, it can't find openssl even though I've yum installed it (debugging and failing, that is, and my next step is the bundled openssl build, the nuclear option I'm afraid). I'm currently skipping that problem though and working on the patch to aws-sdk-cpp so that git isn't required.

kou · 2020-10-01T00:09:19Z

Thanks for sharing the current status.
I'll also take a look the OpenSSL problem.

nealrichardson · 2020-10-01T00:46:54Z

I got past that particular openssl problem by installing openssl-static as @pitrou suggested, though I suspect there's more to the issue.

corleyma · 2020-10-01T13:27:52Z

I know that this is not quite related, but since we're looking upstream now... is there any thought to when/if we'll update the AWS SDK version? I ask because as of 1.8.0, the default credentials resolution, etc, was enhanced to more fully match what you might expect from awscli/boto3.

corleyma · 2020-10-01T13:30:19Z

I know that this is not quite related, but since we're looking upstream now... is there any thought to when/if we'll update the AWS SDK version? I ask because as of 1.8.0, the default credentials resolution, etc, was enhanced to more fully match what you might expect from awscli/boto3.

Actually, ignore me, I see the version change in this PR now. That's exciting! Given these changes it might be worth (at some point) taking another look at how credentials specification is handled to see if there's anything that can be streamlined.

nealrichardson · 2020-10-01T15:12:39Z

@corleyma for R on macOS and Windows we're still using binaries for aws-sdk-cpp that are 1.7, so we can't assume 1.8 in the Arrow code just yet. cf. autobrew/homebrew-core#25

nealrichardson · 2020-10-01T16:08:54Z

The latest R failures from the CentOS 7 builds are compiler warnings being converted to errors in the awssdk_ep. As it turns out, aws-sdk-cpp requires gcc >= 4.9, and these builds are on 4.8. So somewhere we're going to need to add a if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS "4.9") check.

nealrichardson · 2020-10-01T16:11:54Z

@github-actions crossbow submit test-r-*

github-actions · 2020-10-01T16:13:59Z

Revision: 41e2b98e6136a935137061cb3e0292371f989c57

Submitted crossbow builds: ursa-labs/crossbow @ actions-573

Task	Status
test-r-linux-as-cran
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos6
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42

nealrichardson · 2020-10-01T18:50:12Z

@github-actions crossbow submit test-r-r*

github-actions · 2020-10-01T18:52:00Z

Revision: a2f90756b9a0f91e4a7c347720b42b1ade567bcb

Submitted crossbow builds: ursa-labs/crossbow @ actions-574

Task	Status
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos6
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42

nealrichardson · 2020-10-01T22:11:51Z

Latest R issue(s):

The R Linux builds do a static Arrow C++ build, which gets pulled into a shared arrow.so library with the R bindings when the R package builds. AFAICT, because the C++ build is static, it insists on the static version of openssl. So IIUC when R makes arrow.so, the openssl symbols will be pulled into the shared library--exactly what folks on https://issues.apache.org/jira/browse/ARROW-10138 were concerned to prevent.
aws-sdk-cpp doesn't build on gcc 4.8, so we need to catch that in cmake. I'd like to have a way to build like ARROW_S3=IF_POSSIBLE, like build with S3 support if we can but fall back to ARROW_S3=OFF rather than fail the build if some prerequisite is not met (similar to ARROW-8155). If there's no interest in supporting that in cmake itself, I'll find some way to hack that into the R Linux build script.

nealrichardson · 2020-10-01T23:04:59Z

@github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic

github-actions · 2020-10-01T23:06:44Z

Revision: 992cb382c84169babf76d85a502c4ec8a16979d8

Submitted crossbow builds: ursa-labs/crossbow @ actions-575

Task	Status
test-r-rstudio-r-base-3.6-bionic

nealrichardson · 2020-10-01T23:32:23Z

@github-actions crossbow submit test-r-rstudio-r-base-3.6-bionic

github-actions · 2020-10-01T23:34:41Z

Revision: 66593cdf2aeaf67c1b087c4e9d789f8fd35620c8

Submitted crossbow builds: ursa-labs/crossbow @ actions-576

Task	Status
test-r-rstudio-r-base-3.6-bionic

kou · 2020-10-02T00:27:35Z

Linux:

We need MinIO for testing S3 filesystem
- There is no package for MinIO in system package manager
- MinIO provides binary only for amd64
- We need to download it in our Dockerfiles for amd64
- We need to build MinIO on non-amd64 platforms such as arm64

kou · 2020-10-02T00:31:46Z

  * We need to build MinIO on non-amd64 platforms such as arm64

It's not true: https://dl.min.io/server/minio/release/linux-arm64/archive/

kou · 2020-10-05T11:00:15Z

We can complete #7997 after we merge this.

kszucs · 2020-10-05T11:01:43Z

We can complete #7997 after we merge this.

I have a working PR #8315 based on this branch. I'm going to rebase that to check the builds are still passing.

kou · 2020-10-05T11:03:11Z

@github-actions crossbow submit *-fedora-cpp -ubuntu-20.04-cpp *-debian-10-cpp

kou · 2020-10-05T11:05:14Z

OK. I closed #7997.

kou · 2020-10-05T11:09:15Z

@github-actions crossbow submit *-fedora-cpp

kou · 2020-10-05T11:13:23Z

@github-actions crossbow submit test-fedora-32-cpp test-debian-10-cpp test-ubuntu-20.04-cpp

github-actions · 2020-10-05T11:15:23Z

Revision: 0657942

Submitted crossbow builds: ursa-labs/crossbow @ actions-599

Task	Status
test-debian-10-cpp
test-fedora-32-cpp
test-ubuntu-20.04-cpp

nealrichardson · 2020-10-05T16:51:32Z

@github-actions crossbow submit -g r

github-actions · 2020-10-05T16:53:22Z

Revision: 0657942

Submitted crossbow builds: ursa-labs/crossbow @ actions-600

Task	Status
homebrew-r-autobrew
test-conda-r-4.0
test-r-linux-as-cran
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos6
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42
test-ubuntu-18.04-r-sanitizer

nealrichardson · 2020-10-05T16:53:50Z

I am revising the R installation guide to mention S3 support and the required system dependencies. Will push that today.

@kszucs @kou are there other developer guides that need revision here? I still think it's odd that we now have a setup with ARROW_DEPENDENCY_SOURCE=BUNDLED but that still requires system dependencies, so I'm guessing there are docs that need to be clarified.

nealrichardson · 2020-10-05T19:10:49Z

@github-actions crossbow submit -g r

github-actions · 2020-10-05T19:13:09Z

Revision: 548b6f3

Submitted crossbow builds: ursa-labs/crossbow @ actions-601

Task	Status
homebrew-r-autobrew
test-conda-r-4.0
test-r-linux-as-cran
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos6
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42
test-ubuntu-18.04-r-sanitizer

nealrichardson · 2020-10-05T19:56:43Z

@github-actions crossbow submit test-r-rhub-ubuntu-gcc-release test-r-rocker-r-base-latest

github-actions · 2020-10-05T20:02:55Z

Revision: 406febf

Submitted crossbow builds: ursa-labs/crossbow @ actions-603

Task	Status
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest

nealrichardson

+1 from me. R Linux builds now have ARROW_S3 on where possible, and they (nearly) all run tests with minio too. Thanks everyone for pushing on this!

kou · 2020-10-05T21:36:46Z

Thanks!

We have https://arrow.apache.org/docs/developers/cpp/building.html#build-dependency-management for C++.
(I think that updating the documentation can be worked as a follow-up task.)

nealrichardson · 2020-10-05T21:39:31Z

(I think that updating the documentation can be worked as a follow-up task.)

Ok. I'll make a followup JIRA and merge this. Thanks again!

nealrichardson · 2020-10-05T21:44:33Z

https://issues.apache.org/jira/browse/ARROW-10180 for the docs

Depends on #8304 Closes #8315 from kszucs/macos3 Authored-by: Krisztián Szűcs <[email protected]> Signed-off-by: Krisztián Szűcs <[email protected]>