ARROW-9266: [Python][Packaging] Enable S3 support in macOS wheels #8315

kszucs · 2020-10-01T09:55:33Z

Depends on #8304

kszucs · 2020-10-01T09:57:22Z

@github-actions crossbow submit wheel-osx-mavericks-cp36m wheel-osx-high-sierra-cp36m

github-actions · 2020-10-01T09:59:33Z

Revision: 430ffdf472b9420cdbb5784946e2ba0edaf2f5c8

Submitted crossbow builds: ursa-labs/crossbow @ actions-571

Task	Status
wheel-osx-high-sierra-cp36m
wheel-osx-mavericks-cp36m

github-actions · 2020-10-01T10:10:39Z

https://issues.apache.org/jira/browse/ARROW-9266

kszucs · 2020-10-01T11:09:44Z

@github-actions crossbow submit wheel-osx-*

github-actions · 2020-10-01T11:12:04Z

Revision: 430ffdf472b9420cdbb5784946e2ba0edaf2f5c8

Submitted crossbow builds: ursa-labs/crossbow @ actions-572

Task	Status
wheel-osx-high-sierra-cp35m
wheel-osx-high-sierra-cp36m
wheel-osx-high-sierra-cp37m
wheel-osx-high-sierra-cp38
wheel-osx-mavericks-cp35m
wheel-osx-mavericks-cp36m
wheel-osx-mavericks-cp37m
wheel-osx-mavericks-cp38

kszucs · 2020-10-05T11:02:53Z

@github-actions crossbow submit wheel-osx-*

github-actions · 2020-10-05T11:04:36Z

Revision: 50aaa55f306fd492cb743f011c9968860dbab2d8

Submitted crossbow builds: ursa-labs/crossbow @ actions-598

Task	Status
wheel-osx-high-sierra-cp35m
wheel-osx-high-sierra-cp36m
wheel-osx-high-sierra-cp37m
wheel-osx-high-sierra-cp38
wheel-osx-mavericks-cp35m
wheel-osx-mavericks-cp36m
wheel-osx-mavericks-cp37m
wheel-osx-mavericks-cp38

nealrichardson · 2020-10-05T22:06:00Z

@kszucs please rebase

kszucs · 2020-10-05T22:09:43Z

@github-actions crossbow submit wheel-osx-*

github-actions · 2020-10-05T22:11:54Z

Revision: 16a9f0916e8a9c5e10e0f66a086ad54c84676d34

Submitted crossbow builds: ursa-labs/crossbow @ actions-604

Task	Status
wheel-osx-high-sierra-cp35m
wheel-osx-high-sierra-cp36m
wheel-osx-high-sierra-cp37m
wheel-osx-high-sierra-cp38
wheel-osx-mavericks-cp35m
wheel-osx-mavericks-cp36m
wheel-osx-mavericks-cp37m
wheel-osx-mavericks-cp38

kou · 2020-10-06T01:23:10Z

dev/tasks/python-wheels/osx-build.sh

Could you keep alphabetical order?

kou · 2020-10-06T01:23:25Z

dev/tasks/python-wheels/osx-build.sh

kszucs · 2020-10-06T10:24:10Z

@github-actions crossbow submit wheel-osx-*

github-actions · 2020-10-06T10:25:50Z

Revision: 97b5b32

Submitted crossbow builds: ursa-labs/crossbow @ actions-605

Task	Status
wheel-osx-high-sierra-cp35m
wheel-osx-high-sierra-cp36m
wheel-osx-high-sierra-cp37m
wheel-osx-high-sierra-cp38
wheel-osx-mavericks-cp35m
wheel-osx-mavericks-cp36m
wheel-osx-mavericks-cp37m
wheel-osx-mavericks-cp38

kszucs · 2020-10-07T13:06:17Z

@github-actions crossbow submit wheel-osx-*

kszucs · 2020-10-07T13:08:43Z

The successful builds take between ~42-45 minutes but occasionally it exceeds the 50m build time limit. Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

github-actions · 2020-10-07T13:11:28Z

Revision: c40e6e4

Submitted crossbow builds: ursa-labs/crossbow @ actions-611

Task	Status
wheel-osx-high-sierra-cp35m
wheel-osx-high-sierra-cp36m
wheel-osx-high-sierra-cp37m
wheel-osx-high-sierra-cp38
wheel-osx-mavericks-cp35m
wheel-osx-mavericks-cp36m
wheel-osx-mavericks-cp37m
wheel-osx-mavericks-cp38

kszucs · 2020-10-07T14:34:36Z

@github-actions crossbow submit wheel-osx-mavericks-cp38

github-actions · 2020-10-07T14:46:57Z

Revision: 7bcd83e

Submitted crossbow builds: ursa-labs/crossbow @ actions-613

Task	Status
wheel-osx-mavericks-cp38

corleyma · 2020-10-07T15:38:44Z

The successful builds take between ~42-45 minutes but occasionally it exceeds the 50m build time limit. Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

Does this explain why some of the nightly builds seem to be intermittently missing certain macOS wheels? E.g., sometimes for example there's no cp38 wheel, only for that wheel to come back with the next nightly publish.

nealrichardson · 2020-10-07T15:44:24Z

Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

We could run something on master on the crossbow repo to warm the cache.

corleyma · 2020-10-07T15:47:35Z

I think @nealrichardson is right: my understanding is that travis caching will try to pull from default branch if no cache is available for current branch, so the warming strategy could work.

kszucs · 2020-10-07T20:13:09Z

Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

We could run something on master on the crossbow repo to warm the cache.

We would need to merge the latests tasks (multiple .travis.yml files into a single one) generated by crossbow to master automatically which would be rather complicated. Crossbow also creates diverged branches from master at the moment.

We can make a jira about supporting CI caches in crossbow though I can't see a generic solution at the moment.

kszucs · 2020-10-07T20:15:51Z

Does this explain why some of the nightly builds seem to be intermittently missing certain macOS wheels? E.g., sometimes for example there's no cp38 wheel, only for that wheel to come back with the next nightly publish.

Could you send me build links? I don't entirely see your question.

kszucs

+1, merging.

kszucs mentioned this pull request Oct 5, 2020

ARROW-10068: [C++] Add bundled external project for aws-sdk-cpp #8304

Closed

kszucs force-pushed the macos3 branch from 430ffdf to 50aaa55 Compare October 5, 2020 11:02

kou mentioned this pull request Oct 5, 2020

ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels #7997

Closed

kszucs force-pushed the macos3 branch from 50aaa55 to 16a9f09 Compare October 5, 2020 22:09

kou reviewed Oct 6, 2020

View reviewed changes

kszucs added 2 commits October 6, 2020 12:01

enable s3 in the macos wheels

bd480b7

sort

97b5b32

kszucs force-pushed the macos3 branch from 16a9f09 to 97b5b32 Compare October 6, 2020 10:01

try not updating brew

c40e6e4

turn off s3 on mavericks py38

7bcd83e

kszucs commented Oct 7, 2020

View reviewed changes

kszucs closed this in 732e333 Oct 7, 2020

asfimport mentioned this pull request Oct 7, 2020

[Python][Packaging] Enable S3 support in macOS wheels #17248

Closed

ARROW-9266: [Python][Packaging] Enable S3 support in macOS wheels #8315

ARROW-9266: [Python][Packaging] Enable S3 support in macOS wheels #8315

Uh oh!

Conversation

kszucs commented Oct 1, 2020

Uh oh!

kszucs commented Oct 1, 2020

Uh oh!

github-actions bot commented Oct 1, 2020

Uh oh!

github-actions bot commented Oct 1, 2020

Uh oh!

kszucs commented Oct 1, 2020

Uh oh!

github-actions bot commented Oct 1, 2020

Uh oh!

kszucs commented Oct 5, 2020

Uh oh!

github-actions bot commented Oct 5, 2020

Uh oh!

nealrichardson commented Oct 5, 2020

Uh oh!

kszucs commented Oct 5, 2020

Uh oh!

github-actions bot commented Oct 5, 2020

Uh oh!

kou Oct 6, 2020

Choose a reason for hiding this comment

Uh oh!

kou Oct 6, 2020

Choose a reason for hiding this comment

Uh oh!

kszucs commented Oct 6, 2020

Uh oh!

github-actions bot commented Oct 6, 2020

Uh oh!

kszucs commented Oct 7, 2020

Uh oh!

kszucs commented Oct 7, 2020

Uh oh!

github-actions bot commented Oct 7, 2020

Uh oh!

kszucs commented Oct 7, 2020

Uh oh!

github-actions bot commented Oct 7, 2020

Uh oh!

corleyma commented Oct 7, 2020

Uh oh!

nealrichardson commented Oct 7, 2020

Uh oh!

corleyma commented Oct 7, 2020

Uh oh!

kszucs commented Oct 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Oct 7, 2020

Uh oh!

kszucs left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kszucs commented Oct 7, 2020 •

edited

Loading