ARROW-4649: [C++/CI/R] Add nightly job that tests the homebrew formula #5360

nealrichardson · 2019-09-11T20:57:34Z

This doesn't just test brew install apache-arrow --HEAD as the ticket suggested--it brings the Homebrew formula under our source control and tests against that. This will enable us to know when we need to update the Homebrew formula due to changes in dependencies or cmake configuration.

I've been testing out this approach on a different repository: https://travis-ci.org/nealrichardson/arrow-brew/builds/568531245

nealrichardson · 2019-09-11T20:58:51Z

@ursabot crossbow submit homebrew-formula

ursabot · 2019-09-11T20:59:10Z

AMD64 Conda Crossbow Submit (#61145) builder has been succeeded.

Revision: 2e5254d

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-180

Task	Status
homebrew-formula

nealrichardson · 2019-09-11T21:48:49Z

This fails, and it is correct that it fails. Compare the previous passing build I linked with https://travis-ci.org/nealrichardson/arrow-brew/builds/583867759, run on that repository just now.

The failure is

[ 19%] Generating Flight.pb.cc, Flight.pb.h, Flight.grpc.pb.cc, Flight.grpc.pb.h
cd /tmp/apache-arrow-20190911-98334-mvj5bo/build/src/arrow/flight && /usr/local/bin/protoc -I/tmp/apache-arrow-20190911-98334-mvj5bo/cpp/../format --cpp_out=/tmp/apache-arrow-20190911-98334-mvj5bo/build/src/arrow/flight /tmp/apache-arrow-20190911-98334-mvj5bo/cpp/../format/Flight.proto
cd /tmp/apache-arrow-20190911-98334-mvj5bo/build/src/arrow/flight && /usr/local/bin/protoc -I/tmp/apache-arrow-20190911-98334-mvj5bo/cpp/../format --grpc_out=/tmp/apache-arrow-20190911-98334-mvj5bo/build/src/arrow/flight --plugin=protoc-gen-grpc=/usr/local/Cellar/grpc/1.23.0_2/bin/grpc_cpp_plugin /tmp/apache-arrow-20190911-98334-mvj5bo/cpp/../format/Flight.proto
dyld: Library not loaded: /usr/local/opt/gperftools/lib/libprofiler.0.dylib
  Referenced from: /usr/local/Cellar/grpc/1.23.0_2/bin/grpc_cpp_plugin
  Reason: image not found
--grpc_out: protoc-gen-grpc: Plugin killed by signal 6.
make[2]: *** [src/arrow/flight/Flight.pb.cc] Error 1
make[2]: *** Deleting file `src/arrow/flight/Flight.pb.cc'
make[1]: *** [src/arrow/flight/CMakeFiles/flight_grpc_gen.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 19%] Built target arrow_dataset_objlib
make: *** [all] Error 2

I don't think that failure should block the merging of this patch: it might be more appropriate to fix the formula on its own ticket, just to track the issue and make clear at release time that there's been a change to the homebrew formula that we'll need to publish upstream.

Thoughts @kou @xhochy ?

kou · 2019-09-12T02:33:59Z

I think that the grpc Formula in Homebrew misses the gperftools Formula dependency.

Could you send a pull request to https://github.com/homebrew/homebrew-core ?

diff --git a/Formula/grpc.rb b/Formula/grpc.rb
index dff0ba50cb..b99120414d 100644
--- a/Formula/grpc.rb
+++ b/Formula/grpc.rb
@@ -3,7 +3,7 @@ class Grpc < Formula
   homepage "https://grpc.io/"
   url "https://github.com/grpc/grpc/archive/v1.23.0.tar.gz"
   sha256 "f56ced18740895b943418fa29575a65cc2396ccfa3159fa40d318ef5f59471f9"
-  revision 2
+  revision 3
   head "https://github.com/grpc/grpc.git"
 
   bottle do
@@ -17,6 +17,7 @@ class Grpc < Formula
   depends_on "libtool" => :build
   depends_on "c-ares"
   depends_on "gflags"
+  depends_on "gperftools"
   depends_on "[email protected]"
   depends_on "protobuf"

nealrichardson · 2019-09-12T03:07:00Z

Did we change something since August 6 that now requires gperftools? grpc hasn't changed wrt gperftools in that time. Should we ourselves just add a dependency on gperftools?

kou · 2019-09-12T03:17:18Z

This is not related to the apache-arrow formula.
Because grpc_cpp_plugin provided by the grpc formula uses libprofiler.0.dylib:

$ otool -L /usr/local/Cellar/grpc/1.23.0_2/bin/grpc_cpp_plugin
/usr/local/Cellar/grpc/1.23.0_2/bin/grpc_cpp_plugin:
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1575.17.0)
	/usr/local/opt/protobuf/lib/libprotoc.20.dylib (compatibility version 21.0.0, current version 21.1.0)
	/usr/local/opt/protobuf/lib/libprotobuf.20.dylib (compatibility version 21.0.0, current version 21.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/local/opt/gperftools/lib/libprofiler.0.dylib (compatibility version 5.0.0, current version 5.18.0)
	/usr/local/opt/c-ares/lib/libcares.2.dylib (compatibility version 6.0.0, current version 6.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)

I think that gRPC detects gperftools automatically and gperftools is installed unexpectedly on environment that builds bottles for the grpc formula.

nealrichardson · 2019-09-12T03:32:37Z

Sure. I'm just saying, I got this formula to build successfully on Travis most recently on August 6. And obviously it built fine at our last release. So do we have a theory for why this is now a problem?

kou · 2019-09-12T03:38:50Z

I think that this is caused by Homebrew/homebrew-core@8defc4a . (August 24)

kou · 2019-09-12T03:39:44Z

Or Homebrew/homebrew-core@f48ab37 (August 16)

kou · 2019-09-12T03:40:39Z

Or Homebrew/homebrew-core@2cc1d02 (September 9)

nealrichardson · 2019-09-12T03:41:49Z

So it was just luck that wherever the bottles were made before had it, and suddenly it doesn't?

kou · 2019-09-12T03:51:54Z

I think that we should ask to Homebrew developers. :-)

nealrichardson · 2019-09-12T04:06:05Z

👍 ok I can initiate that at Homebrew-core tomorrow (unless you want to right now); in the meantime, do you want to accept this patch? It's not like a(nother) failing nightly build is going to hold anyone up here.

kou · 2019-09-12T04:15:23Z

Did you copy ci/apache-arrow.rb from https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-arrow.rb ?
Then the license isn't Apache 2.0. It's BSD 2-Clause License https://github.com/Homebrew/homebrew-core/blob/master/LICENSE.txt .

nealrichardson · 2019-09-12T04:16:41Z

I did. So I'll add to our LICENSE file then. Good catch, sorry about that.

kou · 2019-09-12T04:17:32Z

ci/apache-arrow.rb

+  desc "Columnar in-memory analytics layer designed to accelerate big data"
+  homepage "https://arrow.apache.org/"
+  url "https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-0.14.1/apache-arrow-0.14.1.tar.gz"
+  sha256 "9948ddb6d4798b51552d0dca3252dd6e3a7d0f9702714fc6f5a1b59397ce1d28"


When should we update url and sha256?
If it's release time, we should update them automatically in release process.
(dev/release/00-prepare.sh?)

Agreed, will do.

ci/apache-arrow.rb

kou · 2019-09-12T04:18:33Z

ci/apache-arrow.rb

+    sha256 "b5d11576669485b771d2fab149d499b41ec9d0fcda55d7f4ec98716e1e2f5c08" => :sierra
+  end
+
+  depends_on "autoconf" => :build


Note that we can remove this when we bundle jemalloc that has configure.

Cool, is there a ticket for that on which we can note this?

No.
FYI: #5365 will bundle jemalloc that has configure.

Since that merged, I've removed this line

kszucs · 2019-09-12T06:04:29Z

@nealrichardson could you move every the formula under dev/tasks/brew-formulas, similarly like other packaging tasks' assets are placed under dev/tasks.

Also we need to add a license note, that we have copied code from homebrew, it has BSD-2, similarly like we reference the vendored the conda-forge recipes.

dev/tasks/tasks.yml

nealrichardson · 2019-09-13T16:06:54Z

@ursabot crossbow submit homebrew-formula

ursabot · 2019-09-13T16:07:12Z

AMD64 Conda Crossbow Submit (#61857) builder has been succeeded.

Revision: 58bfe1c

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-188

Task	Status
homebrew-formula

nealrichardson · 2019-09-13T18:18:44Z

Ok, feedback addressed, and Homebrew-core has resolved the grpc issue, so the build passes now.

I was able to add the version updating to the release script, but I didn't see where it was appropriate to update the sha after the release happens. But even if that remains a manual step, we're still better off than we were before.

kou

@wesm Do you know any workflow when we send a patch to non ASF projects from our repository?
We'll send a patch to Homebrew from a formula file in this repository. Should we keep the original license (BSD 2-Clause License) of a formula file in this repository? Or should we relicense a formula to Apache License 2.0 in this repository and relicense to BSD 2-Clause License again?
I think that we should keep the original license (BSD 2-Clause License) to avoid relicensing.

LICENSE.txt

dev/tasks/tasks.yml

ci/apache-arrow.rb

dev/release/00-prepare.sh

nealrichardson · 2019-09-17T20:25:54Z

@ursabot crossbow submit homebrew-cpp

ursabot · 2019-09-17T20:26:13Z

AMD64 Conda Crossbow Submit (#62952) builder has been succeeded.

Revision: f155a27

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-195

Task	Status
homebrew-cpp

nealrichardson · 2019-09-17T20:26:20Z

@ursabot crossbow submit homebrew-cpp-autobrew

ursabot · 2019-09-17T20:26:41Z

AMD64 Conda Crossbow Submit (#62954) builder has been succeeded.

Revision: f155a27

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-196

Task	Status
homebrew-cpp-autobrew

nealrichardson · 2019-09-17T20:27:18Z

Revised, PTAL

…requires it

nealrichardson · 2019-09-20T18:05:02Z

Ok, I've addressed @kou's feedback and the lint/release tests pass: https://travis-ci.org/apache/arrow/jobs/587562985

ci/apache-arrow.rb

dev/tasks/tasks.yml

kszucs · 2019-09-20T18:15:03Z

@nealrichardson it looks good to me, I only have concerns with the file organization.

…ion in release script

nealrichardson · 2019-09-20T21:20:22Z

@ursabot crossbow submit homebrew-cpp homebrew-cpp-autobrew

ursabot · 2019-09-20T21:20:43Z

AMD64 Conda Crossbow Submit (#64046) builder has been succeeded.

Revision: 1d6512b

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-210

Task	Status
homebrew-cpp
homebrew-cpp-autobrew

nealrichardson · 2019-09-20T22:27:09Z

@ursabot crossbow submit homebrew-cpp-autobrew

ursabot · 2019-09-20T22:27:29Z

AMD64 Conda Crossbow Submit (#64067) builder has been succeeded.

Revision: ea3f9b5

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-211

Task	Status
homebrew-cpp-autobrew

nealrichardson · 2019-09-20T22:53:07Z

https://travis-ci.org/nealrichardson/arrow/jobs/587687890 lint/release test passing again. Once homebrew-cpp-autobrew passes, this is good to go.

nealrichardson · 2019-09-21T00:17:31Z

@kou we are finally ready to merge here. Thanks for all of your feedback!

kou

+1
Thanks! I'll merge this!

kou reviewed Sep 12, 2019

View reviewed changes

kszucs reviewed Sep 12, 2019

View reviewed changes

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

nealrichardson force-pushed the homebrew-nightly branch from 2e5254d to 7281d8e Compare September 12, 2019 21:16

nealrichardson mentioned this pull request Sep 12, 2019

Add gperftools dependency to grpc Homebrew/homebrew-core#44198

Closed

5 tasks

kou requested changes Sep 14, 2019

View reviewed changes

LICENSE.txt Outdated Show resolved Hide resolved

dev/tasks/tasks.yml Show resolved Hide resolved

ci/apache-arrow.rb Outdated Show resolved Hide resolved

dev/release/00-prepare.sh Outdated Show resolved Hide resolved

nealrichardson force-pushed the homebrew-nightly branch from cf88432 to f155a27 Compare September 17, 2019 20:24

nealrichardson added 12 commits September 20, 2019 09:18

Rename homebrew task yml; add to LICENSE

262c241

Re-alphabetize the seds in 00-prepare.sh

ea4368e

Attempt to set the homebrew formula arrow version in the release script

40fc370

Add missing test assertion

34af2a9

Fix regular expression

6d00fe4

Fix test (hopefully?)

a133385

Remove bottle shas

d61290a

Parametrize homebrew task and also test 'autobrew'

89dea4f

Fix licensing/:rat: for ci/apache-arrow.rb

b25ac6d

Fix autobrew audit check [skip ci]

8816279

Remove autoconf from homebrew formula because new jemalloc no longer …

79e9191

…requires it

Use regular snapshot version for homebrew formula

ff1489e

nealrichardson force-pushed the homebrew-nightly branch from 6403234 to ff1489e Compare September 20, 2019 16:25

kszucs reviewed Sep 20, 2019

View reviewed changes

ci/apache-arrow.rb Outdated Show resolved Hide resolved

kszucs reviewed Sep 20, 2019

View reviewed changes

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

Move homebrew formulae inside dev/tasks. Update autobrew package vers…

1d6512b

…ion in release script

nealrichardson added 2 commits September 20, 2019 14:43

Sort hunks 💪

851ac6e

Move autobrew script again and fix URL to match regular homebrew

ea3f9b5

Shuffle again

4a50a37

kou approved these changes Sep 21, 2019

View reviewed changes

kou closed this in b2785d3 Sep 21, 2019

asfimport mentioned this pull request Sep 21, 2019

[C++/CI/R] Add (nightly) job that builds brew install apache-arrow --HEAD #21182

Closed

ARROW-4649: [C++/CI/R] Add nightly job that tests the homebrew formula #5360

ARROW-4649: [C++/CI/R] Add nightly job that tests the homebrew formula #5360

Conversation

nealrichardson commented Sep 11, 2019

nealrichardson commented Sep 11, 2019

ursabot commented Sep 11, 2019

nealrichardson commented Sep 11, 2019

kou commented Sep 12, 2019 • edited Loading

nealrichardson commented Sep 12, 2019

kou commented Sep 12, 2019

nealrichardson commented Sep 12, 2019

kou commented Sep 12, 2019 • edited Loading

kou commented Sep 12, 2019 • edited Loading

kou commented Sep 12, 2019

nealrichardson commented Sep 12, 2019

kou commented Sep 12, 2019

nealrichardson commented Sep 12, 2019

kou commented Sep 12, 2019

nealrichardson commented Sep 12, 2019

kou Sep 12, 2019

Choose a reason for hiding this comment

nealrichardson Sep 12, 2019

Choose a reason for hiding this comment

kou Sep 12, 2019

Choose a reason for hiding this comment

nealrichardson Sep 12, 2019

Choose a reason for hiding this comment

kou Sep 13, 2019

Choose a reason for hiding this comment

nealrichardson Sep 18, 2019

Choose a reason for hiding this comment

kszucs commented Sep 12, 2019

nealrichardson commented Sep 13, 2019

ursabot commented Sep 13, 2019

nealrichardson commented Sep 13, 2019

kou left a comment

Choose a reason for hiding this comment

nealrichardson commented Sep 17, 2019

ursabot commented Sep 17, 2019

nealrichardson commented Sep 17, 2019

ursabot commented Sep 17, 2019

nealrichardson commented Sep 17, 2019

nealrichardson commented Sep 20, 2019

kszucs commented Sep 20, 2019

nealrichardson commented Sep 20, 2019

ursabot commented Sep 20, 2019

nealrichardson commented Sep 20, 2019

ursabot commented Sep 20, 2019

nealrichardson commented Sep 20, 2019

nealrichardson commented Sep 21, 2019

kou left a comment

Choose a reason for hiding this comment

kou commented Sep 12, 2019 •

edited

Loading

kou commented Sep 12, 2019 •

edited

Loading

kou commented Sep 12, 2019 •

edited

Loading