Skip to content

Commit

Permalink
ARROW-4649: [C++/CI/R] Add nightly job that tests the homebrew formula
Browse files Browse the repository at this point in the history
This doesn't just test `brew install apache-arrow --HEAD` as the ticket suggested--it brings the Homebrew formula under our source control and tests against that. This will enable us to know when we need to update the Homebrew formula due to changes in dependencies or cmake configuration.

I've been testing out this approach on a different repository: https://travis-ci.org/nealrichardson/arrow-brew/builds/568531245

Closes #5360 from nealrichardson/homebrew-nightly and squashes the following commits:

4a50a37 <Neal Richardson> Shuffle again
ea3f9b5 <Neal Richardson> Move autobrew script again and fix URL to match regular homebrew
851ac6e <Neal Richardson> Sort hunks 💪
1d6512b <Neal Richardson> Move homebrew formulae inside dev/tasks. Update autobrew package version in release script
ff1489e <Neal Richardson> Use regular snapshot version for homebrew formula
79e9191 <Neal Richardson> Remove autoconf from homebrew formula because new jemalloc no longer requires it
8816279 <Neal Richardson> Fix autobrew audit check
b25ac6d <Neal Richardson> Fix licensing/:rat: for ci/apache-arrow.rb
89dea4f <Neal Richardson> Parametrize homebrew task and also test 'autobrew'
d61290a <Neal Richardson> Remove bottle shas
a133385 <Neal Richardson> Fix test (hopefully?)
6d00fe4 <Neal Richardson> Fix regular expression
34af2a9 <Neal Richardson> Add missing test assertion
40fc370 <Neal Richardson> Attempt to set the homebrew formula arrow version in the release script
ea4368e <Neal Richardson> Re-alphabetize the seds in 00-prepare.sh
262c241 <Neal Richardson> Rename homebrew task yml; add to LICENSE
163c67a <Neal Richardson> Add homebrew formula and crossbow nightly task

Authored-by: Neal Richardson <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
  • Loading branch information
nealrichardson authored and kou committed Sep 21, 2019
1 parent 13916dc commit b2785d3
Show file tree
Hide file tree
Showing 9 changed files with 213 additions and 26 deletions.
34 changes: 32 additions & 2 deletions LICENSE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1838,9 +1838,39 @@ for PyArrow. Ibis is released under the Apache License, Version 2.0.

This project includes code from the autobrew project.

* r/tools/autobrew and r/tools/apache-arrow.rb are based on code
from the autobrew project.
* r/tools/autobrew and dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb
are based on code from the autobrew project.

Copyright: Copyright (c) 2017 - 2019, Jeroen Ooms.
All rights reserved.
Homepage: https://github.com/jeroen/autobrew

--------------------------------------------------------------------------------

dev/tasks/homebrew-formulae/apache-arrow.rb has the following license:

BSD 2-Clause License

Copyright (c) 2009-present, Homebrew contributors
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28 changes: 28 additions & 0 deletions dev/release/00-prepare-test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,20 @@ def test_update_version_pre_tag
"+ <Version>#{@release_version}</Version>"],
],
},
{
path: "dev/tasks/homebrew-formulae/apache-arrow.rb",
hunks: [
["- url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@snapshot_version}/apache-arrow-#{@snapshot_version}.tar.gz\"",
"+ url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@release_version}/apache-arrow-#{@release_version}.tar.gz\""],
],
},
{
path: "dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb",
hunks: [
["- url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@previous_version}.9000/apache-arrow-#{@previous_version}.9000.tar.gz\"",
"+ url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@release_version}/apache-arrow-#{@release_version}.tar.gz\""],
],
},
{
path: "js/package.json",
hunks: [
Expand Down Expand Up @@ -262,6 +276,20 @@ def test_update_version_post_tag
"+ <Version>#{@next_snapshot_version}</Version>"],
],
},
{
path: "dev/tasks/homebrew-formulae/apache-arrow.rb",
hunks: [
["- url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@release_version}/apache-arrow-#{@release_version}.tar.gz\"",
"+ url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@next_snapshot_version}/apache-arrow-#{@next_snapshot_version}.tar.gz\""],
],
},
{
path: "dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb",
hunks: [
["- url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@release_version}/apache-arrow-#{@release_version}.tar.gz\"",
"+ url \"https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-#{@release_version}.9000/apache-arrow-#{@release_version}.9000.tar.gz\""],
],
},
{
path: "js/package.json",
hunks: [
Expand Down
48 changes: 29 additions & 19 deletions dev/release/00-prepare.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,6 @@ update_versions() {
;;
esac

cd "${SOURCE_DIR}/../../cpp"
sed -i.bak -E -e \
"s/^set\(ARROW_VERSION \".+\"\)/set(ARROW_VERSION \"${version}\")/" \
CMakeLists.txt
rm -f CMakeLists.txt.bak
git add CMakeLists.txt
cd -

cd "${SOURCE_DIR}/../../c_glib"
sed -i.bak -E -e \
"s/^m4_define\(\[arrow_glib_version\], .+\)/m4_define([arrow_glib_version], ${version})/" \
Expand All @@ -61,6 +53,22 @@ update_versions() {
git add configure.ac meson.build
cd -

cd "${SOURCE_DIR}/../../ci"
sed -i.bak -E -e \
"s/^pkgver=.+/pkgver=${r_version}/" \
PKGBUILD
rm -f PKGBUILD.bak
git add PKGBUILD
cd -

cd "${SOURCE_DIR}/../../cpp"
sed -i.bak -E -e \
"s/^set\(ARROW_VERSION \".+\"\)/set(ARROW_VERSION \"${version}\")/" \
CMakeLists.txt
rm -f CMakeLists.txt.bak
git add CMakeLists.txt
cd -

cd "${SOURCE_DIR}/../../csharp"
sed -i.bak -E -e \
"s/^ <Version>.+<\/Version>/ <Version>${version}<\/Version>/" \
Expand All @@ -69,6 +77,19 @@ update_versions() {
git add Directory.Build.props
cd -

cd "${SOURCE_DIR}/../../dev/tasks/homebrew-formulae"
sed -i.bak -E -e \
"s/arrow-[0-9.]+[0-9]+/arrow-${r_version}/g" \
autobrew/apache-arrow.rb
rm -f autobrew/apache-arrow.rb.bak
git add autobrew/apache-arrow.rb
sed -i.bak -E -e \
"s/arrow-[0-9.\-]+[0-9SNAPHOT]+/arrow-${version}/g" \
apache-arrow.rb
rm -f apache-arrow.rb.bak
git add apache-arrow.rb
cd -

cd "${SOURCE_DIR}/../../js"
sed -i.bak -E -e \
"s/^ \"version\": \".+\"/ \"version\": \"${version}\"/" \
Expand Down Expand Up @@ -99,17 +120,6 @@ update_versions() {
DESCRIPTION
rm -f DESCRIPTION.bak
git add DESCRIPTION
cd -

cd "${SOURCE_DIR}/../../ci"
sed -i.bak -E -e \
"s/^pkgver=.+/pkgver=${r_version}/" \
PKGBUILD
rm -f PKGBUILD.bak
git add PKGBUILD
cd -

cd "${SOURCE_DIR}/../../r"
if [ ${type} = "snapshot" ]; then
# Add a news entry for the new dev version
echo "dev"
Expand Down
1 change: 1 addition & 0 deletions dev/release/rat_exclude_files.txt
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ cpp/src/plasma/thirdparty/ae/config.h
cpp/src/plasma/thirdparty/ae/zmalloc.h
cpp/src/plasma/thirdparty/dlmalloc.c
dev/release/rat_exclude_files.txt
dev/tasks/homebrew-formulae/apache-arrow.rb
dev/tasks/linux-packages/debian.ubuntu-xenial/compat
dev/tasks/linux-packages/debian.ubuntu-xenial/control
dev/tasks/linux-packages/debian.ubuntu-xenial/gir1.2-arrow-1.0.install
Expand Down
57 changes: 57 additions & 0 deletions dev/tasks/homebrew-formulae/apache-arrow.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
class ApacheArrow < Formula
desc "Columnar in-memory analytics layer designed to accelerate big data"
homepage "https://arrow.apache.org/"
url "https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-0.15.0-SNAPSHOT/apache-arrow-0.15.0-SNAPSHOT.tar.gz"
sha256 "9948ddb6d4798b51552d0dca3252dd6e3a7d0f9702714fc6f5a1b59397ce1d28"
head "https://github.com/apache/arrow.git"

depends_on "cmake" => :build
depends_on "boost"
depends_on "brotli"
depends_on "double-conversion"
depends_on "flatbuffers"
depends_on "glog"
depends_on "grpc"
depends_on "lz4"
depends_on "numpy"
depends_on "openssl"
depends_on "protobuf"
depends_on "python"
depends_on "rapidjson"
depends_on "snappy"
depends_on "thrift"
depends_on "zstd"

def install
ENV.cxx11
args = %W[
-DARROW_FLIGHT=ON
-DARROW_ORC=ON
-DARROW_PARQUET=ON
-DARROW_PLASMA=ON
-DARROW_PROTOBUF_USE_SHARED=ON
-DARROW_PYTHON=ON
-DARROW_INSTALL_NAME_RPATH=OFF
-DPYTHON_EXECUTABLE=#{Formula["python"].bin/"python3"}
]

mkdir "build"
cd "build" do
system "cmake", "../cpp", *std_cmake_args, *args
system "make"
system "make", "install"
end
end

test do
(testpath/"test.cpp").write <<~EOS
#include "arrow/api.h"
int main(void) {
arrow::int64();
return 0;
}
EOS
system ENV.cxx, "test.cpp", "-std=c++11", "-I#{include}", "-L#{lib}", "-larrow", "-o", "test"
system "./test"
end
end
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
class ApacheArrow < Formula
desc "Columnar in-memory analytics layer designed to accelerate big data"
homepage "https://arrow.apache.org/"
url "https://archive.apache.org/dist/arrow/arrow-0.14.1/apache-arrow-0.14.1.tar.gz"
url "https://www.apache.org/dyn/closer.cgi?path=arrow/arrow-0.14.1.9000/apache-arrow-0.14.1.9000.tar.gz"
sha256 "9948ddb6d4798b51552d0dca3252dd6e3a7d0f9702714fc6f5a1b59397ce1d28"
head "https://github.com/apache/arrow.git"

Expand All @@ -31,11 +31,11 @@ class ApacheArrow < Formula

depends_on "cmake" => :build
depends_on "flatbuffers" => :build
depends_on "double-conversion"
depends_on "boost"
depends_on "double-conversion"
depends_on "lz4"
depends_on "thrift"
depends_on "snappy"
depends_on "thrift"

def install
ENV.cxx11
Expand Down
41 changes: 41 additions & 0 deletions dev/tasks/homebrew-formulae/travis.osx.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

os: osx
language: ruby
cache: bundler

# don't build twice
if: tag IS blank

env:
global:
- TRAVIS_TAG={{ task.tag }}
- ARROW_FORMULA=./arrow/dev/tasks/homebrew-formulae/{{ formula }}

before_script:
- git clone --no-checkout {{ arrow.remote }} arrow
- git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
- if [ $CROSSBOW_USE_COMMIT_ID == true ]; then git -C arrow checkout {{ arrow.head }}; else git -C arrow checkout FETCH_HEAD; fi
- brew update
- brew audit $ARROW_FORMULA
script:
- brew install -v --build-from-source --HEAD $ARROW_FORMULA
- brew test $ARROW_FORMULA

notifications:
email:
- {{ job.email }}
18 changes: 18 additions & 0 deletions dev/tasks/tasks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,8 @@ groups:
- ubuntu-disco-arm64
- centos-6
- centos-7
- homebrew-cpp
- homebrew-cpp-autobrew
- gandiva-jar-trusty
- gandiva-jar-osx
- docker-r
Expand Down Expand Up @@ -1085,6 +1087,22 @@ tasks:
- plasma-glib-libs-{no_rc_version}-1.el7.x86_64.rpm
- plasma-libs-{no_rc_version}-1.el7.x86_64.rpm

############################## Homebrew Tasks ################################

homebrew-cpp:
ci: travis
platform: osx
template: homebrew-formulae/travis.osx.yml
params:
formula: apache-arrow.rb

homebrew-cpp-autobrew:
ci: travis
platform: osx
template: homebrew-formulae/travis.osx.yml
params:
formula: autobrew/apache-arrow.rb

############################## Gandiva Tasks ################################

gandiva-jar-trusty:
Expand Down
6 changes: 4 additions & 2 deletions r/configure
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,12 @@ if [ "$ARROW_R_DEV" == "TRUE" ]; then
fi

if [ "$LOCAL_AUTOBREW" == "TRUE" ]; then
# LOCAL_AUTOBREW means use the formula in tools/
# LOCAL_AUTOBREW means use the script in tools/
# If you want to use a local apache-arrow.rb formula, do:
# $ cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb apache-arrow.rb
# (assuming a local checkout of the apache/arrow repository)
# FORCE_AUTOBREW without LOCAL_AUTOBREW means to pull from jeroen.github.io
cp tools/autobrew .
cp tools/apache-arrow.rb .
FORCE_AUTOBREW="TRUE"
fi

Expand Down

0 comments on commit b2785d3

Please sign in to comment.