ARROW-6148: [Packaging] Improve aarch64 support #5024

hrw · 2019-08-06T12:47:59Z

Fetching Thrift fails without 'ca-certificates':

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
(_ssl.c:1056)

And build fails without 'meson' installed:

make[2]: Leaving directory '/build/apache-arrow-0.14.1/cpp_build'
dh_auto_configure                               \
  --sourcedirectory=c_glib                      \
  --builddirectory=c_glib_build                 \
  --buildsystem=meson+ninja                     \
  --                                            \
  -Darrow_cpp_build_type=release                \
  -Darrow_cpp_build_dir=/build/apache-arrow-0.14.1/cpp_build    \
  -Dgtk_doc=true
        cd c_glib_build && LC_ALL=C.UTF-8 meson ../c_glib --wrap-mode=nodownload --buildtype=plain --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=lib/aarch64-linux-gnu -Darrow_cpp_build_type=release -Darrow_cpp_build_dir=/build/apache-arrow-0.14.1/cpp_build -Dgtk_doc=true
Can't exec "meson": No such file or directory at /usr/share/perl5/Debian/Debhelper/Dh_Lib.pm line 475.

hrw · 2019-08-06T12:48:34Z

Also worth noting: there is 'clang-7' package on arm64. In Debian 'buster'.

fsaintjacques · 2019-08-06T13:21:15Z

I've created a Jira ticket at https://jira.apache.org/jira/browse/ARROW-6148, if you want to assign it to yourself, you'll get the resolution credits.

kou

We can't require clang-7 package because there isn't clang-7 package on Debian stretch arm64.

If we drop Debian stretch arm64 or we create dev/tasks/linux-packages/debian.debian-buster, we can require clang-7 package as a build dependency.

dev/tasks/linux-packages/debian/control

hrw · 2019-09-10T16:32:02Z

With those changes I am able to build packages for aarch64 Debian 'stretch' and Debian 'buster' (Ubuntu builds are in progress now).

./buster/main/a/apache-arrow:
apache-arrow_0.15.0~dev20190910-1.debian.tar.xz           libarrow-glib-doc_0.15.0~dev20190910-1_all.deb
apache-arrow_0.15.0~dev20190910-1.dsc                     libarrow-python15_0.15.0~dev20190910-1_arm64.deb
apache-arrow_0.15.0~dev20190910.orig.tar.gz               libarrow-python15-dbgsym_0.15.0~dev20190910-1_arm64.deb
gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb           libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb
gir1.2-gandiva-1.0_0.15.0~dev20190910-1_arm64.deb         libgandiva15_0.15.0~dev20190910-1_arm64.deb
gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb         libgandiva15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow15_0.15.0~dev20190910-1_arm64.deb                 libgandiva-dev_0.15.0~dev20190910-1_arm64.deb
libarrow15-dbgsym_0.15.0~dev20190910-1_arm64.deb          libgandiva-glib15_0.15.0~dev20190910-1_arm64.deb
libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb         libgandiva-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow-dataset15-dbgsym_0.15.0~dev20190910-1_arm64.deb  libgandiva-glib-dev_0.15.0~dev20190910-1_arm64.deb
libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb       libgandiva-glib-doc_0.15.0~dev20190910-1_arm64.deb
libarrow-dev_0.15.0~dev20190910-1_arm64.deb               libparquet15_0.15.0~dev20190910-1_arm64.deb
libarrow-flight15_0.15.0~dev20190910-1_arm64.deb          libparquet15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow-flight15-dbgsym_0.15.0~dev20190910-1_arm64.deb   libparquet-dev_0.15.0~dev20190910-1_arm64.deb
libarrow-flight-dev_0.15.0~dev20190910-1_arm64.deb        libparquet-glib15_0.15.0~dev20190910-1_arm64.deb
libarrow-glib15_0.15.0~dev20190910-1_arm64.deb            libparquet-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb     libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb
libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb          libparquet-glib-doc_0.15.0~dev20190910-1_all.deb


./stretch/main/a/apache-arrow:
apache-arrow_0.15.0~dev20190910-1.debian.tar.xz           libarrow-glib-doc_0.15.0~dev20190910-1_all.deb
apache-arrow_0.15.0~dev20190910-1.dsc                     libarrow-python15_0.15.0~dev20190910-1_arm64.deb
apache-arrow_0.15.0~dev20190910.orig.tar.gz               libarrow-python15-dbgsym_0.15.0~dev20190910-1_arm64.deb
gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb           libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb
gir1.2-gandiva-1.0_0.15.0~dev20190910-1_arm64.deb         libgandiva15_0.15.0~dev20190910-1_arm64.deb
gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb         libgandiva15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow15_0.15.0~dev20190910-1_arm64.deb                 libgandiva-dev_0.15.0~dev20190910-1_arm64.deb
libarrow15-dbgsym_0.15.0~dev20190910-1_arm64.deb          libgandiva-glib15_0.15.0~dev20190910-1_arm64.deb
libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb         libgandiva-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow-dataset15-dbgsym_0.15.0~dev20190910-1_arm64.deb  libgandiva-glib-dev_0.15.0~dev20190910-1_arm64.deb
libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb       libgandiva-glib-doc_0.15.0~dev20190910-1_arm64.deb
libarrow-dev_0.15.0~dev20190910-1_arm64.deb               libparquet15_0.15.0~dev20190910-1_arm64.deb
libarrow-flight15_0.15.0~dev20190910-1_arm64.deb          libparquet15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow-flight15-dbgsym_0.15.0~dev20190910-1_arm64.deb   libparquet-dev_0.15.0~dev20190910-1_arm64.deb
libarrow-flight-dev_0.15.0~dev20190910-1_arm64.deb        libparquet-glib15_0.15.0~dev20190910-1_arm64.deb
libarrow-glib15_0.15.0~dev20190910-1_arm64.deb            libparquet-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb
libarrow-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb     libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb
libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb          libparquet-glib-doc_0.15.0~dev20190910-1_all.deb

hrw · 2019-09-10T17:31:10Z

I have Xenial related update in test now - sync of debian.ubuntu-xenial/ with debian/ files.

hrw · 2019-09-11T11:11:48Z

With all those changes I can run "rake apt" on x86-64 and aarch64 systems and get packages for

Debian 'stretch', 'buster'
Ubuntu 'xenial', 'bionic', 'cosmic', 'disco'

CUDA/Plasma/Gandiva are enabled for x86 architectures in all those OSes. Gandiva gets enabled on non-x86 architectures for everything except 'xenial'.

'nvidia-cuda-toolkit' and 'clang-7' are installed if needed by additional check in respecting Dockerfile.

hrw · 2019-09-11T11:13:18Z

Potential change on top: enabling 'flight' on Xenial.

dev/tasks/linux-packages/debian.ubuntu-xenial/rules

kou

This pull request is for "Add missing build dependencies for Debian".
Adding support for arm64 is out of scope.

Could you create another JIRA issue and pull request for arm64 support?

dev/tasks/linux-packages/apt/build.sh

kou · 2019-09-12T02:15:49Z

dev/tasks/linux-packages/apt/debian-buster/Dockerfile

We use separated Dockerfile for arm64 because base image is different. (debian:buster and arm64v8/debian:buster.)
See dev/tasks/linux-pakcages/debian-stretch-arm64 for details.

You use 'debian-stretch-arm64' because you are crossbuilding for arm64 on x86-64. We use normal Dockerfiles when we build Arrow directly on arm64.

This change does not affect your x86-64 builds but helps us to build on arm64.

./debian/pool/buster/main/a/apache-arrow: apache-arrow_0.15.0~dev20190910-1.debian.tar.xz apache-arrow_0.15.0~dev20190910-1.dsc apache-arrow_0.15.0~dev20190910.orig.tar.gz gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-gandiva-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb libarrow15_0.15.0~dev20190910-1_arm64.deb libarrow15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb libarrow-dev_0.15.0~dev20190910-1_arm64.deb libarrow-flight15_0.15.0~dev20190910-1_arm64.deb libarrow-flight15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-flight-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib15_0.15.0~dev20190910-1_arm64.deb libarrow-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib-doc_0.15.0~dev20190910-1_all.deb libarrow-python15_0.15.0~dev20190910-1_arm64.deb libarrow-python15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb libgandiva15_0.15.0~dev20190910-1_arm64.deb libgandiva15-dbgsym_0.15.0~dev20190910-1_arm64.deb libgandiva-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib15_0.15.0~dev20190910-1_arm64.deb libgandiva-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-doc_0.15.0~dev20190910-1_all.deb libparquet15_0.15.0~dev20190910-1_arm64.deb libparquet15-dbgsym_0.15.0~dev20190910-1_arm64.deb libparquet-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib15_0.15.0~dev20190910-1_arm64.deb libparquet-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib-doc_0.15.0~dev20190910-1_all.deb ./debian/pool/stretch/main/a/apache-arrow: apache-arrow_0.15.0~dev20190910-1.debian.tar.xz apache-arrow_0.15.0~dev20190910-1.dsc apache-arrow_0.15.0~dev20190910.orig.tar.gz gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-gandiva-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb libarrow15_0.15.0~dev20190910-1_arm64.deb libarrow15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb libarrow-dev_0.15.0~dev20190910-1_arm64.deb libarrow-flight15_0.15.0~dev20190910-1_arm64.deb libarrow-flight15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-flight-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib15_0.15.0~dev20190910-1_arm64.deb libarrow-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib-doc_0.15.0~dev20190910-1_all.deb libarrow-python15_0.15.0~dev20190910-1_arm64.deb libarrow-python15-dbgsym_0.15.0~dev20190910-1_arm64.deb libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb libgandiva15_0.15.0~dev20190910-1_arm64.deb libgandiva15-dbgsym_0.15.0~dev20190910-1_arm64.deb libgandiva-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib15_0.15.0~dev20190910-1_arm64.deb libgandiva-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-doc_0.15.0~dev20190910-1_all.deb libparquet15_0.15.0~dev20190910-1_arm64.deb libparquet15-dbgsym_0.15.0~dev20190910-1_arm64.deb libparquet-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib15_0.15.0~dev20190910-1_arm64.deb libparquet-glib15-dbgsym_0.15.0~dev20190910-1_arm64.deb libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib-doc_0.15.0~dev20190910-1_all.deb ./ubuntu/pool/bionic/universe/a/apache-arrow: apache-arrow_0.15.0~dev20190910-1.debian.tar.xz apache-arrow_0.15.0~dev20190910-1.dsc apache-arrow_0.15.0~dev20190910.orig.tar.gz gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-gandiva-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb libarrow15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb libarrow-dev_0.15.0~dev20190910-1_arm64.deb libarrow-flight15_0.15.0~dev20190910-1_arm64.deb libarrow-flight-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib15_0.15.0~dev20190910-1_arm64.deb libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib-doc_0.15.0~dev20190910-1_all.deb libarrow-python15_0.15.0~dev20190910-1_arm64.deb libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb libgandiva15_0.15.0~dev20190910-1_arm64.deb libgandiva-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib15_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-doc_0.15.0~dev20190910-1_all.deb libparquet15_0.15.0~dev20190910-1_arm64.deb libparquet-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib15_0.15.0~dev20190910-1_arm64.deb libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib-doc_0.15.0~dev20190910-1_all.deb ./ubuntu/pool/cosmic/universe/a/apache-arrow: apache-arrow_0.15.0~dev20190910-1.debian.tar.xz apache-arrow_0.15.0~dev20190910-1.dsc apache-arrow_0.15.0~dev20190910.orig.tar.gz gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-gandiva-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb libarrow15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb libarrow-dev_0.15.0~dev20190910-1_arm64.deb libarrow-flight15_0.15.0~dev20190910-1_arm64.deb libarrow-flight-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib15_0.15.0~dev20190910-1_arm64.deb libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib-doc_0.15.0~dev20190910-1_all.deb libarrow-python15_0.15.0~dev20190910-1_arm64.deb libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb libgandiva15_0.15.0~dev20190910-1_arm64.deb libgandiva-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib15_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-doc_0.15.0~dev20190910-1_all.deb libparquet15_0.15.0~dev20190910-1_arm64.deb libparquet-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib15_0.15.0~dev20190910-1_arm64.deb libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib-doc_0.15.0~dev20190910-1_all.deb ./ubuntu/pool/disco/universe/a/apache-arrow: apache-arrow_0.15.0~dev20190910-1.debian.tar.xz apache-arrow_0.15.0~dev20190910-1.dsc apache-arrow_0.15.0~dev20190910.orig.tar.gz gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-gandiva-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb libarrow15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb libarrow-dev_0.15.0~dev20190910-1_arm64.deb libarrow-flight15_0.15.0~dev20190910-1_arm64.deb libarrow-flight-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib15_0.15.0~dev20190910-1_arm64.deb libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib-doc_0.15.0~dev20190910-1_all.deb libarrow-python15_0.15.0~dev20190910-1_arm64.deb libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb libgandiva15_0.15.0~dev20190910-1_arm64.deb libgandiva-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib15_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-dev_0.15.0~dev20190910-1_arm64.deb libgandiva-glib-doc_0.15.0~dev20190910-1_all.deb libparquet15_0.15.0~dev20190910-1_arm64.deb libparquet-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib15_0.15.0~dev20190910-1_arm64.deb libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib-doc_0.15.0~dev20190910-1_all.deb ./ubuntu/pool/xenial/universe/a/apache-arrow: apache-arrow_0.15.0~dev20190910-1.debian.tar.xz apache-arrow_0.15.0~dev20190910-1.dsc apache-arrow_0.15.0~dev20190910.orig.tar.gz gir1.2-arrow-1.0_0.15.0~dev20190910-1_arm64.deb gir1.2-parquet-1.0_0.15.0~dev20190910-1_arm64.deb libarrow15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset15_0.15.0~dev20190910-1_arm64.deb libarrow-dataset-dev_0.15.0~dev20190910-1_arm64.deb libarrow-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib15_0.15.0~dev20190910-1_arm64.deb libarrow-glib-dev_0.15.0~dev20190910-1_arm64.deb libarrow-glib-doc_0.15.0~dev20190910-1_all.deb libarrow-python15_0.15.0~dev20190910-1_arm64.deb libarrow-python-dev_0.15.0~dev20190910-1_arm64.deb libparquet15_0.15.0~dev20190910-1_arm64.deb libparquet-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib15_0.15.0~dev20190910-1_arm64.deb libparquet-glib-dev_0.15.0~dev20190910-1_arm64.deb libparquet-glib-doc_0.15.0~dev20190910-1_all.deb

Can we use arm64v8/debian image on arm64 too?
Then we can use arm64v8/debian instead of debian for arm64.

If we use arm64v8/debian, we can use the same Dockerfile on amd64 and arm64. Then we can build .deb files for arm64 on Azure Pipelines.

We can:

08:03 (1s) hrw@j13-qrep-04:repositories$ uname -a Linux j13-qrep-04 4.19.0-5-arm64 #1 SMP Debian 4.19.37-5 (2019-06-19) aarch64 GNU/Linux 08:03 (1s) hrw@j13-qrep-04:repositories$ docker run --rm -it -u root arm64v8/debian:buster Unable to find image 'arm64v8/debian:buster' locally buster: Pulling from arm64v8/debian ecfa071771e7: Pull complete Digest: sha256:6b3715575add1050c27869d1c2ad907de4d1ab9fa3c01cf901f074181ef1071a Status: Downloaded newer image for arm64v8/debian:buster root@74363d13a449:/# uname -a Linux 74363d13a449 4.19.0-5-arm64 #1 SMP Debian 4.19.37-5 (2019-06-19) aarch64 GNU/Linux root@74363d13a449:/# exit 08:03 (5s) hrw@j13-qrep-04:repositories$

The thing is that when you do native builds do you not need special 'debian.stretch.arm64' thing.

hrw · 2019-09-12T06:14:43Z

This pull request is for "Add missing build dependencies for Debian".
Adding support for arm64 is out of scope.

Could you create another JIRA issue and pull request for arm64 support?

As it turned out that there are no missing build dependencies can we just rename both pull request and Jira card to "add support for arm64 builds"?

kou · 2019-09-12T08:02:57Z

As it turned out that there are no missing build dependencies can we just rename both pull request and Jira card to "add support for arm64 builds"?

Really? I think that we need to add ca-certificates to debian/control.

hrw · 2019-09-12T08:13:11Z

As it turned out that there are no missing build dependencies can we just rename both pull request and Jira card to "add support for arm64 builds"?

Really? I think that we need to add ca-certificates to debian/control.

I did builds for 2 Debian releases and 4 Ubuntu releases on both amd64 and arm64 without a need to add 'ca-certificates' to build dependencies. Package was fetched as dependency of 'python3-pip' (disco, cosmic, bionic, buster, stretch) or 'libwww-perl' (xenial).

kou · 2019-09-12T08:21:57Z

OK. Then we can rename existing pull request and JIRA issue and reuse them.

hrw · 2019-09-13T06:25:06Z

So, what's next?

kou · 2019-09-13T06:40:13Z

Create dev/tasks/linux-packages/apt/*-arm64/Dockerfile
Add entries for arm64 to dev/tasks/tasks.yml
Test package building by @ursabot crossbow package -g linux (e.g. ARROW-6440: [Packaging][deb] Follow plasma-store-server name change #5284 (comment) )

hrw · 2019-09-13T07:04:38Z

1. Create `dev/tasks/linux-packages/apt/*-arm64/Dockerfile`

Would be nice to find a way to not duplicate them as the only change is "FROM". Will look at using "--build-arg" in 'docker build' calls to pass 'FROM'.

Too bad that Docker lacks 'include' ;(

hrw · 2019-09-13T07:21:39Z

My idea is to not create "*-arm64/Dockerfile" but rather pass "ARCH=arm64" to "rake apt" build and then in 'task-package.rb' add proper argument to Docker to change "FROM".

Will take some time as I have to learn basics of Ruby.

kou · 2019-09-17T20:46:21Z

OK. I'll take over this.

Gandiva and Plasma can be enabled separately so let use proper switch.

Debian 'stretch' got LLVM-7 in August 2019. Only Ubuntu 'xenial' lacks it now.

'nvidia-cuda-toolkit' is x86 only so install it only on x86 architectures. Also handle 'clang-7' for Ubuntu 'xenial'.

Documentation is architecture independent.

Mark CUDA, Plasma and Gandiva as x86 specific due to lack of LLVM7 for other architectures.

hrw · 2019-09-18T07:55:37Z

Thanks! Nice and clean solution :D

kszucs · 2019-09-18T11:03:21Z

@kou yes, tasks.yml and tests.yml were merged, so I've updated the Buildbot comment bot to closely align with crossbow's CLI.

kszucs · 2019-09-18T11:06:41Z

@kou for building docker images more conveniently we could use the docker build tool from ursabot.

kou · 2019-09-18T21:06:08Z

@kszucs Thanks for verifying ursabot usage! I understand.

In this case, we didn't need the ursabot's Docker related feature. Because we can use a Docker's feature to change FROM image dynamically. Thanks for the information.

kou · 2019-09-18T21:06:24Z

@ursabot crossbow submit -g linux

ursabot · 2019-09-18T21:06:45Z

AMD64 Conda Crossbow Submit (#63462) builder has been succeeded.

Revision: eddc53f

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-202

Task	Status
centos-6
centos-7
debian-buster
debian-buster-arm64
debian-stretch
debian-stretch-arm64
ubuntu-bionic
ubuntu-bionic-arm64
ubuntu-disco
ubuntu-disco-arm64
ubuntu-xenial
ubuntu-xenial-arm64

kou · 2019-09-18T21:11:37Z

I've finished all work. I'll merge this once all packages can be built.

kou · 2019-09-18T21:11:51Z

@ursabot crossbow submit --help

ursabot · 2019-09-18T21:11:52Z

Usage: @ursabot crossbow submit [OPTIONS] [TASK]...

  Submit crossbow testing tasks.

  See groups defined in arrow/dev/tasks/tests.yml

Options:
  -g, --group [docker|integration|cpp-python|conda|wheel|linux|gandiva|nightly]
                                  Submit task groups as defined in tests.yml
  --help                          Show this message and exit.

kou · 2019-09-18T21:19:11Z

Concern: arm64.deb build takes almost 5h except debian-stretch-arm64. debian-strech-arm64 takes almost 3h. We may want to stop nightly arm64 build.

kou · 2019-09-18T21:26:03Z

@kszucs One more question. Crossbow's default package version (0.14.0.dev584) and rake apt:build/rake yum:build's default package version (0.15.0-dev20190918) are different. So crossbow status ursabot-201 doesn't work.

Can we specify package version from ursabot explicitly? We can specify package version explicitly on the local machine by ARROW_VERSION=0.15.0-dev20190918 crossbow ... --arrow-version=0.15.0-dev20190918.

hrw · 2019-09-19T06:34:31Z

Concern: arm64.deb build takes almost 5h except debian-stretch-arm64. debian-strech-arm64 takes almost 3h. We may want to stop nightly arm64 build.

In #5038 @wesm wrote that he works on adding arm64 machine into CI.

kou

+1
All green. I'll merge this.

hrw · 2019-09-19T08:03:10Z

Thanks!

kszucs · 2019-09-19T08:11:34Z

@kou sure, I've created an issue for that. We could also update crossbow to generate the version like rake does.

kou · 2019-09-19T08:18:00Z

@hrw Thanks for your work too!

kou · 2019-09-19T08:18:08Z

@kszucs Thanks!

kszucs · 2019-09-19T08:21:38Z

JIRA about unifying the version numbers: https://issues.apache.org/jira/browse/ARROW-6617

hrw · 2019-09-19T08:27:17Z

Now I can finish adding it to Linaro CI so our devs will get Debian packages :)

https://ci.linaro.org/job/ldcg-bigdata-apache-arrow/

fsaintjacques changed the title ~~Add missing build dependencies for Debian.~~ ARROW-6148: [Packaging] Add missing build dependencies for Debian. Aug 6, 2019

fsaintjacques requested a review from kou August 6, 2019 13:21

kou requested changes Aug 7, 2019

View reviewed changes

dev/tasks/linux-packages/debian/control Outdated Show resolved Hide resolved

kou changed the title ~~ARROW-6148: [Packaging] Add missing build dependencies for Debian.~~ ARROW-6148: [Packaging] Add missing build dependencies for Debian Aug 17, 2019

hrw requested a review from kou September 11, 2019 11:11

fsaintjacques reviewed Sep 11, 2019

View reviewed changes

dev/tasks/linux-packages/debian.ubuntu-xenial/rules Outdated Show resolved Hide resolved

kou reviewed Sep 12, 2019

View reviewed changes

hrw changed the title ~~ARROW-6148: [Packaging] Add missing build dependencies for Debian~~ ARROW-6148: [Packaging] Improve aarch64 support Sep 12, 2019

hrw and others added 7 commits September 18, 2019 06:09

deb: use proper switch to enable Plasma

64a7d4a

Gandiva and Plasma can be enabled separately so let use proper switch.

deb: enable clang-7 for arm64

12d9881

Debian 'stretch' got LLVM-7 in August 2019. Only Ubuntu 'xenial' lacks it now.

deb: enable Gandiva for more architectures

25b11f8

docker: handle installation of x86 only packages

aeaffef

'nvidia-cuda-toolkit' is x86 only so install it only on x86 architectures. Also handle 'clang-7' for Ubuntu 'xenial'.

deb: documentation packages should be 'all'

a0c9709

Documentation is architecture independent.

Sync Ubuntu 'xenial' packaging

abc522f

Mark CUDA, Plasma and Gandiva as x86 specific due to lack of LLVM7 for other architectures.

Share Dockerfile with amd64 and arm64

c949fcd

guyuqi mentioned this pull request Sep 18, 2019

ARROW-6131: [C++] Optimize the Arrow Non-Ascii-string-validation #5038

Closed

kou added 5 commits September 19, 2019 05:51

Update comment

2bc1644

Align continuous line marks

b3ec88b

Remove needless option

c321dab

Remove needless condition

3206128

Remove Ubuntu cosmic

eddc53f

kou approved these changes Sep 19, 2019

View reviewed changes

kou changed the title ~~ARROW-6148: [Packaging] Improve aarch64 support~~ ARROW-6148: [Packaging] Improve aarch64 support Sep 19, 2019

kou closed this in 606af08 Sep 19, 2019

kszucs mentioned this pull request Sep 19, 2019

Allow passing arrow version to crossbow command explicitly voltrondata-labs/ursabot#178

Open

kou mentioned this pull request Sep 26, 2019

ARROW-6703: [Packaging][Linux] Restore ARROW_VERSION environment variable #5506

Closed

asfimport mentioned this pull request Sep 19, 2019

[C++][Packaging] Improve aarch64 support #22542

Closed

ARROW-6148: [Packaging] Improve aarch64 support #5024

ARROW-6148: [Packaging] Improve aarch64 support #5024

Uh oh!

Conversation

hrw commented Aug 6, 2019 • edited by kou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hrw commented Aug 6, 2019

Uh oh!

fsaintjacques commented Aug 6, 2019

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hrw commented Sep 10, 2019

Uh oh!

hrw commented Sep 10, 2019

Uh oh!

hrw commented Sep 11, 2019

Uh oh!

hrw commented Sep 11, 2019

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kou Sep 12, 2019

Choose a reason for hiding this comment

Uh oh!

hrw Sep 12, 2019

Choose a reason for hiding this comment

Uh oh!

hrw Sep 12, 2019

Choose a reason for hiding this comment

Uh oh!

kou Sep 12, 2019

Choose a reason for hiding this comment

Uh oh!

hrw Sep 12, 2019

Choose a reason for hiding this comment

Uh oh!

hrw commented Sep 12, 2019

Uh oh!

kou commented Sep 12, 2019

Uh oh!

hrw commented Sep 12, 2019

Uh oh!

kou commented Sep 12, 2019

Uh oh!

hrw commented Sep 13, 2019

Uh oh!

kou commented Sep 13, 2019

Uh oh!

hrw commented Sep 13, 2019

Uh oh!

hrw commented Sep 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Sep 17, 2019

Uh oh!

hrw commented Sep 18, 2019

Uh oh!

kszucs commented Sep 18, 2019

Uh oh!

kszucs commented Sep 18, 2019

Uh oh!

kou commented Sep 18, 2019

Uh oh!

kou commented Sep 18, 2019

Uh oh!

ursabot commented Sep 18, 2019

Uh oh!

kou commented Sep 18, 2019

Uh oh!

kou commented Sep 18, 2019

Uh oh!

ursabot commented Sep 18, 2019

Uh oh!

hrw commented Aug 6, 2019 •

edited by kou

Loading

hrw commented Sep 13, 2019 •

edited

Loading

kszucs commented Sep 19, 2019 •

edited

Loading