ARROW-6477: [Packaging][Crossbow] Use Azure Pipelines to build linux packages #5296

kszucs · 2019-09-05T15:29:54Z

Also done some crossbow refactoring, need to update the wiki because the sign command has changed to download-artficats.

kszucs · 2019-09-05T15:41:02Z

@ursabot crossbow package centos-6

ursabot · 2019-09-05T15:41:24Z

AMD64 Conda Crossbow (#58985) builder has been succeeded.

Revision: 0fff4f9

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-144

Task	Status
centos-6
centos-7
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37
debian-buster
debian-stretch
gandiva-jar-osx
gandiva-jar-trusty
ubuntu-bionic
ubuntu-cosmic
ubuntu-disco
ubuntu-xenial
wheel-manylinux1-cp27m
wheel-manylinux1-cp27mu
wheel-manylinux1-cp35m
wheel-manylinux1-cp36m
wheel-manylinux1-cp37m
wheel-manylinux2010-cp27m
wheel-manylinux2010-cp27mu
wheel-manylinux2010-cp35m
wheel-manylinux2010-cp36m
wheel-manylinux2010-cp37m
wheel-osx-cp27m
wheel-osx-cp35m
wheel-osx-cp36m
wheel-osx-cp37m
wheel-win-cp35m
wheel-win-cp36m
wheel-win-cp37m

kszucs · 2019-09-05T16:03:50Z

@ursabot crossbow package centos-6

ursabot · 2019-09-05T16:04:09Z

AMD64 Conda Crossbow (#59004) builder has been succeeded.

Revision: c6ec845

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-145

Task	Status
centos-6

codecov-io · 2019-09-05T22:02:25Z

Codecov Report

❗ No coverage uploaded for pull request base (master@26d72f3). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #5296   +/-   ##
=========================================
  Coverage          ?    87.7%           
=========================================
  Files             ?     1036           
  Lines             ?   149741           
  Branches          ?     1437           
=========================================
  Hits              ?   131327           
  Misses            ?    18052           
  Partials          ?      362

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26d72f3...8d52aa4. Read the comment docs.

kszucs · 2019-09-06T07:35:28Z

@ursabot crossbow package centos-6

ursabot · 2019-09-06T07:35:47Z

AMD64 Conda Crossbow (#59298) builder has been succeeded.

Revision: a2bc9f7

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-146

Task	Status
centos-6

kszucs · 2019-09-06T07:53:22Z

@ursabot crossbow package centos-6

ursabot · 2019-09-06T07:53:41Z

AMD64 Conda Crossbow (#59299) builder has been succeeded.

Revision: 3e708b0

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-147

Task	Status
centos-6

kszucs · 2019-09-06T07:58:11Z

@ursabot crossbow package centos-6

ursabot · 2019-09-06T07:58:30Z

AMD64 Conda Crossbow (#59300) builder has been succeeded.

Revision: d99c920

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-148

Task	Status
centos-6

kszucs · 2019-09-06T10:34:37Z

@ursabot crossbow package centos-6

kszucs · 2019-09-06T14:27:10Z

@ursabot crossbow package -g conda -g linux

ursabot · 2019-09-06T14:27:30Z

AMD64 Conda Crossbow (#59394) builder has been succeeded.

Revision: a9c401d

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-156

Task	Status
centos-6
centos-7
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37
debian-buster
debian-stretch
ubuntu-bionic
ubuntu-cosmic
ubuntu-disco
ubuntu-xenial

kou

It seems that packager information isn't set correctly:

https://dev.azure.com/ursa-labs/crossbow/_build/results?buildId=818&view=logs&jobId=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&taskId=5b4cc83a-7bb0-5664-5bb1-588f7e4dc05b&lineStart=9389&lineEnd=9390&colStart=1&colEnd=1

E: apache-arrow changes: changed-by-address-malformed <>
E: apache-arrow changes: changed-by-name-missing <>
E: apache-arrow changes: changed-by-address-missing <>

I'll push a commit to fix this.

kou · 2019-09-06T21:14:10Z

dev/tasks/tasks.yml

-    ci: travis
+    ci: azure
    platform: linux
    template: linux-packages/travis.linux.arm64.yml


Can we enable this too?
We can't run this on Travis CI because it takes 50min+.

I'm trying this.

kou · 2019-09-06T21:30:46Z

@ursabot crossbow package debian-buster

ursabot · 2019-09-06T21:31:05Z

AMD64 Conda Crossbow (#59580) builder has been succeeded.

Revision: 69f7cd8

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-157

Task	Status
debian-buster

kou · 2019-09-06T21:39:33Z

@ursabot crossbow package debian-stretch-arm64

ursabot · 2019-09-06T21:39:53Z

AMD64 Conda Crossbow (#59581) builder has been succeeded.

Revision: 913c9f2

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-158

Task	Status
debian-stretch-arm64

kou · 2019-09-06T21:48:32Z

@ursabot crossbow package debian-stretch-arm64

ursabot · 2019-09-06T21:48:51Z

AMD64 Conda Crossbow (#59582) builder has been succeeded.

Revision: 6bb6a47

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-159

Task	Status
debian-stretch-arm64

kou · 2019-09-06T22:23:37Z

Packager information has been solved.

We can build it on Azure Pipelines. It takes 2h 30m+.

kou · 2019-09-07T05:04:38Z

Wow. We can build debian-stretch-arm64 on Azure Pipelines. (It takes 2h 38m.)

@kszucs Should we fix conda-osx-clang-py27 failure in this pull request?

kszucs · 2019-09-07T08:23:25Z

@kou thanks for updating that! Almost three hours sounds pretty bad, but we can improve that in the future.

We can deal with the conda-osx-clang-py27 failure in another PR.

kou · 2019-09-07T11:34:32Z

OK! I'll merge this once CI is green.

kszucs

+1, Thanks @kou!

This is a rebased version of #5319. Crossbow had a major refactoring in #5296 so the two implementation have diverged. Closes #5332 from kszucs/crossbow-report and squashes the following commits: ed62de4 <Krisztián Szűcs> unused import 76a3269 <Krisztián Szűcs> share output redirection; use pass_obj instead of pass_context 9a0069c <Krisztián Szűcs> start incrementing from 0 16015d2 <Krisztián Szűcs> merge tests.yaml with tasks.yaml 00f5766 <Krisztián Szűcs> latest-prefix 162a0fb <Krisztián Szűcs> polling 1a8790f <Wes McKinney> Add polling logic to wait for job completion 6aa2a7e <Wes McKinney> Test disabling stop_if_pending 3e4580a <Wes McKinney> Implement e-mail report for a Crossbow job Lead-authored-by: Krisztián Szűcs <[email protected]> Co-authored-by: Wes McKinney <[email protected]> Signed-off-by: Krisztián Szűcs <[email protected]>

This is a rebased version of apache#5319. Crossbow had a major refactoring in apache#5296 so the two implementation have diverged. Closes apache#5332 from kszucs/crossbow-report and squashes the following commits: ed62de4 <Krisztián Szűcs> unused import 76a3269 <Krisztián Szűcs> share output redirection; use pass_obj instead of pass_context 9a0069c <Krisztián Szűcs> start incrementing from 0 16015d2 <Krisztián Szűcs> merge tests.yaml with tasks.yaml 00f5766 <Krisztián Szűcs> latest-prefix 162a0fb <Krisztián Szűcs> polling 1a8790f <Wes McKinney> Add polling logic to wait for job completion 6aa2a7e <Wes McKinney> Test disabling stop_if_pending 3e4580a <Wes McKinney> Implement e-mail report for a Crossbow job Lead-authored-by: Krisztián Szűcs <[email protected]> Co-authored-by: Wes McKinney <[email protected]> Signed-off-by: Krisztián Szűcs <[email protected]>

kszucs added 3 commits September 5, 2019 17:29

crossbow gardening and azure for linux packages

45e7c23

set ci to azure [skip ci]

fa1860b

fix types

632ebef

kszucs changed the title ~~[Packaging][Crossbow] Use Pipelines to build linux packages~~ [Packaging][Crossbow] Use Azure Pipelines to build linux packages Sep 5, 2019

use ubuntu-latest [skip ci]

0fff4f9

default whitelists [skip ci]

c6ec845

apache deleted a comment from ursabot Sep 5, 2019

left print

551d98d

debug pahts

a2bc9f7

correct requirements' path

3e708b0

use conda-forge

d99c920

kszucs added 7 commits September 6, 2019 11:07

update conda

f8828fe

don't set default origin

ce4d923

raise for missing remote just before pushing

5ab7802

remote [skip ci]

740d148

branch detection [skip ci]

31469aa

pass queue to the template; force remote_url [skip ci]

851ffcf

emnable build step [skip ci]

56df799

kszucs changed the title ~~[Packaging][Crossbow] Use Azure Pipelines to build linux packages~~ ARROW-6477: [Packaging][Crossbow] Use Azure Pipelines to build linux packages Sep 6, 2019

newlines

a9c401d

kou mentioned this pull request Sep 6, 2019

ARROW-6440: [Packaging][deb] Follow plasma-store-server name change #5284

Closed

kou reviewed Sep 6, 2019

View reviewed changes

Get packager information from the last commit [skip ci]

69f7cd8

Add support for building deb for arm64 [skip ci]

913c9f2

Update qemu-user-static URL [skip ci]

6bb6a47

Enable debian-stretch-arm64 [skip ci]

7bfb9c2

We can build it on Azure Pipelines. It takes 2h 30m+.

Remove unused files

3a3bead

kou added 2 commits September 8, 2019 06:41

Add missing license header

e74faf2

Fix format

8d52aa4

kszucs commented Sep 8, 2019

View reviewed changes

kszucs closed this in 6ed87b1 Sep 8, 2019

This was referenced Sep 8, 2019

ARROW-6480: [Developer][Crossbow] Implement summary report e-mailer with polling logic #5319

Closed

ARROW-6480: [Crossbow] Summary report e-mailer with polling logic #5332

Closed

asfimport mentioned this pull request Sep 8, 2019

[Packaging][Crossbow] Use Azure Pipelines to build linux packages #22846

Closed

ARROW-6477: [Packaging][Crossbow] Use Azure Pipelines to build linux packages #5296

ARROW-6477: [Packaging][Crossbow] Use Azure Pipelines to build linux packages #5296

Uh oh!

Conversation

kszucs commented Sep 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Sep 5, 2019

Uh oh!

ursabot commented Sep 5, 2019

Uh oh!

kszucs commented Sep 5, 2019

Uh oh!

ursabot commented Sep 5, 2019

Uh oh!

codecov-io commented Sep 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kszucs commented Sep 6, 2019

Uh oh!

ursabot commented Sep 6, 2019

Uh oh!

kszucs commented Sep 6, 2019

Uh oh!

ursabot commented Sep 6, 2019

Uh oh!

kszucs commented Sep 6, 2019

Uh oh!

ursabot commented Sep 6, 2019

Uh oh!

kszucs commented Sep 6, 2019

Uh oh!

kszucs commented Sep 6, 2019

Uh oh!

ursabot commented Sep 6, 2019

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

kou Sep 6, 2019

Choose a reason for hiding this comment

Uh oh!

kou Sep 6, 2019

Choose a reason for hiding this comment

Uh oh!

kou commented Sep 6, 2019

Uh oh!

ursabot commented Sep 6, 2019

Uh oh!

kou commented Sep 6, 2019

Uh oh!

ursabot commented Sep 6, 2019

Uh oh!

kou commented Sep 6, 2019

Uh oh!

ursabot commented Sep 6, 2019

Uh oh!

kou commented Sep 6, 2019

Uh oh!

kou commented Sep 7, 2019

Uh oh!

kszucs commented Sep 7, 2019

Uh oh!

kou commented Sep 7, 2019

Uh oh!

kszucs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kszucs commented Sep 5, 2019 •

edited

Loading

codecov-io commented Sep 5, 2019 •

edited

Loading

kszucs left a comment •

edited

Loading