ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649

kszucs · 2019-06-21T13:07:06Z

artifact uploading
osx build
win build (using appveyor, because of Azure: CMake does not find MSBuild options conda-forge/conda-forge.github.io#703)
linux build
package gandiva

kszucs · 2019-06-24T23:21:49Z

@ursabot crossbow package -g conda

ursabot · 2019-06-24T23:22:09Z

AMD64 Conda Crossbow (#24978) builder has been succeeded.

Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60

Task	Status
conda-linux-gcc-py27	unsupported CI service `azure`
conda-win-vs2015-py37
conda-win-vs2015-py36
conda-linux-gcc-py37	unsupported CI service `azure`
conda-osx-clang-py37	unsupported CI service `azure`
conda-osx-clang-py27	unsupported CI service `azure`
conda-osx-clang-py36	unsupported CI service `azure`
conda-linux-gcc-py36	unsupported CI service `azure`

kszucs · 2019-06-24T23:52:24Z

@ursabot crossbow package -g conda

ursabot · 2019-06-24T23:52:44Z

AMD64 Conda Crossbow (#24997) builder has been succeeded.

Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61

Task	Status
conda-win-vs2015-py36
conda-osx-clang-py36	unsupported CI service `azure`
conda-osx-clang-py27	unsupported CI service `azure`
conda-linux-gcc-py36	unsupported CI service `azure`
conda-osx-clang-py37	unsupported CI service `azure`
conda-linux-gcc-py37	unsupported CI service `azure`
conda-linux-gcc-py27	unsupported CI service `azure`
conda-win-vs2015-py37

kszucs · 2019-06-25T00:24:12Z

@ursabot crossbow package -g conda

ursabot · 2019-06-25T00:24:29Z

AMD64 Conda Crossbow (#25036) builder has been succeeded.

Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62

Task	Status
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37

kszucs · 2019-06-25T09:11:20Z

@ursabot crossbow package -g conda

ursabot · 2019-06-25T09:20:15Z

AMD64 Conda Crossbow (#25299) builder has been succeeded.

Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63

Task	Status
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37

kszucs · 2019-06-25T10:54:16Z

One of the azure build has lost connection, but I cannot restart it on the UI.

kszucs · 2019-06-25T10:54:20Z

@ursabot crossbow package -g conda

ursabot · 2019-06-25T10:54:40Z

AMD64 Conda Crossbow (#25411) builder has been succeeded.

Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65

Task	Status
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37

kszucs · 2019-06-25T11:26:05Z

@ursabot crossbow package -g conda

ursabot · 2019-06-25T11:26:26Z

AMD64 Conda Crossbow (#25445) builder has been succeeded.

Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67

Task	Status
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37

kszucs · 2019-06-25T12:47:41Z

@ursabot crossbow package -g conda

ursabot · 2019-06-25T12:47:57Z

AMD64 Conda Crossbow (#25501) builder has been succeeded.

Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69

Task	Status
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37

kszucs · 2019-06-25T13:50:24Z

The builds are passing, but the artifacts are not uploaded correctly.

kszucs · 2019-06-25T13:50:30Z

@ursabot crossbow package -g conda

ursabot · 2019-06-25T13:50:50Z

AMD64 Conda Crossbow (#25538) builder has been succeeded.

Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70

Task	Status
conda-linux-gcc-py27
conda-linux-gcc-py36
conda-linux-gcc-py37
conda-osx-clang-py27
conda-osx-clang-py36
conda-osx-clang-py37
conda-win-vs2015-py36
conda-win-vs2015-py37

pitrou

Not sure I have anything constructive to say about this, sorry. Some questions though...

dev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml

dev/tasks/conda-recipes/README.md

pitrou · 2019-06-25T14:55:02Z

dev/tasks/conda-recipes/appveyor.win.yml

@@ -1,20 +1,3 @@
-# Licensed to the Apache Software Foundation (ASF) under one


I don't really mind, but is there a reason why you removed the license headers?

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.

pitrou · 2019-06-25T14:57:28Z

dev/tasks/conda-recipes/arrow-cpp/build.sh

+
+# Include g++'s system headers
+if [ "$(uname)" == "Linux" ]; then
+  SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')


Ideally there would be clean way to write this...
btw, here I get:

$ CXX=g++ $ echo $(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';') -isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;

I'm open to any suggestions.

Unfortunately I have none, sorry.

dev/tasks/conda-recipes/azure.linux.yml

dev/tasks/docker-tests/azure.linux.yml

kszucs · 2019-06-25T16:09:45Z

@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37

ursabot · 2019-06-25T16:10:03Z

AMD64 Conda Crossbow (#25665) builder has been succeeded.

Revision: 20f6cec

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73

Task	Status
conda-win-vs2015-py36
conda-win-vs2015-py37

kszucs · 2019-06-26T10:43:58Z

@pitrou the artifacts are properly uploaded to github releases now

pitrou · 2019-06-26T10:46:44Z

Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.

kszucs · 2019-06-26T10:56:34Z

It is more complicated then the report above.

python crossbow.py submit ... creates the branches and triggers the builds on various CIs
python crossbow.py status ... polls the github api about these statuses, checks and release assets

The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.

On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).

TLDR it is possible, but requires more time.

pitrou · 2019-06-26T11:25:55Z

I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...

kszucs · 2019-06-26T11:28:51Z

This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155

pitrou

+1. No objection from me.

kszucs added the WIP PR is work in progress label Jun 21, 2019

kszucs mentioned this pull request Jun 24, 2019

ARROW-5511: [Packaging] Enable Flight in Conda packages #4473

Closed

kszucs force-pushed the crossbow-azure branch from fab0884 to 14a15b6 Compare June 24, 2019 23:21

kszucs removed the WIP PR is work in progress label Jun 25, 2019

kszucs changed the title ~~[Crossbow] Port conda recipes to azure pipelines~~ ARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019

kszucs force-pushed the crossbow-azure branch from 2605fba to a3eefa1 Compare June 25, 2019 13:43

kszucs added 6 commits June 25, 2019 16:49

port conda recipes to azure pipelines

2037f78

use variables [skip ci]]

b2fd21a

don't use azure templates [skip ci]

53e8eb2

fix working directory [skip ci]

b2425e6

set config [skip ci]

501d553

trying to fix feedstock and recipe roots [skip ci]

df31ff7

kszucs added 15 commits June 25, 2019 16:49

docker shm [skip ci]

611222e

touch done_canary [skip ci]

d6c4ce9

use appveyor for the win packages; upload assets scripts [skip ci]

7fbce5d

git commit additional log [skip ci]

e0d8fb9

asset uploading script [skip ci]

750f624

azure template for docker tests

2aa4977

use pip3

9db3db1

tabulate win template

2a59894

use pip module for installing dependencies

f75efae

use the default python on osx

8f58839

rat

cbb9c9c

use FETCH_HEAD in the CI templates

95cb442

fix artifact patterns

1119649

combine status api and checks api

68d88a8

configure output folder for the artifacts

b6a8607

kszucs force-pushed the crossbow-azure branch from be7037a to b6a8607 Compare June 25, 2019 14:50

pitrou reviewed Jun 25, 2019

View reviewed changes

kszucs added 2 commits June 25, 2019 17:10

readme fixes

0c470bb

update conda-win artifact patterns

20f6cec

pitrou approved these changes Jun 26, 2019

View reviewed changes

pitrou closed this in ebff606 Jun 26, 2019

jorisvandenbossche mentioned this pull request Apr 8, 2021

ARROW-12246: [CI] Sync conda recipes with upstream feedstock #9923

Closed

asfimport mentioned this pull request Jun 26, 2019

[Crossbow] Port conda recipes to azure pipelines #22152

Closed

		@@ -1,20 +1,3 @@
		# Licensed to the Apache Software Foundation (ASF) under one

ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649

ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649

Uh oh!

Conversation

kszucs commented Jun 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Jun 24, 2019

Uh oh!

ursabot commented Jun 24, 2019

Uh oh!

kszucs commented Jun 24, 2019

Uh oh!

ursabot commented Jun 24, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

ursabot commented Jun 25, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

ursabot commented Jun 25, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

ursabot commented Jun 25, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

ursabot commented Jun 25, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

ursabot commented Jun 25, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

ursabot commented Jun 25, 2019

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou Jun 25, 2019

Choose a reason for hiding this comment

Uh oh!

kszucs Jun 25, 2019

Choose a reason for hiding this comment

Uh oh!

pitrou Jun 25, 2019

Choose a reason for hiding this comment

Uh oh!

kszucs Jun 25, 2019

Choose a reason for hiding this comment

Uh oh!

pitrou Jun 26, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kszucs commented Jun 25, 2019

Uh oh!

ursabot commented Jun 25, 2019

Uh oh!

kszucs commented Jun 26, 2019

Uh oh!

pitrou commented Jun 26, 2019

Uh oh!

kszucs commented Jun 26, 2019

Uh oh!

pitrou commented Jun 26, 2019

Uh oh!

kszucs commented Jun 21, 2019 •

edited

Loading