ARROW-6686: [CI] Pull and push docker images to speed up the nightly builds #5485

kszucs · 2019-09-24T11:56:56Z

The images are available under https://hub.docker.com/u/ursalab with arrow prefix.

Also resolves https://issues.apache.org/jira/browse/ARROW-2602

… ci]

kszucs · 2019-09-24T12:01:12Z

@ursabot crossbow submit docker-cpp

ursabot · 2019-09-24T12:01:32Z

AMD64 Conda Crossbow Submit (#64889) builder has been succeeded.

Revision: 16ec7f9

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-218

Task	Status
docker-cpp

kszucs · 2019-09-24T12:43:49Z

@ursabot crossbow submit docker-cpp

ursabot · 2019-09-24T12:44:09Z

AMD64 Conda Crossbow Submit (#64890) builder has been succeeded.

Revision: 16ec7f9

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-219

Task	Status
docker-cpp

…_caching [skip ci]

kszucs · 2019-09-24T13:08:03Z

@ursabot crossbow submit docker-cpp

ursabot · 2019-09-24T13:08:24Z

AMD64 Conda Crossbow Submit (#64891) builder has been succeeded.

Revision: d7ab350

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-220

Task	Status
docker-cpp

kszucs · 2019-09-24T13:11:26Z

@ursabot crossbow submit docker-cpp

ursabot · 2019-09-24T13:11:45Z

AMD64 Conda Crossbow Submit (#64892) builder has been succeeded.

Revision: 4c7cee0

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-221

Task	Status
docker-cpp

kszucs · 2019-09-24T13:15:14Z

@ursabot crossbow submit docker-cpp

ursabot · 2019-09-24T13:15:34Z

AMD64 Conda Crossbow Submit (#64893) builder has been succeeded.

Revision: 9718b00

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-222

Task	Status
docker-cpp

kszucs · 2019-09-24T13:18:43Z

@ursabot crossbow submit docker-cpp

ursabot · 2019-09-24T13:21:58Z

AMD64 Conda Crossbow Submit (#64905) builder has been succeeded.

Revision: 60c40f5

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-223

Task	Status
docker-cpp

kszucs · 2019-09-24T13:24:47Z

@ursabot crossbow submit docker-cpp

ursabot · 2019-09-24T13:31:50Z

AMD64 Conda Crossbow Submit (#64918) builder has been succeeded.

Revision: a0b7fb1

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-224

Task	Status
docker-cpp

kszucs · 2019-09-24T13:35:45Z

@ursabot crossbow submit -g cpp-python

ursabot · 2019-09-24T13:48:21Z

AMD64 Conda Crossbow Submit (#64944) builder has been succeeded.

Revision: a0b7fb1

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-225

Task	Status
docker-cpp
docker-cpp-alpine
docker-cpp-cmake32
docker-python-2.7
docker-python-2.7-alpine
docker-python-2.7-nopandas
docker-python-3.6
docker-python-3.6-alpine
docker-python-3.6-nopandas
docker-python-3.7

kszucs · 2019-09-24T14:19:33Z

@ursabot crossbow submit -g cpp-python

ursabot · 2019-09-24T14:22:40Z

AMD64 Conda Crossbow Submit (#65003) builder has been succeeded.

Revision: 44f0585

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-226

Task	Status
docker-cpp
docker-cpp-alpine
docker-cpp-cmake32
docker-python-2.7
docker-python-2.7-alpine
docker-python-2.7-nopandas
docker-python-3.6
docker-python-3.6-alpine
docker-python-3.6-nopandas
docker-python-3.7

kszucs · 2019-09-24T14:31:49Z

@ursabot crossbow submit -g cpp-python

ursabot · 2019-09-24T14:34:23Z

AMD64 Conda Crossbow Submit (#65019) builder has been succeeded.

Revision: 36bcaa9

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-227

Task	Status
docker-cpp
docker-cpp-alpine
docker-cpp-cmake32
docker-python-2.7
docker-python-2.7-alpine
docker-python-2.7-nopandas
docker-python-3.6
docker-python-3.6-alpine
docker-python-3.6-nopandas
docker-python-3.7

codecov-io · 2019-09-24T15:17:52Z

Codecov Report

Merging #5485 into master will decrease coverage by 3.68%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5485      +/-   ##
==========================================
- Coverage   69.97%   66.29%   -3.69%     
==========================================
  Files         755      507     -248     
  Lines       89329    69997   -19332     
  Branches     1498        0    -1498     
==========================================
- Hits        62509    46402   -16107     
+ Misses      26455    23595    -2860     
+ Partials      365        0     -365

Impacted Files	Coverage Δ
cpp/src/arrow/filesystem/filesystem.cc	`49.7% <0%> (-21.13%)`	⬇️
cpp/src/arrow/filesystem/filesystem.h	`91.66% <0%> (-8.34%)`	⬇️
python/pyarrow/array.pxi	`79.11% <0%> (-0.13%)`	⬇️
r/src/recordbatch.cpp
go/arrow/math/uint64_amd64.go
r/R/filesystem.R
r/R/list.R
r/src/symbols.cpp
r/src/array_to_vector.cpp
go/arrow/ipc/file_reader.go
... and 252 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35b6d07...36bcaa9. Read the comment docs.

kszucs · 2019-09-24T17:51:39Z

@ursabot crossbow submit -g cpp-python

ursabot · 2019-09-24T17:57:48Z

AMD64 Conda Crossbow Submit (#65128) builder has been succeeded.

Revision: 36bcaa9

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-228

Task	Status
docker-cpp
docker-cpp-alpine
docker-cpp-cmake32
docker-python-2.7
docker-python-2.7-alpine
docker-python-2.7-nopandas
docker-python-3.6
docker-python-3.6-alpine
docker-python-3.6-nopandas
docker-python-3.7

kszucs · 2019-09-25T08:09:32Z

@ursabot crossbow submit docker-spark-integration

ursabot · 2019-09-25T08:09:57Z

AMD64 Conda Crossbow Submit (#65329) builder has been succeeded.

Revision: 36bcaa9

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-231

Task	Status
docker-spark-integration

kszucs · 2019-09-25T13:02:47Z

@ursabot crossbow submit -g cpp-python

ursabot · 2019-09-25T13:06:10Z

AMD64 Conda Crossbow Submit (#65487) builder has been succeeded.

Revision: 2f8564f

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-232

Task	Status
docker-cpp
docker-cpp-alpine
docker-cpp-cmake32
docker-python-2.7
docker-python-2.7-alpine
docker-python-2.7-nopandas
docker-python-3.6
docker-python-3.6-alpine
docker-python-3.6-nopandas
docker-python-3.7

kszucs · 2019-09-26T08:28:30Z

@nealrichardson for docker-cpp it took ~40m before and ~25 after. The build time of docker-python-3.6 has decreased from ~28m to 17m.

pitrou · 2019-09-26T09:29:59Z

@wesm

nealrichardson · 2019-09-26T16:13:22Z

@emkornfield FYI in case you want to merge this to make the builds faster

wesm · 2019-09-27T00:02:55Z

Looking

wesm

+1. I think this is OK as a stopgap. It would be good if we can make an Arrow developer org on Docker Hub so that we can provide blanket administration rights to other PMC members.

wesm · 2019-09-27T00:08:58Z

@kszucs @nealrichardson can you ensure that the process to modify these cached images is properly documented? I didn't see anything about that from scanning this.

Looks like there is a lint failure so I can't merge this yet

+pre-commit install
pre-commit installed at .git/hooks/pre-commit
+pre-commit run hadolint -a
[WARNING] Unexpected key(s) present on git://github.com/pre-commit/pre-commit-hooks: sha
[INFO] Initializing environment for local.
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
Docker Format............................................................Failed
hookid: hadolint

kszucs · 2019-09-27T12:53:02Z

Will fix it

github-actions · 2019-09-29T18:46:03Z

https://issues.apache.org/jira/browse/ARROW-6686

kszucs

+1

wesm · 2019-10-03T12:28:17Z

@kszucs do you want to let the Arrow commiters know about the "arrowdev" Docker Hub organization? We should add any committer that wants access

…builds The images are available under https://hub.docker.com/u/ursalab with arrow prefix. Also resolves https://issues.apache.org/jira/browse/ARROW-2602 Closes #5485 from kszucs/docker-push-pull and squashes the following commits: 086c7b3 <Krisztián Szűcs> hadolint f1b5956 <Krisztián Szűcs> use arrowdev org 2f8564f <Krisztián Szűcs> push only images built from arrow's master branch 36bcaa9 <Krisztián Szűcs> update dockerfiles 44f0585 <Krisztián Szűcs> ignore pull failures a0b7fb1 <Krisztián Szűcs> disable master only push for now 60c40f5 <Krisztián Szűcs> jinja 9718b00 <Krisztián Szűcs> quote 4c7cee0 <Krisztián Szűcs> try to fix circle yml d7ab350 <Krisztián Szűcs> increase no_output_timeout; use newer machine image with docker_layer_caching c6f71a9 <Krisztián Szűcs> push from master only 16ec7f9 <Krisztián Szűcs> pull and push docker images to dockerhub to speed up the builds Authored-by: Krisztián Szűcs <[email protected]> Signed-off-by: Krisztián Szűcs <[email protected]>

pull and push docker images to dockerhub to speed up the builds [skip…

16ec7f9

… ci]

push from master only

c6f71a9

increase no_output_timeout; use newer machine image with docker_layer…

d7ab350

…_caching [skip ci]

kszucs force-pushed the docker-push-pull branch from dace9d6 to d7ab350 Compare September 24, 2019 13:07

try to fix circle yml [skip ci]

4c7cee0

quote [skip ci]

9718b00

jinja [skip ci]

60c40f5

disable master only push for now [skip ci]

a0b7fb1

ignore pull failures [skip ci]

44f0585

update dockerfiles [skip ci]

36bcaa9

kszucs changed the title ~~[Testing] Pull and push docker images to speed up the nightly builds~~ ARROW-6686: [CI] Pull and push docker images to speed up the nightly builds Sep 25, 2019

kszucs requested a review from xhochy September 25, 2019 13:00

push only images built from arrow's master branch

2f8564f

kszucs mentioned this pull request Sep 25, 2019

ARROW-6429: [Integration] Adding patch to fix Spark compilation for Integrition tests #5465

Closed

wesm approved these changes Sep 27, 2019

View reviewed changes

kszucs added 2 commits September 27, 2019 15:22

use arrowdev org

f1b5956

hadolint

086c7b3

kszucs commented Oct 3, 2019

View reviewed changes

kszucs closed this in 1165cdb Oct 3, 2019

asfimport mentioned this pull request Nov 29, 2022

[CI] Pull and push docker images to speed up the nightly builds #23034

Closed

ARROW-6686: [CI] Pull and push docker images to speed up the nightly builds #5485

ARROW-6686: [CI] Pull and push docker images to speed up the nightly builds #5485

Uh oh!

Conversation

kszucs commented Sep 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

codecov-io commented Sep 24, 2019

Codecov Report

Uh oh!

kszucs commented Sep 24, 2019

Uh oh!

ursabot commented Sep 24, 2019

Uh oh!

kszucs commented Sep 25, 2019

Uh oh!

ursabot commented Sep 25, 2019

Uh oh!

kszucs commented Sep 25, 2019

Uh oh!

ursabot commented Sep 25, 2019

Uh oh!

kszucs commented Sep 26, 2019

Uh oh!

pitrou commented Sep 26, 2019

Uh oh!

nealrichardson commented Sep 26, 2019

Uh oh!

wesm commented Sep 27, 2019

Uh oh!

wesm left a comment

Choose a reason for hiding this comment

Uh oh!

wesm commented Sep 27, 2019

Uh oh!

kszucs commented Sep 27, 2019

Uh oh!

github-actions bot commented Sep 29, 2019

Uh oh!

kszucs left a comment

kszucs commented Sep 24, 2019 •

edited

Loading