Docker builds fail silenty for build-centos_8.sh / ....

[laeubi]

If I run build-centos_8.sh locally I get the following output:

+ set -e
+ pushd centos-gtk3-metacity/8-gtk3
/opt/eclipse/platform-sdk/git/eclipse.platform.releng.aggregator/cje-production/dockerfiles/centos-gtk3-metacity/8-gtk3 /opt/eclipse/platform-sdk/git/eclipse.platform.releng.aggregator/cje-production/dockerfiles
+ echo 'Building Centos 8 docker image'
Building Centos 8 docker image
+ docker build --pull -t eclipse/platformreleng-centos-gtk3-metacity:8 .
[+] Building 1.6s (8/14)                                                                                                                                                                                                                                                                                                                    docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                  0.0s
 => => transferring dockerfile: 1.58kB                                                                                                                                                                                                                                                                                                                0.0s
 => [internal] load metadata for quay.io/centos/centos:stream8                                                                                                                                                                                                                                                                                        0.5s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                       0.0s
 => [ 1/10] FROM quay.io/centos/centos:stream8@sha256:20da069d4f8126c4517ee563e6e723d4cbe79ff62f6c4597f753478af91a09a3                                                                                                                                                                                                                                0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                     0.0s
 => => transferring context: 683B                                                                                                                                                                                                                                                                                                                     0.0s
 => CACHED [ 2/10] COPY scripts/uid_entrypoint /usr/local/bin/uid_entrypoint                                                                                                                                                                                                                                                                          0.0s
 => CACHED [ 3/10] RUN chmod u+x /usr/local/bin/uid_entrypoint &&     chgrp 0 /usr/local/bin/uid_entrypoint &&     chmod g=u /usr/local/bin/uid_entrypoint /etc/passwd                                                                                                                                                                                0.0s
 => ERROR [ 4/10] RUN yum -y update && yum -y install       gtk3       tigervnc-server       tigervnc       metacity       xorg-x11-server-utils       mesa-libGL       xorg-x11-fonts-misc       xorg-x11-fonts-75dpi       xorg-x11-fonts-Type1       xorg-x11-fonts-100dpi       liberation-fonts       gnu-free-fonts-common       dejavu-fonts-  0.9s
------                                                                                                                                                                                                                                                                                                                                                     
 > [ 4/10] RUN yum -y update && yum -y install       gtk3       tigervnc-server       tigervnc       metacity       xorg-x11-server-utils       mesa-libGL       xorg-x11-fonts-misc       xorg-x11-fonts-75dpi       xorg-x11-fonts-Type1       xorg-x11-fonts-100dpi       liberation-fonts       gnu-free-fonts-common       dejavu-fonts-common       unzip       lsof       wget       curl       dbus       webkitgtk4       glibc-locale-source       glibc-langpack-en     && yum clean all:                                                                                                                                                                                                                  
0.690 CentOS Stream 8 - AppStream                     0.0  B/s |   0  B     00:00                                                                                                                                                                                                                                                                          
0.691 Errors during downloading metadata for repository 'appstream':
0.691   - Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppStream&infra=container [Could not resolve host: mirrorlist.centos.org]
0.699 Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppStream&infra=container [Could not resolve host: mirrorlist.centos.org]
------
Dockerfile:13
--------------------
  12 |     
  13 | >>> RUN yum -y update && yum -y install \
  14 | >>>       gtk3 \
  15 | >>>       tigervnc-server \
  16 | >>>       tigervnc \
  17 | >>>       metacity \
  18 | >>>       xorg-x11-server-utils \
  19 | >>>       mesa-libGL \
  20 | >>>       xorg-x11-fonts-misc \
  21 | >>>       xorg-x11-fonts-75dpi \
  22 | >>>       xorg-x11-fonts-Type1 \
  23 | >>>       xorg-x11-fonts-100dpi \
  24 | >>>       liberation-fonts \
  25 | >>>       gnu-free-fonts-common \
  26 | >>>       dejavu-fonts-common \
  27 | >>>       unzip \
  28 | >>>       lsof \
  29 | >>>       wget \
  30 | >>>       curl \
  31 | >>>       dbus \
  32 | >>>       webkitgtk4 \
  33 | >>>       glibc-locale-source \
  34 | >>>       glibc-langpack-en \
  35 | >>>     && yum clean all
  36 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c yum -y update && yum -y install       gtk3       tigervnc-server       tigervnc       metacity       xorg-x11-server-utils       mesa-libGL       xorg-x11-fonts-misc       xorg-x11-fonts-75dpi       xorg-x11-fonts-Type1       xorg-x11-fonts-100dpi       liberation-fonts       gnu-free-fonts-common       dejavu-fonts-common       unzip       lsof       wget       curl       dbus       webkitgtk4       glibc-locale-source       glibc-langpack-en     && yum clean all" did not complete successfully: exit code: 1

But the Jenkins build succeeds https://ci.eclipse.org/releng/view/Miscellaneous/job/Build-Docker-images/ and simply do not publish any updates anymore.

FYI @fredg02 @akurtakov

@sravanlakkimsetti I assigned this to you as you seem to have setup the build (and scripts) here.

There are two issues:

1. The build does not fail so one didn't notice the problem
2. the centos-8 images seem not building any longer

[laeubi]

It seems be cause by this: https://serverfault.com/questions/1161816/mirrorlist-centos-org-no-longer-resolve

To unblock the build I have now removed

                (./build-centos_8.sh|tee centos8.log)&
                (./build-centos_swt_build.sh|tee swt_build.log)&

from the config

[fredg02]

Yep, the Jenkins build should fail. And Centos-8 should be finally laid to rest. 😬

[laeubi]

And Centos-8 should be finally laid to rest. 😬

I restarted the build now to see if we can build centos-9 once thats done I'll try to upgrade SWT to use the centos-9 image to see if we can simply use that.

[laeubi]

@fredg02 as we are talking about does the 83 in the name means it is centos8.3?

I also wondering why we need this specific node for the build of Docker images, are there alternatives? I currently see a loot of free executors lingering around so why not use any other? No Docker installed?

[fredg02]

@fredg02 as we are talking about does the 83 in the name means it is centos8.3?

Yep. https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1018#note_665337 is a strong indication.

I also wondering why we need this specific node for the build of Docker images, are there alternatives? I currently see a loot of free executors lingering around so why not use any other? No Docker installed?

@sravanlakkimsetti should know more why only this agent is used for building docker images.

[HannesWell]

I also wondering why we need this specific node for the build of Docker images, are there alternatives? I currently see a loot of free executors lingering around so why not use any other? No Docker installed?

See also #2271, I think maybe no docker plus missing credentials?

If I run build-centos_8.sh locally I get the following output:

In the oldest build before you removed the old images I also see the following output that indicates that the build didn't succeed:

 An image does not exist locally with the tag: eclipse/platformreleng-centos-swt-build
 An image does not exist locally with the tag: eclipse/platformreleng-centos-gtk4-mutter
 The push refers to repository [docker.io/eclipse/platformreleng-centos-swt-build]
 The push refers to repository [docker.io/eclipse/platformreleng-centos-gtk4-mutter]
 An image does not exist locally with the tag: eclipse/platformreleng-centos-gtk3-metacity
 An image does not exist locally with the tag: eclipse/platformreleng-centos-swt-build
 The push refers to repository [docker.io/eclipse/platformreleng-centos-gtk3-metacity]
 The push refers to repository [docker.io/eclipse/platformreleng-opensuse-gtk3-metacity]
 The push refers to repository [docker.io/eclipse/platformreleng-ubuntu-gtk3-metacity]
 The push refers to repository [docker.io/eclipse/platformreleng-centos-swt-build]

1. The build does not fail so one didn't notice the problem

I assume (but I'm not a Unix bash expert) that's because the builds are executed in parallel and awaited using

( <command 1>)&
( <command 2>)&
wait

?

[HannesWell]

It seems be cause by this: https://serverfault.com/questions/1161816/mirrorlist-centos-org-no-longer-resolve

To unblock the build I have now removed

                (./build-centos_8.sh|tee centos8.log)&
                (./build-centos_swt_build.sh|tee swt_build.log)&

from the config

Btw. I think you missed to remove the 'push' part, but with #2271 I'm cleaning the entire Docker image build.

[HannesWell]

I also wondering why we need this specific node for the build of Docker images, are there alternatives? I currently see a loot of free executors lingering around so why not use any other? No Docker installed?

See also #2271, I think maybe no docker plus missing credentials?

Would it be possible to combine the standard template used by default in this project's repo centos-latest-8gb with the docker docker-image as nested template?
https://plugins.jenkins.io/kubernetes/#plugin-content-nesting-pod-templates

I assume/guess that you cannot combine the images/templates arbitrarily, can you?

[akurtakov]

Should this one be closed now?

[HannesWell]

There are two issues:

1. The build does not fail so one didn't notice the problem
2. the centos-8 images seem not building any longer

The second issue is definitively addressed now, but I don't know if the build now fails if something is missing. At least as far as I can say there is no option set that suppresses any warnings and a fail of the docker build/push should fail the pipeline.

[HannesWell]

Let's consider this completed since it seems we cannot do more.
Maybe using Jenkins parallel pipeline steps in #2271 instead of parallel execution in bash has already helped to let the Jenkins build fail on docker image builds.

If we encounter more trouble in the future we might further adapt then.