[Build] Define Docker images build using Jenkin's Job-DSL

[HannesWell]

This just preserves the existing and currently manually maintained 'Build-Docker-images' job in a Jenkins Job-DSL script under version control.

Ideally the Docker images would be built as part of the master branch build of this repo. With that the images would be built as soon as one submits a change. And because I builds usually push a change each night, the images would updated regularly even without explicit change.
But this would probably require the corresponding credentials available in this repo. I also don't know if docker is available by default.

[HannesWell]

I have now added two more commits to clean-up and simplify the Docker image build:

• Remove the Centos-8 Docker images that cannot be built anymore and the associated build/push scripts (see Docker builds fail silenty for build-centos_8.sh / .... #2263)
• Don't send mails on success only on failure and only to the mailing-list and not specific people (the list of people can become outdated quickly and in fact is it already).
• Delete complete workspace content after the build
• Update to jdt.core git repository at Github
• Inline bash-scripts to build and push docker-images
  • Remove push of images that are not built (anymore)

@sravanlakkimsetti do you want to have a look at this?
@akurtakov and @laeubi do you want to review this as well? Can you tell if the bash syntax is valid? I'm not a bash expert and mainly on Windows.

In general it would of course be nice to make the entire pipeline dynamic to build all images that are in a specific folder, but I don't know an elegant way to define the image id, besides adding an extra file for it?
But for a first shot I think this is ok for a first shot.

As said before, ideally this would all be done as part of the main Jenkinsfile of the repo.

Additionally we could also think about either to merge the swt specific images into their 'main' image or to define the SWT images at the SWT git-rep so that it can be maintained with SWT build. If it is possible to build docker images from the 'main' Jenkinsfile, we could maybe even build the SWT specific image in the same run and just pull it later. Or is it possible to publish images 'locally'? Similar to a local Maven install?
Then eclipse-platform/eclipse.platform.swt#1332 could finally be completed.

[laeubi]

I think having an own job makes sense here, as building an publishing the images takes considerable time and space also it looks like the images might be used by other parties and on the other hand they do not change very often.

Docker images can of course also be used local, but then they must be local to the agent that creates the docker image (there are likely Jenkins plugins for such task available so shell scripts are not directly needed).

[laeubi]

Can't we use usual Jenkins clone tasks?

[HannesWell]

Because this primarily a job definition (using the Job DSL), we have to define the repository URL somewhere. Either in the job configuration itself (and then we can use the usual checkout scm) or at this location.

But since we have to checkout multiple repositories and I would like to have this at one location defined, I think it makes sense to keep it at this location.

[laeubi]

can we probabbly amake it seperate steps instead of one big?

[HannesWell]

I have not reworked this to use a scripted parallel pipeline that uses a method to avoid duplicated definition of the executed action.
This way there is an individual stage for each image.
I have also enabled all the default logging, so that one can check or download the logs from the Jenkins UI as usual.
Therefore I have disabled explicit writing and archiving of log files.

Adding or removing images is then basically a one-line change in the job configuration.

One can see the results for example in
https://ci.eclipse.org/releng/job/Build-Docker-images/745/pipeline-graph/
or in the sub-elements of
https://ci.eclipse.org/releng/job/Build-Docker-images/745/pipeline-console/?selected-node=16

[HannesWell]

As described above I have now reworked the pipeline to build the images in parallel at Jenkins level.

For testing I have applied the new pipeline in the existing job and it successfully built, see:

• https://ci.eclipse.org/releng/job/Build-Docker-images/745/

Besides what has been mention in #2271 (comment) I have now also simplified the cron trigger to run only once a week instead of every two or three days (at different times). The images only change seldomly and rebuilds are not necessary that often.
And if one does an explicit change one usually triggers a build manually anyways.

With that this is ready for submission for me.
So unless there are objections or unresolved remarks I plan to submit this tomorrow (European) evening.

[HannesWell]

With that this is ready for submission for me.
So unless there are objections or unresolved remarks I plan to submit this tomorrow (European) evening.

And since this will create a new job, I will delete the existing job once the new run ran successfully:
https://ci.eclipse.org/releng/job/Build-Docker-images/

[laeubi]

Why do we need a scripted pipeline can't we just use a declarative one here? it looks a little bit strange and I can't see a direct reason for this... The idea seems to create the job itself by a script but why don't we simply reference a Jenkinsfile directly from the repository?

[laeubi]

And since this will create a new job, I will delete the existing job once the new run ran successfully:
https://ci.eclipse.org/releng/job/Build-Docker-images/

I missed that comment but see above, why do we need a job that creates a job, I would simply create a regular jenkins file and then add a Job that reference it (either manually or with a script but I don't see a good reason for having it created by a script as it only complicates the setup?!?

[HannesWell]

Why do we need a scripted pipeline can't we just use a declarative one here? it looks a little bit strange and I can't see a direct reason for this...

We can use a declarative pipeline here, but we would then need to specified the docker-image folder for each parallel stage, because in a declarative pipeline you cannot have a parallel step/stage within a dir step.
Using a scripted pipeline allows this and thus make the definition much less verbose.

And since this will create a new job, I will delete the existing job once the new run ran successfully:
https://ci.eclipse.org/releng/job/Build-Docker-images/

I missed that comment but see above, why do we need a job that creates a job, I would simply create a regular jenkins file and then add a Job that reference it (either manually or with a script but I don't see a good reason for having it created by a script as it only complicates the setup?!?

I contemplated initially to use a dedicated jenkinsfile and define the job that it references it in SCM, but discarded the idea because I wanted to avoid to have two files for one job. And since only very few string had to be escaped I was fine with it. And I think we should definitively generate the job using the job-dsl, ideally all jobs would be generated (then we could configure the seed job to also delete removed elements automatically, but having all jobs in the JIPP defined in SCM would be a longer tearm goal).

However, after think about it more, the definition of the job itself is very small if we have a regular Jenkinsfile, so we can just add it to the FOLDER.groovy script and should have the best of both approaches.
In fact I already had the intention to move the I-build definition from the pure job-defintion to a dedicated I build file, which would avoid escaping, which is much more annoying there too and would allow to leverage branches to simplify the handling. And this could then use the same technique if the job-definition is similarly simple. But that's another issue.