Improve build process for arbitrary-users-patch'd images for Brew compatibility

[nickboldt]

Describe the bug

As an extender of Che 7, I need to be able to rebuild these images:

• quay.io/eclipse/che-java8-maven
• quay.io/eclipse/che-java11-maven

But there appears to be no documentation and no obvious Dockerfiles for how to build these images. I've checked all the eclipse/che-* repos in GH, and I can't find any sources for these. Yet some CI process is definitely producing them from somewhere, as the latest tag in Quay is only 19hrs old.

These are built from https://github.com/eclipse/che-devfile-registry/tree/master/arbitrary-users-patch but the process is not compatible with rhplg/brew builds.

Expected behavior

Need to produce a clear and concise description of how to build these, with links to the Dockerfiles.

See also #14545

[nickboldt]

Thanks to a pointer from @skabashnyuk I've found this is done in https://github.com/eclipse/che-devfile-registry/blob/master/arbitrary-users-patch/build_images.sh#L37 as part of a patch set that @amisevsk created to produce 12 new images based on upstream, but with support for arbitrary users (req'd by Openshift).

So... yeah. This is going to be an interesting exercise to turn into something we can build the Red Hat Way (in Brew, with Errata/Comet, etc.)

@amisevsk how do you feel about migrating these to use UBI8 instead of https://hub.docker.com/_/maven ? Also could we standardize on the same JDK version as used in these two images?

• https://hub.docker.com/r/eclipse/che-remote-plugin-runner-java11/tags?page=1&ordering=last_updated
• https://hub.docker.com/r/eclipse/che-remote-plugin-runner-java8/tags?page=1&ordering=last_updated

[nickboldt]

I've updated both

• https://quay.io/repository/eclipse/che-java8-maven?tab=info and
• https://quay.io/repository/eclipse/che-java11-maven?tab=info
  with notes about how to build/publish via CI.

Next, here's a fork of che-devfile-registry/arbitrary-users-patch which works with UBI8 and should also work in Brew (once maven tarball is pushed to dist-git):

https://github.com/nickboldt/containers/tree/master/ubi8-arbitrary-users-patch

For those already saying "UBI is too big!" let me show some file sizes from my locally built versions of the ubuntu/debian-based images vs. my UBI8 based ones:

# UBI8
quay.io/nickboldt/che-java8-maven         nightly             7efc1023a9ef        42 seconds ago       346MB
quay.io/nickboldt/che-java11-maven        nightly             8cb5846c5b31        About a minute ago   453MB

# UBI8 base image
registry.access.redhat.com/ubi8-minimal   8.0                 8c980b20fbaa        6 days ago          90MB

# ubuntu/debian patched images
quay.io/eclipse/che-java8-maven           nightly             07ba858f6acb        6 minutes ago        499MB
quay.io/eclipse/che-java11-maven          nightly             fefac20c7dcd        7 minutes ago        825MB

# ubuntu/debian base images
maven                                     3.6.1-jdk-8         4c81be38db66        5 weeks ago          499MB
maven                                     3.6.0-jdk-11        592298276402        5 months ago         825MB

Or looking at the sizes as show in in Quay:

Existing Ubuntu/Debian-based ones:

• 419.0M https://quay.io/repository/eclipse/che-java11-maven?tab=tags
• 218.3M https://quay.io/repository/eclipse/che-java8-maven?tab=tags

vs the UBI8 versions:

• 156.7M https://quay.io/repository/nickboldt/che-java11-maven?tab=tags
• 120.4M https://quay.io/repository/nickboldt/che-java8-maven?tab=tags

Additionally, I see these remote plugins are even bigger in my registry:

$➔ docker images | grep che-remote
eclipse/che-remote-plugin-runner-java11   latest              78e23bd41e1e        2 weeks ago         762MB
eclipse/che-remote-plugin-runner-java8    latest              7d2fd4c287af        2 weeks ago         587MB

So... what if we used the same image for both the remote plugin and the jdk-maven container? Could one image do both things (eg., one has a default entrypoint, and the other we override when loading the container) ?

If so that would:

a) reduce the amount of stuff being downloaded to start the first JDK container (JDK8: 499M + 587M, JDK11: 825M + 762M)

b) reduce the number of images to build / publish via CI, and maintain downstream as supported CRW images.

cc: @l0rd @benoitf @tsmaeder @amisevsk

[l0rd]

@nickboldt some good things here:

• we never documented the build process and that's cool that you have done it. We should probably to that in the devfile registry README.md as well though.
• we should definetely re-use the same image for both the remote plugin and the jdk-maven-container. But that would be possible only after Theia remote plug-in runtime in sidecar container improvement #13387 is addressed.

I have breifly reviewed https://github.com/nickboldt/containers/tree/master/ubi8-arbitrary-users-patch and there is some misunderstanding. The content of the current folder arbitrary-users-patch is designed to make any image look beautiful in Che. The images that we patch are ubuntu, alpine or ubi. Yes we already patch ubi images! So you should not update the scripts or the Dockerfile, you should add the maven ubi images to the file base_images. The Dockerfile to build such image should probably live in the che-dockerfiles repository.

As already mentioned on a call, one of the goal was to used some base images maintained by third party teams instead of doing it ourselves.

[tsmaeder]

we should definetely re-use the same image for both the remote plugin and the jdk-maven-container.

The image for the plugin container is determined by the plugin metadata. The image for the jdk-maven-container comes from the devfile. There are many devfiles, but only one plugin, so we would push the responsiblity for ensuring the plugin prerequisites are installed to the author of the devfile. Maybe I'm not understanding the ide here, but I don't think that is desirable.

[nickboldt]

I have breifly reviewed https://github.com/nickboldt/containers/tree/master/ubi8-arbitrary-users-patch and there is some misunderstanding. The content of the current folder arbitrary-users-patch is designed to make any image look beautiful in Che.

If by "beautiful" you mean "works in OpenShift" because Openshift uses a randomly assigned userid and this gets around that design choice... then yes, it's more beautiful this way. :)

The images that we patch are ubuntu, alpine or ubi. Yes we already patch ubi images! So you should not update the scripts or the Dockerfile, you should add the maven ubi images to the file base_images.

There is no such thing as a JDK 11 or 8 image based on UBI8 that includes Maven 3.6.

I've looked at https://hub.docker.com/_/maven and RHCC and I haven't found anything that uses Maven 3.6, but there are some with 3.5.4. Of those, I'm not sure the images are smaller, or OpenShift friendly. Here are some:

• JDK 8 + Maven 3.5.4 container (yum install java-1.8.0-openjdk-devel rh-maven35):

https://access.redhat.com/containers/?tab=docker-file#/registry.access.redhat.com/openshift4/ose-jenkins-agent-maven/images/v4.1.16-201909100604

• Or these two s2i images w/ JDK + maven 3.5:

https://access.redhat.com/containers/?tab=docker-file#/registry.access.redhat.com/openjdk/openjdk-8-rhel8/images/1.0-7
https://access.redhat.com/containers/?tab=docker-file#/registry.access.redhat.com/openjdk/openjdk-11-rhel8/images/1.0-8

** (both run as user 185, not an anonymous anyid user for Openshift compatibility)

• Or this, which includes EAP7.2 and also run as user 185:

https://access.redhat.com/containers/?tab=docker-file#/registry.access.redhat.com/jboss-eap-7/eap72-openjdk11-openshift-rhel8/images/1.2-4

The Dockerfile to build such image should probably live in the che-dockerfiles repository.

Ah, but che-dockerfiles is deprecated, is it not? And since these are all related to the devfile registry, having them built as part of the devfile release makes a certain amount of sense.

[l0rd]

@nickboldt

I'm not sure the images are smaller

We are not looking for the smallest image if we do not replace the community images (ubuntu/alpine/centos). We just need to add some more enterprise friendly images (ubi/rhel/centos) to the base_images list.

OpenShift friendly

We are not looking for OpenShift friendly image neither because the script build_image.sh makes that for us.

Ah, but che-dockerfiles is deprecated, is it not? And since these are all related to the devfile registry, having them built as part of the devfile release makes a certain amount of sense.

We wanted to deprecate the che-dockerfiles repository because we didn't want to maintain the sample stacks images but using the "official" community images. But, if for some reason, we cannot find a valid third party image we should still use che-dockerfile. And none of the base images is built as part of the devfile-registry CI, only the patched images are. Hence I would rather be consistent with that approach.

[l0rd]

@tsmaeder I mean that the mvn jdk8 sample stack image should match (or be based on) the java8 plugin image. And it should be the same for most of the samples stacks (python, php, golang, dotnet, nodejs...). Let's discuss that here.

[amisevsk]

My idea behind the arbitrary users patch is mainly that you can put whatever you want in there; it's container agnostic for the most part. I'm strongly against adding any custom image- or runtime-specific modifications there (e.g. installing/configuring maven), since that greatly increases the complexity.

The only thing that happens in the arb users patch is working around OpenShift limitations, and the entire point is that you can list any image you want and have it build a version that will work on OpenShift. If we require e.g. a ubi8 image with maven and jdk11, we need to build it elsewhere and use it as a base image to be patched.

[nickboldt]

Proposal for making this arbitrary user patching compatible with Brew process:

1. Migrate Arg injection script to dockerfile generator

• fork from devfile repo to crw/stacks/ folder
• update list of base images to be based on RHCC images
• apply patch & optionally rebuild/publish container (for upstream/local build)
• for downstream/Brew, don't trigger the build. instead we'll have...

2. a per-container Jenkinsfile to:

• run the above Dockerfile generator,
• update from RHCC (pull newer base image tags w/ CVE fixes),
• push to pkgs.devel,
• trigger build of stacks-* container in Brew

And we need:

3. a second orchestrator Jenkinsfile to:

• Build all stacks (run the above job x 12)

[nickboldt]

Since we're now reusing 7 stack images and only have 2 new side car images, the above process may instead be simplified to:

• update existing rebuild-all-rhpkg-container-builds job to include these new plugin images
• contribute rhel.Dockerfiles upstream
• add 2 sync jobs for openshift and kubernetes plugin sidecar containers to sync with rhel.Dockerfiles upstream

[nickboldt]

Out of time to contribute rhel.Dockerfiles upstream, and many depend on rhel8 (not ubi8) containers, so they won't be accepted due to the "registry.redhat.io is authenticated and can't be used" rule.

Therefore closing, partially complete.