[Security] Use ubuntu:20.04 base image for Pulsar docker images #11026

lhotari · 2021-06-22T21:42:51Z

Motivation

Pulsar docker images currently use openjdk:11-jdk-slim as the base image.
openjdk:11-jdk-slim/openjdk:11-jdk images are based on Debian 10 which contains a lot of unfixed vulnerabilities.
- this causes the Pulsar docker images to get flagged in Docker image vulnerability
  scanning with docker image vulnerability scanning tools such as Clair

Modifications

Switch the base image to ubuntu:20.04. Install OpenJDK with apt-get.
Ubuntu fixes all critical and high severity security vulnerabilities. New security vulnerabilities get quickly patched in Ubuntu.
Python version is 3.8 in adoptopenjdk:11-jdk base image and that's why the switch requires changes to the Python & Pulsar Python client installation.

eolivelli

LGTM

I believe that it is super important to provide up-to-date base docker images for enterprise users.

It is good to commit this patch early at the beginning of 2.9 release cycle, this way we can get feedback from people who are testing from master branch

eolivelli · 2021-06-23T15:50:01Z

@michaeljmarshall @sijie @rdhabalia @jerrypeng @codelipenghui
PTAL

michaeljmarshall

LGTM. This is great change, and it'll decrease the overall size of our pulsar docker images by a good bit. I just pulled these images and here is the basic docker images info:

REPOSITORY TAG IMAGE ID CREATED SIZE
openjdk 11-jdk bd97a328b384 8 hours ago 647MB
adoptopenjdk 11-jdk 20d606fe6719 5 days ago 437MB

sijie

who maintains adoptopenjdk?

lhotari · 2021-06-24T06:19:31Z

who maintains adoptopenjdk?

The JDK, AdoptOpenJDK itself has moved to Eclipse Foundation and has been renamed to Adoptium. The GitHub organization is adoptium and the transition from AdoptOpenJDK to Adoptium is still in progress.

The adoptopenjdk docker images are maintained in 2 locations:

official Docker Hub adoptopenjdk docker images, source: https://github.com/AdoptOpenJDK/openjdk-docker . Dockerfile
adoptopenjdk docker images under adoptopenjdk/

Both images use the same underlying Java binaries from AdoptOpenJDK, but they support different OS flavours as below. Also the official DockerHub images are updated for underlying OS updates as and when the OS fixes are available. The AdoptOpenJDK maintained images are rebuilt everyday.

lhotari · 2021-06-24T06:28:32Z

Another choice for the adoptopenjdk base image would be to use the official ubuntu:20.04 base image and install the openjdk-11-jdk package in the container. This would also resolve the security issues that come from using the Debian based openjdk docker base image.

Switching to other than Ubuntu based OS would be more work since current Dockerfiles use apt to install packages and thus work for Debian/Ubuntu.

@sijie Please provide advice how to proceed.

sijie · 2021-07-12T23:25:03Z

Another choice for the adoptopenjdk base image would be to use the official ubuntu:20.04 base image and install the openjdk-11-jdk package in the container.

I think this is a better approach.

- Ubuntu fixes critical and high security vulnerabilities. - openjdk:11-jdk-slim/openjdk:11-jdk images are based on Debian 10 which contains a lot of unfixed vulnerabilities. - this causes the Pulsar docker images to get flagged in Docker image vulnerability scanning with docker image vulnerability scanning tools such as Clair

…for input

- JAVA_HOME set by ENV isn't available for RUN commands

eolivelli

Lgtm

* [Security] Use ubuntu:20.04 base image for Pulsar docker images - Ubuntu fixes critical and high security vulnerabilities. - openjdk:11-jdk-slim/openjdk:11-jdk images are based on Debian 10 which contains a lot of unfixed vulnerabilities. - this causes the Pulsar docker images to get flagged in Docker image vulnerability scanning with docker image vulnerability scanning tools such as Clair * Install Ubuntu updates * Set DEBIAN_FRONTEND=noninteractive so that dist-upgrade doesn't wait for input * Set JAVA_HOME * Fix configuring networkaddress.cache.ttl - JAVA_HOME set by ENV isn't available for RUN commands * Configure networkaddress.cache.ttl after installing OpenJDK (cherry picked from commit f989512)

hangc0276 · 2021-08-27T04:32:49Z

Due to python3.7 is not available in openjdk:11-jdk image, this build exception is: https://github.com/apache/pulsar/runs/3428234664

So i cherry picked this PR to branch-2.8, and release in Pulsar 2.8.1 to solve docker image build failed problem.

eolivelli · 2021-08-27T05:26:10Z

@hangc0276 please advertise this on dev@

This is an important change, we must be sure that the community is up to date

hangc0276 · 2021-08-27T07:35:38Z

@hangc0276 please advertise this on dev@

This is an important change, we must be sure that the community is up to date

@eolivelli Ok, I have sent an email to dev@ mail list, Thanks.

### Motivation When build the pulsar-standalone docker image, it throw the following exception [INFO] + sudo -u postgres /usr/lib/postgresql/11/bin/initdb /data/ [INFO] [INFO] sudo: /usr/lib/postgresql/11/bin/initdb: command not found [INFO] [ERROR] The command '/bin/sh -c /pulsar/django/init-postgres.sh' returned a non-zero code: 1 [WARNING] An attempt failed, will retry 1 more times org.apache.maven.plugin.MojoExecutionException: Could not build image at com.spotify.plugin.dockerfile.BuildMojo.buildImage (BuildMojo.java:247) at com.spotify.plugin.dockerfile.BuildMojo.execute (BuildMojo.java:135) at com.spotify.plugin.dockerfile.AbstractDockerMojo.tryExecute (AbstractDockerMojo.java:265) at com.spotify.plugin.dockerfile.AbstractDockerMojo.execute (AbstractDockerMojo.java:254) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) The root cause is when we use ubuntu 20.04 use base docker image instead of openjdk:11-jdk image introduce by #11026 , the ubuntu 20.04 will install postgresql 12 instead of postgresql 11 by default. However, the init and start script has been hard code with postgresql 11 install path, which will lead to command not found exception. ### Modification 1. add postgresql 11 resource address for ubuntu 20.04 and install postgresql 11

2. Build python client by Python3.8 cherry-pick: 1. apache#11026 2. apache#11623 3. apache#11862

### Motivation Currently, the docker is from `openjdk:8-jdk-slim `, it has a new release a few days ago and the Linux version was changed to `Impish`, it didn't support install `python3.7` by apt-get tool, the minimum Python version is 3.9, so we need to change the python version or use a different Linux release, I cherry-pick some commits from branch master to build docker image from the `ubuntu:20.04`. Mainly related PRs: 1. #11026 2. #11623 3. #11862 ### Modifications Build docker image from the `ubuntu:20.04`. upgrade pulsar-go-client from `0.2.0` to `0.6.0` and rever #9124 from branch-2.7

…he#11026) * [Security] Use ubuntu:20.04 base image for Pulsar docker images - Ubuntu fixes critical and high security vulnerabilities. - openjdk:11-jdk-slim/openjdk:11-jdk images are based on Debian 10 which contains a lot of unfixed vulnerabilities. - this causes the Pulsar docker images to get flagged in Docker image vulnerability scanning with docker image vulnerability scanning tools such as Clair * Install Ubuntu updates * Set DEBIAN_FRONTEND=noninteractive so that dist-upgrade doesn't wait for input * Set JAVA_HOME * Fix configuring networkaddress.cache.ttl - JAVA_HOME set by ENV isn't available for RUN commands * Configure networkaddress.cache.ttl after installing OpenJDK

Currently, the docker is from `openjdk:8-jdk-slim `, it has a new release a few days ago and the Linux version was changed to `Impish`, it didn't support install `python3.7` by apt-get tool, the minimum Python version is 3.9, so we need to change the python version or use a different Linux release, I cherry-pick some commits from branch master to build docker image from the `ubuntu:20.04`. Mainly related PRs: 1. apache#11026 2. apache#11623 3. apache#11862 Build docker image from the `ubuntu:20.04`. upgrade pulsar-go-client from `0.2.0` to `0.6.0` and rever apache#9124 from branch-2.7 (cherry picked from commit 259c698)

lhotari requested review from aahmed-se, sijie, merlimat, eolivelli and codelipenghui June 22, 2021 21:43

lhotari added the area/security label Jun 22, 2021

lhotari added this to the 2.9.0 milestone Jun 22, 2021

lhotari self-assigned this Jun 22, 2021

lhotari force-pushed the lh-change-docker-baseimage branch 2 times, most recently from 8af5b5b to 26ac6ea Compare June 23, 2021 11:39

eolivelli approved these changes Jun 23, 2021

View reviewed changes

michaeljmarshall approved these changes Jun 23, 2021

View reviewed changes

lhotari marked this pull request as draft June 23, 2021 17:51

lhotari force-pushed the lh-change-docker-baseimage branch from 26ac6ea to 38b9e1e Compare June 23, 2021 18:49

lhotari marked this pull request as ready for review June 23, 2021 18:49

sijie requested changes Jun 24, 2021

View reviewed changes

lhotari requested a review from sijie June 24, 2021 06:29

lhotari added 2 commits July 23, 2021 15:45

Install Ubuntu updates

b91b672

lhotari force-pushed the lh-change-docker-baseimage branch from 38b9e1e to b91b672 Compare July 23, 2021 12:54

lhotari changed the title ~~[Security] Use adoptopenjdk:11-jdk base image for Pulsar docker images~~ [Security] Use ubuntu:20.04 base image for Pulsar docker images Jul 23, 2021

Set DEBIAN_FRONTEND=noninteractive so that dist-upgrade doesn't wait …

70e02ab

…for input

lhotari force-pushed the lh-change-docker-baseimage branch from 43f4407 to 70e02ab Compare July 23, 2021 17:22

lhotari added 3 commits July 23, 2021 20:50

Set JAVA_HOME

10d9a24

Fix configuring networkaddress.cache.ttl

7ff11f9

- JAVA_HOME set by ENV isn't available for RUN commands

Configure networkaddress.cache.ttl after installing OpenJDK

c90e212

sijie approved these changes Jul 23, 2021

View reviewed changes

eolivelli approved these changes Jul 24, 2021

View reviewed changes

lhotari merged commit f989512 into apache:master Aug 2, 2021

timmyyuan mentioned this pull request Aug 10, 2021

[Build] Building the pulsar-standalone image requires user interaction #11620

Closed

sijie mentioned this pull request Aug 10, 2021

ISSUE-11620: [Build] Building the pulsar-standalone image requires user interaction streamnative/pulsar-archived#2892

Closed

hangc0276 added the release/2.8.1 label Aug 26, 2021

hangc0276 added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Aug 26, 2021

timmyyuan mentioned this pull request Aug 26, 2021

Compatible with pulsar-all based on ubuntu 20.04 streamnative/function-mesh#270

Closed

hangc0276 mentioned this pull request Aug 31, 2021

[Branch-2.8]Fix pulsar_standalone docker image build failed #11862

Merged

gaoran10 added a commit to gaoran10/pulsar that referenced this pull request Sep 10, 2021

1. Use ubuntu:20.04 base image for Pulsar docker images

ebc2b97

2. Build python client by Python3.8 cherry-pick: 1. apache#11026 2. apache#11623 3. apache#11862

This was referenced Sep 10, 2021

[WIP][Branch-2.7][Build] Use ubuntu:20.04 base image for Pulsar docker images #11992

Closed

[Branch-2.7][Build] Use ubuntu 20.04 as docker image base #12017

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security] Use ubuntu:20.04 base image for Pulsar docker images #11026

[Security] Use ubuntu:20.04 base image for Pulsar docker images #11026

lhotari commented Jun 22, 2021 •

edited

Loading

eolivelli left a comment

eolivelli commented Jun 23, 2021

michaeljmarshall left a comment

sijie left a comment

lhotari commented Jun 24, 2021

lhotari commented Jun 24, 2021

sijie commented Jul 12, 2021

eolivelli left a comment

hangc0276 commented Aug 27, 2021

eolivelli commented Aug 27, 2021

hangc0276 commented Aug 27, 2021

[Security] Use ubuntu:20.04 base image for Pulsar docker images #11026

[Security] Use ubuntu:20.04 base image for Pulsar docker images #11026

Conversation

lhotari commented Jun 22, 2021 • edited Loading

Motivation

Modifications

eolivelli left a comment

Choose a reason for hiding this comment

eolivelli commented Jun 23, 2021

michaeljmarshall left a comment

Choose a reason for hiding this comment

sijie left a comment

Choose a reason for hiding this comment

lhotari commented Jun 24, 2021

lhotari commented Jun 24, 2021

sijie commented Jul 12, 2021

eolivelli left a comment

Choose a reason for hiding this comment

hangc0276 commented Aug 27, 2021

eolivelli commented Aug 27, 2021

hangc0276 commented Aug 27, 2021

lhotari commented Jun 22, 2021 •

edited

Loading