feat: podman support

[defanator]

Proposed changes

This change introduces an ability to use podman container management tool in addition to docker, which remains the default. Podman provides daemonless architecture and allows to run containers in unprivileged mode natively on Linux.

On systems with both docker and podman installed, the CONTAINER_CLITOOL environment variable can be used to choose a desired option.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING document
• If applicable, I have added tests that prove my fix is effective or that my feature works
• If applicable, I have checked that any relevant tests pass after adding my changes
• I have updated any relevant documentation (README.md)

Details on performed tests

Platform / versions, Docker:

Host - MacOS on M1 pro

% sw_vers 
ProductName:	macOS
ProductVersion:	12.6.1
BuildVersion:	21G217

% docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:18 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.15.0 (93002)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:41 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Platform / versions, podman:

Host - Fedora Linux (x86_64)

$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="36 (Cloud Edition)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Cloud Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
SUPPORT_END=2023-05-16
VARIANT="Cloud Edition"
VARIANT_ID=cloud

$ podman version
Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.18.7
Built:        Fri Nov 11 15:24:13 2022
OS/Arch:      linux/amd64

Targets tested with both docker and podman (rootless):

• make build-docker
• make build-benchmark-docker
• make build-txz-packager-docker
• make local-txz-package

Targets tested with podman (rootless) only:

• make performance-test (fails with docker on M1)

Untested targets:

• make test-docker-component (it's unclear from Makefile how to create proper environment)

Additional findings

https://github.com/containers/podman/issues/16969

[netlify]

✅ Deploy Preview for agent-public-docs canceled.

Name	Link
🔨 Latest commit	ca9c86b
🔍 Latest deploy log	https://app.netlify.com/sites/agent-public-docs/deploys/63d4bad47ec14d0009f5a012

[oliveromahony]

@PrabhatDixit and @mtbChef do we want to support other container runtimes in make besides docker (devtools)? I think by doing what is in this PR we would need to adjust the naming of make targets that @defanator modified to not use the word docker in them. If we want to do this in a separate effort, should we consider more than Podman? e.g. LXC, ContainerD/runc ?

[defanator]

TWIMC, tested container targets on Fedora 37 host with docker-ce:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:45 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:25 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="37 (Workstation Edition)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora Linux 37 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f37/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=37
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=37
SUPPORT_END=2023-11-14
VARIANT="Workstation Edition"
VARIANT_ID=workstation

$ uname -a
Linux t480.lan 6.0.16-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Dec 31 16:47:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Everything works just fine when docker is running without selinux support (default mode). If selinux is enabled (daemon.json contains {"selinux-enabled": true}), things are getting worse:

$ make build-txz-packager-docker
Building Local Packager
#1 [internal] load build definition from Dockerfile
#1 sha256:18a33db6857f6a9d9e6d74ceb29822e5ce145bde77f9cca5d64b09e64a28c40a
#1 transferring dockerfile: 98B done
#1 DONE 0.0s
[..]
#5 [base 2/3] RUN apt-get update &&     apt-get install -y make jq gnupg gnupg1 gpgv1 git aptly debsig-verify createrepo-c dnf rpm                        curl gettext-base make monkeysphere libtool unzip libssl-dev libbz2-dev libbsd-dev libarchive-dev liblzma-dev zlib1g-dev
#5 sha256:a99880e9b9825b6cc4c42366a491abb7c6f2ef08d27f5670c0672534455ea1d1
#5 0.244 Reading package lists...
#5 0.254 E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
#5 ERROR: executor failed running [/bin/sh -c apt-get update &&     apt-get install -y make jq gnupg gnupg1 gpgv1 git aptly debsig-verify createrepo-c dnf rpm                        curl gettext-base make monkeysphere libtool unzip libssl-dev libbz2-dev libbsd-dev libarchive-dev liblzma-dev zlib1g-dev]: exit code: 100
------
 > [base 2/3] RUN apt-get update &&     apt-get install -y make jq gnupg gnupg1 gpgv1 git aptly debsig-verify createrepo-c dnf rpm                        curl gettext-base make monkeysphere libtool unzip libssl-dev libbz2-dev libbsd-dev libarchive-dev liblzma-dev zlib1g-dev:
------
executor failed running [/bin/sh -c apt-get update &&     apt-get install -y make jq gnupg gnupg1 gpgv1 git aptly debsig-verify createrepo-c dnf rpm                        curl gettext-base make monkeysphere libtool unzip libssl-dev libbz2-dev libbsd-dev libarchive-dev liblzma-dev zlib1g-dev]: exit code: 100
make: *** [Makefile:118: build-txz-packager-docker] Error 1

$ make build-docker
Building Docker
#1 [internal] load build definition from Dockerfile
#1 sha256:2b3a200ebd205cef64500bfdc0351cbc02b2e40d334aeffd6586bf56bfed64e6
#1 transferring dockerfile: 3.18kB done
#1 DONE 0.0s
[..]
#10 [install 6/6] RUN --mount=type=secret,id=nginx-crt,dst=/nginx-repo.crt     --mount=type=secret,id=nginx-key,dst=/nginx-repo.key     set -x     && addgroup --system --gid 101 nginx     && adduser --system --disabled-login --ingroup nginx --no-create-home --home /nonexistent --gecos "nginx user" --shell /bin/false --uid 101 nginx     && apt-get update     && apt-get install --no-install-recommends --no-install-suggests -y                         ca-certificates                         gnupg1                         lsb-release                         git                         wget                         make     &&     NGINX_GPGKEY=573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62;     found='';     for server in         hkp://keyserver.ubuntu.com:80         pgp.mit.edu     ; do         echo "Fetching GPG key $NGINX_GPGKEY from $server";         apt-key adv --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$NGINX_GPGKEY" && found=yes && break;     done;     test -z "$found" && echo >&2 "error: failed to fetch GPG key $NGINX_GPGKEY" && exit 1;     apt-get remove --purge --auto-remove -y gnupg1 && rm -rf /var/lib/apt/lists/*     && nginxPackages="         nginx-plus         nginx-agent     "     && echo "Acquire::https::pkgs.nginx.com::Verify-Peer "true";" > /etc/apt/apt.conf.d/90nginx     && echo "Acquire::https::pkgs.nginx.com::Verify-Host "true";" >> /etc/apt/apt.conf.d/90nginx     && echo "Acquire::https::pkgs.nginx.com::SslCert     "/etc/ssl/nginx/nginx-repo.crt";" >> /etc/apt/apt.conf.d/90nginx     && echo "Acquire::https::pkgs.nginx.com::SslKey      "/etc/ssl/nginx/nginx-repo.key";" >> /etc/apt/apt.conf.d/90nginx     && apt-get install apt-transport-https lsb-release ca-certificates     && apt-cache policy | awk '{print $2" "$3}' | sort -u     && printf "deb https://pkgs.nginx.com/plus/ubuntu/ `lsb_release -cs` nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list     && printf "deb https://pkgs.nginx.com/nginx-agent/ubuntu/ `lsb_release -cs` nginx-plus\n" > /etc/apt/sources.list.d/nginx-agent.list     && mkdir -p /etc/ssl/nginx     && cat /nginx-repo.crt > /etc/ssl/nginx/nginx-repo.crt     && cat /nginx-repo.key > /etc/ssl/nginx/nginx-repo.key     && apt-get update     && apt-get install $nginxPackages -y      && rm /etc/ssl/nginx/nginx-repo.crt /etc/ssl/nginx/nginx-repo.key
#10 sha256:56558e156042b9bc3fc02523e328965d15598ec56553420acf40ec61dc88f426
#10 0.246 + addgroup --system --gid 101 nginx
#10 0.269 Adding group `nginx' (GID 101) ...
#10 0.270 groupadd: cannot lock /etc/group; try again later.
#10 0.270 addgroup: `/sbin/groupadd -g 101 nginx' returned error code 10. Exiting.
#10 0.272 Fetching GPG key  from hkp://keyserver.ubuntu.com:80
#10 0.272 + found=
#10 0.272 + echo Fetching GPG key  from hkp://keyserver.ubuntu.com:80
#10 0.272 + apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --keyserver-options timeout=10 --recv-keys 
#10 0.304 E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
#10 0.304 Fetching GPG key  from pgp.mit.edu
#10 0.304 + echo Fetching GPG key  from pgp.mit.edu
#10 0.304 + apt-key adv --keyserver pgp.mit.edu --keyserver-options timeout=10 --recv-keys 
#10 0.336 E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
#10 0.336 + test -z 
#10 0.336 + echo error: failed to fetch GPG key 
#10 0.337 error: failed to fetch GPG key 
#10 0.337 + exit 1

The failures seem to be related to moby/buildkit#2320. With buildkit disabled, some targets are working (like make build-txz-packager-docker), some are failing with specific features missing, e.g.:

$ make build-docker
Building Docker
Sending build context to Docker daemon  116.4MB
Step 1/16 : ARG DOCKER_IMAGE
[..]
Step 9/16 : RUN --mount=type=secret,id=nginx-crt,dst=/nginx-repo.crt     --mount=type=secret,id=nginx-key,dst=/nginx-repo.key     set -x     && addgroup --system --gid 101 nginx     && adduser --system --disabled-login --ingroup nginx --no-create-home --home /nonexistent --gecos "nginx user" --shell /bin/false --uid 101 nginx     && apt-get update     && apt-get install --no-install-recommends --no-install-suggests -y                         ca-certificates                         gnupg1                         lsb-release                         git                         wget                         make     &&     NGINX_GPGKEY=573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62;     found='';     for server in         hkp://keyserver.ubuntu.com:80         pgp.mit.edu     ; do         echo "Fetching GPG key $NGINX_GPGKEY from $server";         apt-key adv --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$NGINX_GPGKEY" && found=yes && break;     done;     test -z "$found" && echo >&2 "error: failed to fetch GPG key $NGINX_GPGKEY" && exit 1;     apt-get remove --purge --auto-remove -y gnupg1 && rm -rf /var/lib/apt/lists/*     && nginxPackages="         nginx-plus         nginx-agent     "     && echo "Acquire::https::$PACKAGES_REPO::Verify-Peer \"true\";" > /etc/apt/apt.conf.d/90nginx     && echo "Acquire::https::$PACKAGES_REPO::Verify-Host \"true\";" >> /etc/apt/apt.conf.d/90nginx     && echo "Acquire::https::$PACKAGES_REPO::SslCert     \"/etc/ssl/nginx/nginx-repo.crt\";" >> /etc/apt/apt.conf.d/90nginx     && echo "Acquire::https::$PACKAGES_REPO::SslKey      \"/etc/ssl/nginx/nginx-repo.key\";" >> /etc/apt/apt.conf.d/90nginx     && apt-get install apt-transport-https lsb-release ca-certificates     && apt-cache policy | awk '{print $2" "$3}' | sort -u     && printf "deb https://$PACKAGES_REPO/plus/ubuntu/ `lsb_release -cs` nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list     && printf "deb https://$PACKAGES_REPO/nginx-agent/ubuntu/ `lsb_release -cs` nginx-plus\n" > /etc/apt/sources.list.d/nginx-agent.list     && mkdir -p /etc/ssl/nginx     && cat /nginx-repo.crt > /etc/ssl/nginx/nginx-repo.crt     && cat /nginx-repo.key > /etc/ssl/nginx/nginx-repo.key     && apt-get update     && apt-get install $nginxPackages -y      && rm /etc/ssl/nginx/nginx-repo.crt /etc/ssl/nginx/nginx-repo.key
the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled
make: *** [Makefile:218: build-docker] Error 1

[dhurley]

Since we will now have the option of using docker or podman for running containers, I think it might make sense to rename some of the make targets to make them a bit more generic.

Like for example maybe change make build-docker to make build-container-image and run-docker to run-container. There are some other make targets as well to do with docker that should be renamed as well.

Other than that the PR looks good.

[defanator]

Since we will now have the option of using docker or podman for running containers, I think it might make sense to rename some of the make targets to make them a bit more generic.

@dhurley - addressed this in 76db8ae:

% make | egrep -i "container|image"
  txz-packager-image        Builds txz packager container image
  test-container-component  Run integration tests in container
  benchmark-image           Build benchmark test container image for NGINX Plus, need nginx-repo.crt and nginx-repo.key in build directory
  image                     Build agent container image for NGINX Plus, need nginx-repo.crt and nginx-repo.key in build directory
  run-container             Run container from specified IMAGE_TAG

[dhurley]

Since we will now have the option of using docker or podman for running containers, I think it might make sense to rename some of the make targets to make them a bit more generic.

@dhurley - addressed this in 76db8ae:

% make | egrep -i "container|image"
  txz-packager-image        Builds txz packager container image
  test-container-component  Run integration tests in container
  benchmark-image           Build benchmark test container image for NGINX Plus, need nginx-repo.crt and nginx-repo.key in build directory
  image                     Build agent container image for NGINX Plus, need nginx-repo.crt and nginx-repo.key in build directory
  run-container             Run container from specified IMAGE_TAG

Thanks @defanator. The changes look good. I think you just need to resolve some merge conflicts.

[defanator]

I think you just need to resolve some merge conflicts.

@dhurley should be done now, just rebased on updated main.

[dhurley]

Looks like the integration tests are failing as its looking for a DOCKER_IMAGE environment variable. I think you need to merge the latest from main into your branch and update the integration tests to use the new environment variable BASE_IMAGE.

Here are the relative places where the changes need to be made.
https://github.com/nginx/agent/blob/main/scripts/docker/nginx-oss/ubuntu/Dockerfile#L1
https://github.com/nginx/agent/blob/main/test/integration/api/docker-compose.yml#L14
https://github.com/nginx/agent/blob/main/test/integration/api/api_test.go#L37
https://github.com/nginx/agent/blob/main/Makefile#L171

[defanator]

@dhurley addressed in ca9c86b

Probably it would be nice to add a short instruction on mandatory testing steps that must pass before submitting a PR.