Make podman first class citizens for quarkus usecases

[maxandersen]

Description

Recent podman desktop (0.12) and podman (4.4) by default now activates docker compatibility making it simple to setupand start across all three major os's.

This issue is to capture outstanding issues in quarkus, testcontainers, podman,devsevices containers etc that prevents a smooth podman experience.

Overall setup

Issues that makes installation and setup problematic/challenging/more difficult than we would prefer

• have docker socket enabled by default on windows and OSX (Fixed since podman 4.4.1+ and desktop 0.12.0+)
• test containers requiring enabling privileged ryuk to be enabled

Privileged vs non-priviliged by default

In general docker runs in a setup that defaults to risky/unsafe; where as podman does the opposite. runs in non-priviliged mode. We can probably explicitly make our usage enable unsafe running for both docker and podman.

Downside is that security concerned podman users would not like to do that by default; but on the other hand users running on docker already run it this way.

A classic security first vs usability first issue.

What approach we take here is yet unlcear

Affected Devservices

• Elasticsearch (chroot: cannot change root directory to '/': Operation not permitted)
• DB2 (multitude of root / syscall issues)

Affected container centric usecases

• quarkus docker build extension fails on podman
• docker compose with volumes behaves differently

[holly-cummins]

Here are some sources of friction. Many of these aren't 'defects', exactly, because they can be resolved with appropriate config, but they require debugging and config work when people start using podman - so a definitely "not-first-class" experience:

• Testcontainers ryuk configuration. This is needed for test containers. With no config, users will get an error similar to panic: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/_ping: dial unix /var/run/docker.sock: connect: permission denied. Many people set export TESTCONTAINERS_RYUK_DISABLED=true, but this prevents container cleanup. Editing ~/.testcontainers.properties and adding ryuk.container.privileged=true is a better solution. Podman support testcontainers/testcontainers-java#2088 has some of this discussion (but the issue is closed, and there's a lot of different errors discussed in the one issue). Permission denied when trying to use the `/var/run/docker.sock` file generated by the mac-helper containers/podman#14238 also tracks this issue (among others). We should perhaps raise an issue against testcontainers asking them to detect podman and suggest changing the setting.

• Short names of containers. Testcontainers and Quarkus Dev Services also expect the container service they make requests against to be non-interactive. If someone has multiple registries configured, and when using short image names, Podman responds with a prompt asking which registry should be used to pull images. Testcontainers defaults to using short names internally, so users need to change this default this prompt by setting the short-name-mode="disabled" configuration property of Podman in /etc/containers/registries.conf. Error, when pulling short-named image via docker api containers/podman#16400 may be tracking this, or it may be a related testcontainers + podman issue. I can't quite tell.

• Missing qemu on Arm podman machines. Users need to hand-tune the machine image created by podman to install qemu, for AMD64 images to work. This step isn't needed in Docker, so it can surprise some users. It needs to be redone every time a new podman image is set up. It looks like this is podman default machine on macOS M1/M2 could ship qemu-user-static out of the box to use non-ARM64 containers containers/podman#16148 and it was done in November, so we should test the update and update our instructions if it works.

• It's often necessary to specify the architecture explicitly with podman, in a way it isn't for Docker. See, for example, Use health check to detect mariadb startup and skip volume modifier on M1 #25805, Disable Oracle db tests on Mac M1 because of segfault (and updated readiness check) #25832, MySQL on M1: use architecture-specific image, and health check for readiness detection #25878. These are a mix of M1 issues and Podman issues, but what we've seen is that Podman behaves worse on M1 than Docker does. In some cases we were able to update the dev service configuration to work with podman+M1, but in other cases we had to disable the test (ie we now pass CI, but don't actually work), or do configuration in the test launching that users would have to replicate.

• Test coverage. Although this isn't the same sort of issue, we have a hard time running our CI on podman because of intermittent podman launch errors. We've raised Podman machine start fails with 'no such file or directory', reboot resolves issue on Mac M1 containers/podman#16163 to track this, but it's hard to provide a good reproducer. We're also seeing purgeable space leaks on the machine, which may or may not be related to podman.

[edeandrea]

podman version 4.4.1 on Darwin edeandrea-m1pro 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64

Here is another one - try to run the Quarkus Superheroes via Docker Compose - https://github.com/quarkusio/quarkus-super-heroes#running-locally-via-docker-compose

This does not work. I don't think it is a Quarkus thing either - more that podman (at least on macOS M1) doesn't seem to like volumes.

╰─ docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml up  
[+] Running 6/10
 ⠿ Network docker-compose_default     Created                                                                                0.0s
 ⠿ Container ui-super-heroes          Created                                                                                0.1s
 ⠹ Container villains-db              Creating                                                                               0.2s
 ⠹ Container fights-db                Creating                                                                               0.2s
 ⠿ Container jaeger                   Created                                                                                0.1s
 ⠿ Container apicurio                 Created                                                                                0.1s
 ⠹ Container prometheus               Creating                                                                               0.2s
 ⠹ Container heroes-db                Creating                                                                               0.2s
 ⠋ Container otel-collector           Creating                                                                               0.0s
 ⠿ Container fights-kafka             Created                                                                                0.0s
 ⠿ Container event-statistics-java17  Created                                                                                0.0s
Error response from daemon: make cli opts(): making volume mountpoint for volume /Users/edeandre/workspaces/quarkus/quarkus-super-heroes-main/rest-fights/deploy/db-init/initialize-database.js: mkdir /Users: operation not permitted

[n1hility]

• Testcontainers ryuk configuration. This is needed for test containers.

-snip-

Improving this experience has been on my TODO for awhile, the current solution is as you mention to use the ryuku privileged flag and to run as rootful on Mac and Windows. On Linux, you need a podman service and some file permission changes. The underlying problem is some assumptions in the ryuk containers that do not interplay well with rootless. A solution with improved usability likely involves patches to testcontainers, and perhaps some on the Quarkus side.

• Short names of containers. Testcontainers and Quarkus Dev Services also expect the container service they make requests against to be non-interactive.

I need to double check this one, I think there were improvements to this with recent images of podman machine

• Missing qemu on Arm podman machines.

This should be resolved now on Mac. I still need to bring this change for windows.

• It's often necessary to specify the architecture explicitly with podman, in a way it isn't for Docker. See, for example, Use health check to detect mariadb startup and skip volume modifier on M1 #25805, Disable Oracle db tests on Mac M1 because of segfault (and updated readiness check) #25832, MySQL on M1: use architecture-specific image, and health check for readiness detection #25878.

IIUC this is about selinux label enforcement and 9p. I wonder if the recent change to switch the qemu 9p security model away from mapped-xattr helps here. A possible improvement could be to ignore the Z/z flags when the FS does not support them

" but what we've seen is that Podman behaves worse on M1 than Docker does."

Do you mean from these issues you linked or something else?

• Test coverage. Although this isn't the same sort of issue, we have a hard time running our CI on podman because of intermittent podman launch errors. We've raised Podman machine start fails with 'no such file or directory', reboot resolves issue on Mac M1 containers/podman#16163 to track this, but it's hard to provide a good reproducer.

We really need to get a reproducer and some logs on this one. I haven't been luck (unlucky?) enough to see this one locally.

We're also seeing purgeable space leaks on the machine, which may or may not be related to podman.

I think this is unlikely to be related to podman since mac has no insight into the linux filesystem podman is using inside a single contiguous loopback file. It's more likely this is coming from files directly on the APFS. One way to test could be to do a podman machine rm and see if purgeable space recovers.

[n1hility]

@edeandrea hey i have super heroes working here. Are you running in rootful mode? I recommend doing the following:

podman machine rm -f
podman machine init
podman machine set -m 8192
podman machine set --rootful
podman machine start

This would blow away any stale image you might have from an old podman release, and set things as rootful. One issue I did notice is that the flights-kaka container will fail on subsequent starts because /tmp is not mounted to tmpfs so is preserved across restarts. If you add the following to your compose under fights-kaka it resolves it:

  tmpfs:
      - /tmp

I need to look into why this isn't done by default.

[edeandrea]

@n1hility I did exactly this today:

podman machine ls

# Then removed all the ones that showed up

brew update && brew upgrade

podman machine init --cpus 4 --memory 6144 --rootful

PODMAN_VERSION=`podman -v | sed 's/[a-zA-Z ]*//'`

sudo /opt/homebrew/Cellar/podman/$PODMAN_VERSION/bin/podman-mac-helper install

# I already had ryuk.container.privileged=true in ~/.testcontainers.properties

podman machine start

podman machine ssh
sudo -i
rpm-ostree install qemu-user-static

# add short-name-mode="disabled" configuration property in /etc/containers/registries.conf

systemctl reboot

podman machine stop

podman machine start

# git clone superheroes
cd quarkus-super-heroes

docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml pull

docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml up

# Boom
[+] Running 6/10
 ⠿ Network docker-compose_default     Created                                                                                0.0s
 ⠿ Container ui-super-heroes          Created                                                                                0.1s
 ⠹ Container villains-db              Creating                                                                               0.2s
 ⠹ Container fights-db                Creating                                                                               0.2s
 ⠿ Container jaeger                   Created                                                                                0.1s
 ⠿ Container apicurio                 Created                                                                                0.1s
 ⠹ Container prometheus               Creating                                                                               0.2s
 ⠹ Container heroes-db                Creating                                                                               0.2s
 ⠋ Container otel-collector           Creating                                                                               0.0s
 ⠿ Container fights-kafka             Created                                                                                0.0s
 ⠿ Container event-statistics-java17  Created                                                                                0.0s
Error response from daemon: make cli opts(): making volume mountpoint for volume /Users/edeandre/workspaces/quarkus/quarkus-super-heroes-main/rest-fights/deploy/db-init/initialize-database.js: mkdir /Users: operation not permitted

If I then podman machine stop then start docker desktop & re-run docker compose -f deploy/docker-compose/java17.yml -f deploy/docker-compose/monitoring.yml up everything works fine

[n1hility]

@edeandrea hmm interesting what version of Docker cli are you using?

[edeandrea]

╰─ docker version                                                                                                  
Client:
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Wed Jun  1 21:26:39 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.16.2 (95914)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:43 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[edeandrea]

[n1hility]

@edeandrea Interesting I’m on a slightly newer version I’ll try to mirror your setup and see what happens

[edeandrea]

Newer version? Docker Desktop says I'm on the latest....

[n1hility]

@edeandrea Yeah I'm on 20.10.21. Although I am on arm so maybe different versions released?

I can't seem to reproduce for some reason with 20.10.16 either. Could you paste a podman info and then a docker info with podman running? I'll keep trying other things to see if I can figure out how you are getting that.

I am on an older mac os release, and I have been looking for an excuse to upgrade, so might try that this weekend.

[n1hility]

oh you are arm too? not sure how we ended up with different versions