Improve podman compatibility on M1 for build test suites which use fabric8 docker-maven-plugin

[holly-cummins]

Describe the bug

#25339 improved podman compatibility in some of the test suites which use fabric8 containers for databases. However, I'm still seeing failures on Mac M1 with podman.

Once I worked out I needed to do podman machine init -v $HOME:$HOME for podman < 4.1, most of the issues I've been seeing relate to filesystem mounting and health checks.

Volume mounts

The ':Z' SELinux access label seems to cause problems on M1. I think this is related to containers/podman#13631. (src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildLocalContainerRunner.java also used :Z so I'm not sure why I didn't have to change anything there. )

Health checks

This is more troublesome. I'm struggling to get the docker maven plugin to successfully <wait> for mariadb to be up. I've tried a range of options.

For example, I could not find a wait of doing a (from http://dmp.fabric8.io/#build-healthcheck) that didn't treat the <time> as both a minimum and a maximum. Several of the options I tried always passed, even if the container wasn't ready, or always failed, even if the container was actually ready.

What about test containers?

The fabric8 plugin is not under super-active development, and we're starting to see some non-ideal behaviours. Longer term, we probably want to move away from the fabric8 docker-maven-plugin to test containers, but last time we looked into it, the fabric8 plugin gave us more flexibility to do things like define a sophisticated health check for container readiness, or inject a volume into the container with config to speed up startup.

Expected behavior

Tests should pass cleanly with TESTCONTAINERS_RYUK_DISABLED="true" ./mvnw -Dquickly -DskipTests=false -Dstart-containers

Actual behavior

SELinux access issues

holly@hcummins-mac quarkus % MYSQL_USER=hibernate_orm_test MYSQL_PASSWORD=hibernate_orm_test MYSQL_DATABASE=hibernate_orm_test MYSQL_RANDOM_ROOT_PASSWORD=true podman run --env MYSQL\*  -v ./integration-tests/hibernate-orm-tenancy/connection-resolver/custom-mariadbconfig/:/etc/mysql/conf.d:Z docker.io/mariadb:10.7
Error: error preparing container 0c92a88aedfb270b2c2c81aa9e98b025170295b2f7e0a1b8ac42bd6492bd42b9 for attach: setxattr /Users/holly/Code/quarkus/myfork/quarkus/integration-tests/hibernate-orm-tenancy/connection-resolver/custom-mariadbconfig: operation not supported

Health checks

I'm seeing a range of problematic behaviours, which mostly involve waits reporting themselves as failed when they should be successful, or reporting as failed when they should be successful.

How to Reproduce?

TESTCONTAINERS_RYUK_DISABLED="true" ./mvnw -Dquickly -DskipTests=false -Dstart-containers -f extensions/reactive-mysql-client/deployment 

on M1 without docker installed (only podman)

Output of uname -a or ver

Darwin hcummins-mac 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @quarkusio/devtools

[holly-cummins]

Pull request incoming, once I get things sorted out.

[holly-cummins]

Here's some more context on the health check. The current pom.xmls tend to use tcp checks, of the form

                                           <tcp>
                                                <mode>direct</mode>
                                                <ports>
                                                    <port>3306</port>
                                                </ports>
                                            </tcp>

These don't seem to be behaving well in podman on my M1. A mysqladmin ping check is generally more reliable and desirable. Doing it in a <postStart> will sometimes work as intended, but with a few bad side effects:

• the build will wait <time> ms before even trying the script
• sometimes the script can fail but the build will still pass

A <healthy> check seems ideal, but I'm having a hard time making it work. What I'm seeing is that even if the script returns 0, the healthcheck times out. I've reproduced this with the simplest possible check (exit 0). I assume this is some unhappy interaction between the docker maven plugin and podman, but it could just be user error:

20:08:19.000 MariaDB:2022-05-06 19:08:19 0 [Note] mariadbd: ready for connections.
20:08:19.000 MariaDB:Version: '10.7.3-MariaDB-1:10.7.3+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
[ERROR] DOCKER> [docker.io/mariadb:10.7-healthed] "quarkus-test-mariadb": Timeout after 20030 ms while waiting on healthcheck '"CMD-SHELL", "exit 0"'
[ERROR] DOCKER> Error occurred during container startup, shutting down...
ERRO[0059] accept tcp [::]:3308: use of closed network connection 
[INFO] DOCKER> [docker.io/mariadb:10.7-healthed] "quarkus-test-mariadb": Stop and removed container 6045fab24caf after 0 ms
[ERROR] DOCKER> I/O Error [[docker.io/mariadb:10.7-healthed] "quarkus-test-mariadb": Timeout after 20030 ms while waiting on healthcheck '"CMD-SHELL", "exit 0"']
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  36.309 s
[INFO] Finished at: 2022-05-06T20:08:35+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.39.1:start (docker-start) on project quarkus-reactive-mysql-client-deployment: I/O Error: [docker.io/mariadb:10.7-healthed] "quarkus-test-mariadb": Timeout after 20030 ms while waiting on healthcheck '"CMD-SHELL", "exit 0"' -> [Help 1]

[holly-cummins]

Confirmed user error(ish): filling in the timeout values gets the health check working.

   <healthCheck>
                                            <interval>5</interval>
                                            <timeout>3</timeout>
                                            <retries>3</retries>

[Sanne]

you're right, mysql should not be checked just by it opening the port.

It's weird, I knew about that as I've had similar problems on other projects and yet never noticed we had it wrong here - I guess we simply copied the approach from the postgresql module, which was the first one to be created: AFAIK the approach to monitor ports being opened works fine with postgresql (but only with postgresql).

[holly-cummins]

Fixed by above PRs, so closing. We're not there on M1, but we're closer.