Quarkus OpenShift Test Suite

For running the tests, it is expected that the user is logged into an OpenShift project:

with OpenShift 4, run oc login https://api.<cluster name>.<domain>:6443
with Minishift (which provides OpenShift 3.11), run something like oc login https://192.168.99.100:8443

To verify that you're logged into correct project, you can run oc whoami and oc project.

If you don't have a project, use oc new-project ... to create one. Alternatively, see Running tests in ephemeral namespaces below.

All tests will run in the project you're logged into, so it should be empty. If there are resources deployed in the project, you should not expect they will survive.

Running the tests amounts to standard mvn clean verify. This will use a specific Quarkus version, which can be modified by setting the version.quarkus property. Alternatively, you can use -Dquarkus-core-only to run the test suite against Quarkus 999-SNAPSHOT. In that case, make sure you have built Quarkus locally prior to running the tests.

All the tests currently use the RHEL 7 OpenJDK 11 image. This is configured in the application.properties file in each module. Since this is standard Quarkus configuration, it's possible to override using a system property. Therefore, if you want to run the tests with a different Java S2I image, run mvn clean verify -Dquarkus.s2i.base-jvm-image=....

Other prerequisites

In addition to common prerequisites for Java projects (JDK, Maven) and OpenShift (oc), the following utilities must also be installed on the machine where the test suite is being executed:

tar
an OpenShift instance where is enabled the monitoring for user-defined projects (see documentation)
the OpenShift user must have permission to create ServiceMonitor CRDs and access to the openshift-user-workload-monitoring namespace:

For example, if the OpenShift user is called qe, then we should have this cluster role assigned to it:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: monitor-crd-edit
rules:
- apiGroups: ["monitoring.coreos.com", "apiextensions.k8s.io"]
  resources: ["prometheusrules", "servicemonitors", "podmonitors", "customresourcedefinitions"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
EOF

And also, the same user qe should have access to the openshift-user-workload-monitoring namespace (this namespace is automatically created by the Prometheus operator when enabling the monitoring of user-defined projects):

oc adm policy add-role-to-user edit qe -n openshift-user-workload-monitoring

These requirements are necessary to verify the micrometer/prometheus and micrometer/prometheus-kafka tests.

Running against Red Hat build of Quarkus

When running against released Red Hat build of Quarkus make sure https://maven.repository.redhat.com/ga/ repository is defined in settings.xml.

Example command for released Red Hat build of Quarkus:

mvn -fae clean verify \
 -Dts.authenticated-registry \
 -Dversion.quarkus=1.3.4.Final-redhat-00004 \
 -Dquarkus.platform.group-id=com.redhat.quarkus

Example command for not yet released version of Red Hat build of Quarkus:

mvn -fae clean verify \
 -Dts.authenticated-registry \
 -Dversion.quarkus=1.7.1.Final-redhat-00001 \
 -Dquarkus.platform.group-id=com.redhat.quarkus \
 -Dmaven.repo.local=/Users/rsvoboda/Downloads/rh-quarkus-1.7.1.GA-maven-repository/maven-repository

If you want to run the tests using ephemeral namespaces, go to the Running tests in ephemeral namespaces section for more information.

Branching Strategy

The main branch is always meant for latest upstream/downstream development. For each downstream major.minor version, there's a corresponding maintenance branch:

1.3 for Red Hat build of Quarkus 1.3.z (corresponding upstream version: 1.3.0.Final+)
1.7 for Red Hat build of Quarkus 1.7.z (corresponding upstream version: 1.7.0.Final+)

Test Framework

The app-metadata and common directories contain a tiny framework for testing Quarkus applications on OpenShift. This is how you use it.

Your test application needs to include the app-metadata dependency in the default scope:

<dependency>
    <groupId>io.quarkus.ts.openshift</groupId>
    <artifactId>app-metadata</artifactId>
</dependency>

It is actually a little Quarkus extension. It produces a file target/app-metadata.properties with some data that the test framework needs to know about the application.

Your test application also needs to include the common dependency in the test scope:

<dependency>
    <groupId>io.quarkus.ts.openshift</groupId>
    <artifactId>common</artifactId>
    <scope>test</scope>
</dependency>

This is the actual test framework. It is in fact a JUnit 5 extension.

You will also typically have dependencies on Quarkus JUnit, RestAssured and Awaitility:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-junit5</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>io.rest-assured</groupId>
    <artifactId>rest-assured</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.awaitility</groupId>
    <artifactId>awaitility</artifactId>
    <scope>test</scope>
</dependency>

Test classes named *Test will be typical Quarkus tests, executed by Surefire, using Quarkus JUnit. OpenShift tests should be named *OpenShiftIT and be executed by Failsafe.

Quarkus typically usurps the *IT suffix for native tests. In my opinion, this is wrong. We don't have native tests here yet, but when we do, I propose it's a dedicated Failsafe execution and test classes are named *TestNative. In any case, the OpenShift tests must be executed by Failsafe, because the test framework expects the application to be already built. (In other words, the tests must run after mvn package, which is when OpenShift resources for deploying the application are created.)

The test class should be annotated with @OpenShiftTest:

@OpenShiftTest
public class HelloOpenShiftIT {
    @Test
    public void hello() {
        ...
    }
}

This will make sure that OpenShift resources are deployed before this test class is executed, and also undeployed after this test class is executed. It is expected that a YAML file with a complete list of OpenShift resources to deploy the application is present in target/kubernetes/openshift.yml. This is what the Quarkus Kubernetes extension (or the Quarkus OpenShift extension) does, when configured correctly.

After the application is deployed, the application's route is awaited. If there's a readiness probe, it is used, otherwise a liveness probe is used if it exists; if there's no health probe, the root path / is awaited. Only after the route responds successfully with status code 200 will the tests be allowed to run.

The @Test methods can use RestAssured, which is preconfigured to point to the deployed application. When you need to wait for something to happen, use Awaitility; don't write your own wait loops with Thread.sleep etc.

@OpenShiftTest
public class HelloOpenShiftIT {
    @Test
    public void hello() {
        // this is an EXAMPLE, waiting here isn't necessary! the test framework
        // already waits for the application to start responding
        await().atMost(1, TimeUnit.MINUTES).untilAsserted(() -> {
            when()
                    .get("/hello")
            .then()
                    .statusCode(200)
                    .body("content", is("Hello, World!"));
        });
    }
}

Dependency injection

For more complex test scenarios, you can obtain some useful objects by annotating a test instance field or a test method parameter with @TestResource:

@OpenShiftTest
public class HelloOpenShiftIT {
    @TestResource
    private OpenShiftClient oc;

    @Test
    public void hello() {
        oc.routes().list().getItems().forEach(System.out::println);

        ...
    }
}

The full set of objects that you can inject is:

OpenShiftClient: the Fabric8 Kubernetes client, in default configuration
KnativeClient: the Fabric8 Knative client, in default configuration
AppMetadata: provides convenient access to data collected in target/app-metadata.properties
AwaitUtil: utility to wait for some OpenShift resources
OpenShiftUtil: utility to perform higher-level actions on some OpenShift resources
Config: simple configuration utility for the test suite; currently only reads system properties
URL: URL of deployed route, application route is the default, can be adjusted using @WithName annotation

Deploying additional resources

The test application might require additional OpenShift resources to be deployed, such as ConfigMaps or other applications. To do that, you can use the @AdditionalResources annotation:

@OpenShiftTest
@AdditionalResources("classpath:configmap.yaml")
public class HelloOpenShiftIT {
    ...
}

These resources are deployed before the test application is deployed, and are also undeployed after the test application is undeployed. This annotation is @Repeatable, so you can include it more than once.

Also, additional resources with routes sometimes need to be injected into the Quarkus applications. For example, if we have the next route.yaml file with content:

apiVersion: v1
kind: List
items:
- apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    name: keycloak-plain
  spec:
    to:
      name: keycloak-plain

Then, We can use the @InjectRouteUrlIntoApp annotation to inject the route URL into the Quarkus application as an environment variable:

@OpenShiftTest
@AdditionalResources("classpath:route.yaml")
@InjectRouteUrlIntoApp(route = "keycloak-plain", envVar = "KEYCLOAK_HTTP_URL")
public class HelloOpenShiftIT {
    ...
}

After deploying the route, the framework will deploy the module test application with the environment variable KEYCLOAK_HTTP_URL and the route URL.

Note that environment variables are available to be used in properties files like:

quarkus.my.property=${KEYCLOAK_HTTP_URL:default_value}

Further information about usage of configuration in here.

Note: In case that you have several unrelated additional resources as Redis and postgres together in the same application, you can deploy those resources in parallel by adding @ParallelAdditionalResourcesEnabled annotation. Note that this label applies to all additional resources of this scenario.

Running tests in ephemeral namespaces

By default, the test framework expects that the user is logged into an OpenShift project, and that project is used for all tests.

If you want to run the tests using ephemeral namespaces, you need to build and install the TS framework modules beforehand:

mvn clean install -pl '.,app-metadata/deployment,app-metadata/runtime,common'

Then, you can start the tests with the -Dts.use-ephemeral-namespaces property, the test framework will create an ephemeral namespace for each test. After the test is finished, the ephemeral namespace is automatically dropped:

mvn clean verify -Dts.use-ephemeral-namespaces

The ephemeral namespaces are named ts-<unique suffix>, where the unique suffix is 10 random a-z characters.

Retaining resources on failure

When the test finishes, all deployed resources are deleted. Sometimes, that's not what you want: if the test fails, you might want all the OpenShift resources to stay intact, so that you can investigate the problem. To do that, run the tests with -Dts.retain-on-failure.

This works with and without ephemeral namespaces, but note that if you're not using ephemeral namespaces, all the tests run in a single namespace. In such case, when you enable retaining resources on test failure, it's best to only run a single test.

Enabling/disabling tests

The @OnlyIfConfigured annotation can be used to selectively enable/disable execution of tests based on a configuration property. Similarly, the @OnlyIfNotConfigured annotation can be used to selectively enable/disable the execution of tests based on a configuration property unavailability.

This can be used for example to disable tests that require access to authenticated registry. The tests that do require such access are annotated @OnlyIfConfigured("ts.authenticated-registry") and are not executed by default. When executing tests against an OpenShift cluster that has configured access to the authenticated registry, you just run the tests with -Dts.authenticated-registry and the tests will be executed.

The @DisabledOnQuarkus annotation can be used to selectively disable execution of tests based on Quarkus version. The Quarkus version used in the test suite is matched against a regular expression provided in the annotation, and if it matches, the test is disabled.

This can be used to disable tests that are known to fail on certain Quarkus versions. In such case, the reason attribute of the annotation should point to corresponding issue (or pull request).

Deployment using Quarkus OpenShift extension

If the @OpenShiftTest annotation is configured to use the strategy UsingQuarkusPluginDeploymentStrategy and the module contains the quarkus-openshift Quarkus extension, the test framework will use Quarkus to build and deploy the module into OpenShift. Note that the property quarkus.kubernetes.deploy must NOT be set in the module configuration in order to avoid to perform the deployment at build time.

Deployment using Quarkus docker build strategy

If the @OpenShiftTest annotation is configured to use the strategy UsingQuarkusPluginDeploymentStrategy.UsingDockerStrategy and the module contains the quarkus-openshift extension, the test framework will use Quarkus to orchestrate a docker build of the module and its deployment to OpenShift. Note that the property quarkus.kubernetes.deploy must NOT be set in the module configuration in order to avoid the deployment at build time. Also note that the default dockerfiles generated by Quarkus project bootstrapping must be present for this strategy to work.

Custom application deployment

If the @OpenShiftTest annotation is configured to use the strategy ManualDeploymentStrategy, the target/kubernetes/openshift.yml file is ignored and the test application is not deployed automatically. Instead, you should use @AdditionalResources, @CustomizeApplicationDeployment and @CustomizeApplicationUndeployment to deploy the application manually.

This can be used to write tests that excersise an external application. You have full control over the application deployment and undeployment process, but the rest of the test can be written as if the application was part of the test suite. In such case, the deployed application can't use the app-metadata Quarkus extension, and you have to use the @CustomAppMetadata annotation to provide all the necessary information.

Image overrides

It is sometimes useful to globally override certain images, for example when testing with a pre-release version of an image that is not yet available publicly. In such case, you can set -Dts.image-overrides to a path of a file that looks like this:

registry.access.redhat.com/openjdk/openjdk-11-rhel7=registry.access.redhat.com/ubi8/openjdk-11
registry.access.redhat.com/rhscl/postgresql-10-rhel7=registry.redhat.io/rhscl/postgresql-12-rhel7

This looks like a .properties format, but it is not! Specifically, the format is:

empty lines and lines that begin with # are ignored;
other lines must have a source image name (possibly with a tag), followed by =, followed by a target image name (possibly with a tag).

When a YAML file refers to the source image, it is changed to use the target image before it is deployed. If there's no tag in the configuration of the source image, it will match all tags.

Note that this is not dumb string search & replace. We actually edit the Kubernetes resources on a few specific places (such as container definition or image stream definition), the rest is left unchanged.

This currently works automatically for the target/kubernetes/openshift.yml file and all the files deployed with @AdditionalResources.

Note that it is usually a good idea to set -Dts.image-overrides to a full path, because Maven changes the current working directory when running tests.

Native image

The test suite contains a Maven profile configuring required system properties for native image build. The profile is activated by specifying the native Quarkus profile (-Dquarkus.profile=native). This profile does not specify the container runtime for Quarkus native build, so by default, native build will use local GraalVM. The following command will execute the whole test suite using Docker to run containers for native build:

./mvnw clean verify \
  -Dquarkus.profile=native \ 
  -Dquarkus.native.container-runtime=docker \
  -Dts.authenticated-registry

Currently used builder image is quarkus/ubi-quarkus-mandrel-builder-image and the base image for OpenShift deployment is quarkus/ubi-quarkus-native-binary-s2i.

OpenShift Serverless / Knative

The test suite contains a Maven profile activated using the include.serverless property or serverless profile name. This profile includes additional modules with serverless test coverage into the execution of the testsuite. Serverless test coverage supports both JVM and Native mode.

The following command will execute the whole test suite including serverless tests:

./mvnw clean verify -Dinclude.serverless

OpenShift Pod Logs on Failures

When a test fails, the test suite will copy the pod logs in order to troubleshoot the issue. These logs will be copied in the [module]/target/logs/[namespace] folder. We can configure the output folder using the ts.instance-logs property:

./mvnw clean verify -Dts.instance-logs=../custom/path

OpenShift Security Configuration

Go to the Security guide for further details about how to configure the OpenShift suite with security enabled.

Default timeout

When the test framework needs to await something, such as deployment readiness, it uses a default timeout of 10 minutes. If you want to change that value, you can use -Dts.default-timeout to override the default value. The unit is minutes, so e.g. -Dts.default-timeout=5 means default timeout of 5 minutes.

Custom Maven Repository for S2I

OpenShift provides an S2I (Source-to-Image) process to build applications directly using the source code. By default, OpenShift will build the application's source code using the RedHat maven repository https://maven.repository.redhat.com/ga/. However, our applications might require some dependencies from other remote Maven repositories. In order to allow us to add another remote Maven repository, you can use -Dts.s2i.maven.remote.repository=http://host:port/repo/name.

Then, we need to instruct our OpenShiftTest to load the Maven settings using:

@OpenShiftTest
@AdditionalResources("classpath:deployments/maven/s2i-maven-settings.yaml")
@AdditionalResources("classpath:myApp.yaml")
public class MyAppIT {
   // ...
}

The above code snippet will load the Maven settings and the myApp.yaml must contain the S2I build including the configMap settings-mvn and the MAVEN_ARGS to overwrite the settigs.xml file:

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: myApp
spec:
  output:
    to:
      kind: ImageStreamTag
      name: myapp:latest
  source:
    git:
      uri: https://github.com/repo/name.git
    type: Git
    configMaps:
    - configMap:
        name: settings-mvn
      destinationDir: "/configuration"
  strategy:
    type: Source
    sourceStrategy:
      env:
      - name: ARTIFACT_COPY_ARGS
        value: -p -r lib/ *-runner.jar
      - name: MAVEN_ARGS
        value: -s /configuration/settings.xml
      from:
        kind: ImageStreamTag
        name: openjdk-11:latest
  triggers:
  - type: ConfigChange
  - type: ImageChange
    imageChange: {}

TODO

There's a lot of possible improvements that haven't been implemented yet. The most interesting probably are:

Automation for some alternative deployment options. Currently, we support binary S2I builds, with the expectation that quarkus-kubernetes (or quarkus-openshift) is used. We also support completely custom application deployment, which provides most flexibility, but also requires most work. At the very least, we should provide some automation for S2I-less deployments with -Dquarkus.container-image.build -Dquarkus.kubernetes.deploy. Supporting S2I source builds would be more complex (you need the application source stored in some Git repo, and when you're working on the test suite, you want your local changes, not something out there on GitHub) and is probably best left to ManualDeploymentStrategy strategy.
To be able to customize URL path for route awaiting.
To be able to cleanup the project before running the test. Could be just a simple annotation @CleanupProject added on the test class.
To be able to inject more resources automatically. For example, the OpenShift objects corresponding to the application (such as DeploymentConfig, Service, Route, or even Pod if there's only one).
To be able to configure the connection to OpenShift cluster (that is, how OpenShiftClient is created). Currently, this isn't possible, and doesn't make too much sense, because for some actions, we just run oc, while for some, we use the Java client. We could possibly move towards doing everything through the Java library, but some things are hard (e.g. oc start-build). If we ever do this, then it becomes easily possible to consisently apply image overrides everywhere.
Implement a build workflow with ubi8/ubi-minimal instead of using quarkus/ubi-quarkus-native-binary-s2i.

It would also be possible to create a Kubernetes variant: @KubernetesTest, kubernetes.yml, injection of KubernetesClient, etc. Most of the code could easily be shared. The main difference is that there's no S2I in Kubernetes, so we'd have to do the S2I-less thing mentioned above.

Existing tests

`http-minimum`

Verifies that you can deploy a simple HTTP endpoint to OpenShift and access it.

`http-advanced`

Verifies Server/Client http_2/1.1, Grpc and http redirections.

`configmap/file-system`

Checks that the application can read configuration from a ConfigMap. The ConfigMap is exposed by mounting it into the container file system.

`configmap/api-server`

Checks that the application can read configuration from a ConfigMap. The ConfigMap is obtained directly from the Kubernetes API server.

`config-secret/file-system`

Checks that the application can read configuration from a Secret. The Secret is exposed by mounting it into the container file system.

`config-secret/api-server`

Checks that the application can read configuration from a Secret. The Secret is obtained directly from the Kubernetes API server.

`lifecycle-application`

Verifies lifecycle application features like @QuarkusMain and @CommandLineArguments.

`sql-db`

Verifies that the application can connect to a SQL database and persist data using Hibernate ORM with Panache. The application also uses RESTEasy to expose a RESTful API, Jackson for JSON serialization, and Hibernate Validator to validate inputs. There are actually multiple Maven modules in the sql-db directory:

app: the main application and the test code; no JDBC drivers (except H2 for unit test)
postgresql: depends on app and the PostgreSQL JDBC driver; produces the PostgreSQL-specific build of the application and runs the OpenShift test with PostgreSQL
mysql: same for MysQL
mariadb: same for MariaDB
mssql: same for MSSQL
multiple-pus: An application with two persistence units defined in application.properties.

All the tests deploy a SQL database directly into OpenShift, alongside the application. This might not be recommended for production, but is good enough for test. Container images used in the tests are:

PostgreSQL:
- version 10: registry.access.redhat.com/rhscl/postgresql-10-rhel7
- version 12: registry.redhat.io/rhscl/postgresql-12-rhel7 (only if ts.authenticated-registry is set)
MySQL:
- version 8.0: registry.access.redhat.com/rhscl/mysql-80-rhel7
MariaDB:
- version 10.2: registry.access.redhat.com/rhscl/mariadb-102-rhel7
- version 10.3: registry.redhat.io/rhscl/mariadb-103-rhel7 (only if ts.authenticated-registry is set)
MSSQL: mcr.microsoft.com/mssql/rhel/server

`infinispan-client`

Verifies the way of the sharing cache by Datagrid operator and Infinispan cluster and data consistency after failures.

Prerequisites

Datagrid operator installed in datagrid-operator namespace. This needs cluster-admin rights to install.
The operator supports only single-namespace so it has to watch another well-known namespace datagrid-cluster. This namespace must be created by "qe" user or this user must have access to it because tests are connecting to it.
These namespaces should be prepared after the Openshift installation - See Installing Data Grid Operator

The test suite contains a Maven profile activated using the include.datagrid. To execute Datagrid tests use the following switch in the maven command:

-Dinclude.datagrid

Tests create an Infinispan cluster in the datagrid-cluster namespace. Cluster is created before tests by infinispan_cluster_config.yaml. To allow parallel runs of tests this cluster is renamed for every test run - along with configmap infinispan-config. The configmap contains configuration property quarkus.infinispan-client.server-list. Value of this property is a path to the infinispan cluster from test namespace, its structure is infinispan-cluster-name.datagrid-cluster-namespace.svc.cluster.local:11222. It is because the testsuite uses dynamically generated namespaces for tests. So this path is needed for the tests to find Infinispan cluster in another namespace.

The Infinispan cluster needs 2 special secrets - tls-secret with TLS certificate and connect-secret with the credentials. TLS certificate is a substitution of secrets/signing-key in openshift-service-ca namespace, which "qe" user cannot use (doesn't have rights on it). Clientcert secret is generated for "qe" from the tls-secret mentioned above.

Infinispan client tests use the cache directly with @Inject and @RemoteCache. Through the JAX-RS endpoint, we send data into the cache and retrieve it through another JAX-RS endpoint. The next tests are checking a simple fail-over - first client (application) fail, then Infinispan cluster (cache) fail. Tests kill first the Quarkus pod then Infinispan cluster pod and then check data. For the Quarkus application, pod killing is used the same approach as in configmap tests. For the Infinispan cluster, pod killing is updated its YAML snipped and uploaded with zero replicas. By default, when the Infinispan server is down and the application can't open a connection, it tries to connect again, up to 10 times (max_retries) and gives up after 60s (connect_timeout). Because of that we are using the hotrod-client.properties file where are the max_retries and connect_timeout reduced. Without this the application will be still trying to connect to the Infinispan server next 10 minutes and the incremented number can appear later. The last three tests are for testing of the multiple client access to the cache. We simulate the second client by deploying the second deployment config, Service, and Route for these tests. These are copied from the openshift.yml file.

`security/basic`

Verifies the simplest way of doing authn/authz. Authentication is HTTP Basic, with users/passwords/roles defined in application.properties. Authorization is based on roles, restrictions are defined using common annotations (@RolesAllowed etc.).

`security/jwt`

Verifies token-based authn and role-based authz. Authentication is MicroProfile JWT, and tokens are issued manually in the test. Authorization is based on roles, which are embedded in the token. Restrictions are defined using common annotations (@RolesAllowed etc.).

`security/jwt-cookie`

Same example as in security/jwt but setting the token into a Cookie instead of via HTTP headers.

`security/keycloak`

Verifies token-based authn and role-based authz. Authentication is OIDC, and Keycloak is used for issuing and verifying tokens. Authorization is based on roles, which are embedded in the token. Restrictions are defined using common annotations (@RolesAllowed etc.).