Running a QuarkusTest fails due to class load errors with Kotlin after quarkus 3 migration

[mihaipoenaru]

Describe the bug

I have a small kotlin/resteasy-reactive reproducer that has nothing but a single test calling the assertTimeout function from junit jupiter. The reproducer was made by following the official instructions here. When running the test, it fails with

GreetingResourceTest.testHelloEndpoint:13 » Linkage loader constraint violation: loader 'app' wants to load interface kotlin.jvm.functions.Function0. A different interface with the same name was previously loaded by io.quarkus.bootstrap.classloading.QuarkusClassLoader @7b8aebd0. (kotlin.jvm.functions.Function0 is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @7b8aebd0, p arent loader 'app')

Note that this doesn't happen if I remove the @QuarkusTest annotation and just run it as a plain test, which leads me to believe that the Quarkus context and the test runner are knocking heads.

This previously worked fine with Quarkus 2.16. I've tried multiple Quarkus 3 versions, before and after 3.1, same behavior each time.

Expected behavior

Test should pass instantly after Quarkus starts up

Actual behavior

Test fails with the error pasted in the bug description

How to Reproduce?

Reproducer:

Run the quarkus or maven command to generate the project

mvn io.quarkus.platform:quarkus-maven-plugin:3.1.2.Final:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=rest-kotlin-quickstart \
    -Dextensions='kotlin,resteasy-reactive-jackson'
cd rest-kotlin-quickstart

2. (optional) delete every class that isn't GreetingResourceTest, to get the absolute minimal reproducer
3. The test class should be like this:

package org.acme

import io.quarkus.test.junit.QuarkusTest
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertTimeout
import java.time.Duration

@QuarkusTest
class GreetingResourceTest {

    @Test
    fun testHelloEndpoint() {
        assertTimeout(Duration.ofSeconds(1)) {}
    }
}

4. run mvn install and observe the error

Output of uname -a or ver

Windows 10 Enterprise 19044.2846 Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz 2.11 GHz

Output of java -version

openjdk 17.0.5 2022-10-18 OpenJDK Runtime Environment GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08) OpenJDK 64-Bit Server VM GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08, mixed mode, sharing)

GraalVM version (if different from Java)

Environment GraalVM CE 22.3.0

Quarkus version or git rev

3.1.2.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Maven home: C:\Program Files\Maven\apache-maven-3.9.2 Java version: 17.0.5, vendor: GraalVM Community, runtime: C:\Program Files\Java\graalvm-ce-java17-22.3.0 Default locale: en_US, platform encoding: Cp1252 OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

Additional information

This linking issue may actually be affecting another area of my project, but I cannot 100% confirm. I'll write a brief explanation here, maybe it's relevant and it helps.

My main project has several maven modules, one of them is a commons type artifact imported as a dependency by the other modules.

One such file is this JsonUtils.kt, very basic content:

package com.orange.vp.bo.utils

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializationFeature
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper

val objectMapper: ObjectMapper = jacksonObjectMapper()
    .registerModule(JavaTimeModule())
    .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)
    .disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)

fun Any?.asJson(): String = objectMapper.writeValueAsString(this)

inline fun <reified T> Any?.convertTo(): T = objectMapper.convertValue(this, T::class.java)

When a test tries to use the objectMapper, it fails with:

SearchCriterionTest.get null terminal value:32 NoClassDefFound Could not initialize class com.orange.vp.bo.utils.JsonUtilsKt

Again, this worked prior to Quarkus 3. All of the modules (including the commons) inherit the same parent pom that gives the quarkus and the kotlin versions (and some common pom dependencies), so there is no desynchronization here.

Similarly to the main bug, this only happens when running tests. The same objectMapper is used in the main classes, without issue.

[quarkus-bot]

/cc @evanchooly (kotlin), @geoand (kotlin)

[geoand]

Might be related to #29697 @stuartwdouglas @holly-cummins

[holly-cummins]

Yes, I think that seems quite plausible, @geoand. Thanks for the minimal reproducer, @mihaipoenaru.

@mihaipoenaru, do you see the failures in both mvn quarkus:dev and mvn verify?

@mihaipoenaru, while we look at the reproducer on our side, what happens if you try adding quarkus.class-loading.parent-first-artifacts=<group-id-of-your-commonds-module>? to your application.properties? I suspect it will just shift the classloading issue to the next dependency in the chain, but if you're feeling enthusiastic you can continue adding them all.

[mihaipoenaru]

Hello, @holly-cummins!

mvn quarkus:dev starts as normal. My main project behaves similarly, no issues running in dev mode
mvn verify produces the same error, I'll paste more of the stack trace at the end. In my main project and in the reproducer I used mvn install and got the same error.

I'll try your second suggestion but it will take 10/20 minutes. Let's see where the limits of my enthusiasm reach, hah!

-> mvn verify partial stack trace:

2023-06-19 11:56:25,868 INFO  [io.quarkus] (main) panache-reactive-reproducer 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.1.2.Final) started in 19.037s. Listening on: http://localhost:8124
2023-06-19 11:56:25,869 INFO  [io.quarkus] (main) Profile test activated. 
2023-06-19 11:56:25,869 INFO  [io.quarkus] (main) Installed features: [cdi, hibernate-orm, hibernate-reactive, hibernate-reactive-panache, hibernate-reactive-panache-kotlin, kotlin, reactive-pg-client, resteasy-reactive, res
teasy-reactive-jackson, smallrye-context-propagation, vertx]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 36.746 s <<< FAILURE! - in com.example.GreetingResourceTest
[ERROR] com.example.GreetingResourceTest.testHelloEndpoint  Time elapsed: 0.874 s  <<< ERROR!
java.lang.LinkageError: loader constraint violation: loader 'app' wants to load interface kotlin.jvm.functions.Function0. A different interface with the same name was previously loaded by io.quarkus.bootstrap.classloading.Qu
arkusClassLoader @4be12f6c. (kotlin.jvm.functions.Function0 is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @4be12f6c, parent loader 'app')
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        at org.junit.jupiter.api.AssertionsKt$sam$org_junit_jupiter_api_function_ThrowingSupplier$0.get(Assertions.kt)
        at org.junit.jupiter.api.AssertTimeout.assertTimeout(AssertTimeout.java:69)
        at org.junit.jupiter.api.AssertTimeout.assertTimeout(AssertTimeout.java:53)
        at org.junit.jupiter.api.Assertions.assertTimeout(Assertions.java:3328)
        at org.junit.jupiter.api.AssertionsKt.assertTimeout(Assertions.kt:219)

[holly-cummins]

Oh, sorry, @mihaipoenaru, I should have said 'mvn quarkus:devand thenrto run the tests. If it fails in normalmvn:verify` I'd be quite surprised if it would pass in dev mode, but computers are full of surprises. :)

[mihaipoenaru]

Ah oops! I was also confused initially, but Quarkus is pretty much black magic to me, so I don't question it 🤡

Hmmm... it's not exactly the same error, but still in the ballpark

2023-06-19 12:07:46,109 ERROR [io.qua.test] (Test runner thread) Test GreetingResourceTest#testHelloEndpoint() failed 
: java.lang.NoClassDefFoundError: kotlin/jvm/internal/Intrinsics
        at org.junit.jupiter.api.AssertionsKt.assertTimeout(Assertions.kt)
        at com.example.GreetingResourceTest.testHelloEndpoint(GreetingResourceTest.kt:13)
Caused by: java.lang.ClassNotFoundException: kotlin.jvm.internal.Intrinsics
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        ... 2 more

[mihaipoenaru]

Ok @holly-cummins, I tried with that property you suggested

quarkus.class-loading.parent-first-artifacts=com.orange.vp:vp-commons

It's now even worse. Pretty much all tests are failing.

nge/vp/handlers/mapper/shed/impl/ShedRequestMapper, and the class loader 'app' for the method's defining class, com/orange/vp/bo/utils/JsonUtilsKt, have different Class objects for the type com/fasterxml/jackson/databind/ObjectMapper used in the signature (com.orange.vp.handlers.mappe
r.shed.impl.ShedRequestMapper is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @6b9ce7a3, parent loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @4a0ee0ba; com.orange.vp.bo.utils.JsonUtilsKt is in unnamed module of loader 'app')

Interestingly, when I run using Intellij, it seems to be a bit better, as in not all tests are failing. When I try to run a test without @QuarkusTest I get an error like: Could not initialize class com.orange.vp.bo.utils.JsonUtilsKt, which is weird since it's a simple kotlin file, not an actual class (I know, however, that it does eventually get mapped to a class in the jvm). When I try to run a @QuarkusTest class, I get the same errors about the class loading.

[geoand]

@holly-cummins do you have any insights on what we need to do here?

[holly-cummins]

@holly-cummins do you have any insights on what we need to do here?

Umm, not good ones!

For @mihaipoenaru, as a workaround, another option is to make a local clone of the Quarkus 2-level Kotlin extension, and call it something like parent-first-kotlin, and depend on that instead of the main kotlin extension. It might work, although obviously it's not an ideal solution.

For us, it feels like we should get that reproducer into our integration tests (disabled, in the short term). That reproducer is so tiny it really ought to work and stay working in the future. Once we've got the reproducer running locally, we might get an insight into the solution.

I have a feeling that the solution is probably #34681.

[mihaipoenaru]

Thanks for the tip @holly-cummins!

We are not in a rush to migrate to 3 (although it's not low prio either), so maybe I'll wait for something official. In the meantime, about your workaround, why not just try and depend on an older version of the quarkus extension, instead of cloning it locally?

I'm not familiar with the inner workings of Quarkus, so I'm a bit confused with what exactly needs to be done.

[mihaipoenaru]

@holly-cummins any update on this? Quarkus 2 is cool and all, but 3 is my lucky number.

[gsmet]

FWIW, I just tested with latest 3.15.1 and this is still an issue.

What I'm not sure I understand is that we have quite a lot of Kotlin users and I have no idea why this very simple code would fail and why we wouldn't have a gazillion of people complaining.

[mihaipoenaru]

It is a bit suspicious, but to encounter this issue you need to be be writing quarkus tests specifically. Probably a lot of people just prefer to do straightforward unit tests and just mock the hell out of everything. Coupled with the fact that there's still a lot of people that don't really bother with testing, and that some people are scared to increment a major version (this works fine on quarkus 2) and maybe we shouldn't be too surprised.

This type of bug seems like a nightmare to debug tbh.

[geoand]

What I'm not sure I understand is that we have quite a lot of Kotlin users and I have no idea why this very simple code would fail and why we wouldn't have a gazillion of people complaining.

Yeah, I don't get it either....

[ivan-gomes]

FWIW I personally find this to be a high priority issue, have lost a number of hours trying to work around it, and characterize it as one of the roughest edges wrt. Kotlin+Quarkus. The magic incantation that ultimately worked around the problem for me specifically is to add the following to application.properties

%test.quarkus.class-loading.parent-first-artifacts=org.jetbrains.kotlin:kotlin-stdlib,org.jetbrains.kotlin:kotlin-reflect,...

where ... is additional libraries I use.

(Side note, adding it to application.test.properties does not work)

Because of it I lean away from using QuarkusTest whenever possible, despite finding it incredibly convenient and boilerplate reducing in non-Kotlin projects.

[geoand]

Please attach the simplest possible reproducer and we'll look into it when possible

[gsmet]

I was able to reproduce it easily with the instructions in How to Reproduce?. You create a project, you make sure the test class looks like the one provided and you get the error.

[geoand]

So fun fact:

import io.quarkus.test.junit.QuarkusTest
import org.junit.jupiter.api.Assertions
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.function.ThrowingSupplier
import java.time.Duration

@QuarkusTest
class GreetingResourceTest {

    @Test
    fun testHelloEndpoint() {
        Assertions.assertTimeout(Duration.ofSeconds(1), ThrowingSupplier {  })
    }
}

passes, while

import io.quarkus.test.junit.QuarkusTest
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertTimeout
import java.time.Duration

@QuarkusTest
class GreetingResourceTest {

    @Test
    fun testHelloEndpoint() {
        assertTimeout(Duration.ofSeconds(1)) {}
    }
}

fails.

The superficial difference is that the second piece of code forces the invocation of Kotlin code bundled in JUnit while the first one does not.

[gsmet]

Ah so that's probably why we didn't get that many reports.

[geoand]

Yes, the problem only manifests when the tests use the Kotlin code that JUnit ships.

Unfortunately there is no good way around this currently... JUnit obviously needs to be loaded by the app classloader while we use the normal ClassLoader hierachy for Kotlin.

[holly-cummins]

I've just checked on my local fork, and #34681 does not fix the problem (at least in its current implementation). However, thinking about it more, I think "load the tests with the classloader we use to run them" should resolve it.

So then the question is, what causes the gap between theory and reality? In my implementation, we have to pre-load the classes in order to have a look at them, so that might be what's the pollution of the parent classloader. I thought I'd done the pre-load in a way that doesn't impact the parent loader, but maybe there's a bug in that logic. I'll continue investigating.