native compilation : quarkus-jdbc-oracle with elasticsearch-java strange behaviour

[xrobert35]

Describe the bug

Hello,

I'm trying to compile in native mode my project using

• quay.io/quarkus/ubi-quarkus-graalvmce-builder-image:22.3-java17 as nativeBuilder
• quarkus 2.16.3.Final
• quarkus-jdbc-oracle
• quarkus-elasticsearch-rest-client
• elasticsearch-java

But it fail.

I can't figure out the problem. I've tried to add --initialize-at-run-time args but it doesn't change anything.

I'm already using elasticsearch-java in native mode (8.1.2 not 8.6.2) with some initialize-at-run-time and register for reflection it works but the problem seem to come from the jdbc-oracle.

Expected behavior

Compilation without error or at least not the oracle-jdbc one.

Actual behavior

With the actual configuration I get this error :

#15 766.0 Error: Classes that should be initialized at run time got initialized during image building:
#15 766.0  oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser the class was requested to be initialized at run time (from feature io.quarkus.runner.Feature.beforeAnalysis with 'BlockSource$ThreadedCachingBlockSource$BlockReleaser.class'). oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource caused initialization of this class with the following trace:

More detail stack on the project readme : https://github.com/xrobert35/quarkus-oracle-jdbc-error-example/blob/master/README.md

How to Reproduce?

I've made a simplified project to reproduce

Step 1.

https://github.com/xrobert35/quarkus-oracle-jdbc-error-example
docker build -t <name> . --no-cache

Output of uname -a or ver

docker windows or WSL docker

Output of java -version

Java Version 17.0.6+10-jvmci-22.3-b13

GraalVM version (if different from Java)

22.3

Quarkus version or git rev

2.16.3.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Docker

Additional information

Some strange behaviour

• If you remove the use of ElasticsearchClient (including the import) in TestService the compilation will work. (So I suppose the real problem could be the creation of the ElasticsearchClient (in the TestOracleJDBCConfig file), but :
• If you change quarkus-jdbc-oracle to quarkus-jdbc-postgresql and change to db-king to postgresql and I let the ElasticsearchClient the compilation will work

[quarkus-bot]

/cc @gsmet (elasticsearch), @loicmathieu (elasticsearch), @yrodiere (elasticsearch)

[loicmathieu]

Hi,

elasticsearch-java is not compatible with Quarkus 2.16 as it uses the new Jakarta EE packages.
You need to wait for Quarkus 3 that will switch to more recent version of Jakarata EE and also will provide an extension for the Elasticsearch Java Client (see #2262).

[gsmet]

Note that you can already test Quarkus 3.0.0.Alpha4, which should be fairly stable.

[gsmet]

And... in fact, I would be interested if you could do it and use the new Elasticsearch Java client and check if your problem still happens.

[xrobert35]

Ok i'm trying to move to 3.0.0.Alpha4 now.

Also the conclusion seem a bit "fast", why compilation work with quarkus-jdbc-postgresql but not with quarkus-jdbc-oracle ?
Futhermore i've projet running in native quarkus 2.10.1 and elasticsearch java 8.1.2 (which include jakarta also) without problem.

[xrobert35]

Same problem using 3.0.0.Alpha4 with quarkus-jdbc-oracle
Compilation success with quarkus-jdbc-postgresql

i've made a branch with the 3.0.0-Alpha4 : https://github.com/xrobert35/quarkus-oracle-jdbc-error-example/tree/chore/3.0.0-Alpha

[gsmet]

Yeah, that was my intuition that you would have the same issue.

[gsmet]

So you haven't used the Quarkus extension, could you please apply this patch to your reproducer?

diff --git a/pom.xml b/pom.xml
index 6e19606..b3ca1da 100644
--- a/pom.xml
+++ b/pom.xml
@@ -52,12 +52,7 @@
         </dependency>
         <dependency>
             <groupId>io.quarkus</groupId>
-            <artifactId>quarkus-elasticsearch-rest-client</artifactId>
-        </dependency>
-        <dependency>
-            <groupId>co.elastic.clients</groupId>
-            <artifactId>elasticsearch-java</artifactId>
-            <version>8.6.2</version>
+            <artifactId>quarkus-elasticsearch-java-client</artifactId>
         </dependency>
         <dependency>
             <groupId>org.projectlombok</groupId>
diff --git a/src/main/java/org/test/jdbcoracle/config/TestOracleJDBCConfig.java b/src/main/java/org/test/jdbcoracle/config/TestOracleJDBCConfig.java
index ced93be..ed3189e 100644
--- a/src/main/java/org/test/jdbcoracle/config/TestOracleJDBCConfig.java
+++ b/src/main/java/org/test/jdbcoracle/config/TestOracleJDBCConfig.java
@@ -31,12 +31,6 @@ public class TestOracleJDBCConfig {
     @Inject
     ObjectMapper mapper;
 
-    @Singleton
-    public ElasticsearchClient initClient(RestClient client){
-        ElasticsearchTransport transport = new RestClientTransport(client, new JacksonJsonpMapper(mapper));
-        return new ElasticsearchClient(transport);
-    }
-
     void onStart(@Observes StartupEvent ev) {         
         this.repository.findAll();
         new ElasticsearchClient(null);
@@ -47,4 +41,4 @@ public class TestOracleJDBCConfig {
         }
     }
 
-}
\ No newline at end of file
+}

That doesn't solve your problem though.
My guess is that having Elasticsearch Java Client + the Oracle client somehow triggers an incompatibility issue.

FWIW, the error we have is:

Warning: Feature class oracle.nativeimage.NativeImageFeature is annotated with the deprecated annotation @AutomaticFeature. Support for this annotation will be removed in a future version of GraalVM. Applications should register a feature using the option --features=oracle.nativeimage.NativeImageFeature
[1/7] Initializing...                                                                                   (13.3s @ 0.65GB)
 Version info: 'GraalVM 22.3.0 Java 17 CE'
 Java version info: '17.0.5+8-jvmci-22.3-b08'
 C compiler: gcc (redhat, x86_64, 11.3.1)
 Garbage collector: Serial GC
 10 user-specific feature(s)
 - io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
 - io.quarkus.hibernate.orm.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.Version, org.hibernate.annotations.common.Version, org.hibernate.dialect.Dialect] categories
 - io.quarkus.hibernate.validator.runtime.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.hibernate.validator.internal.util.Version] categories
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase for the [org.jboss.threads] categories
 - io.quarkus.runtime.graal.ResourcesFeature: Register each line in META-INF/quarkus-native-resources.txt as a resource on Substrate VM
 - oracle.nativeimage.NativeImageFeature
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
 - org.hibernate.graalvm.internal.GraalVMStaticFeature: Hibernate ORM's static reflection registrations for GraalVM
 - org.hibernate.graalvm.internal.QueryParsingSupport: Hibernate ORM's support for HQL Parser in GraalVM
The bundle named: com/sun/rowset/RowSetResourceBundle, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
[2/7] Performing analysis...  [*******]                                                                 (48.2s @ 2.28GB)
  19,950 (89.51%) of 22,287 classes reachable
  30,619 (60.71%) of 50,431 fields reachable
  98,198 (53.42%) of 183,836 methods reachable
     732 classes,   402 fields, and 3,299 methods registered for reflection
      70 classes,    91 fields, and    56 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/7] Building universe...                                                                               (5.5s @ 2.27GB)
[4/7] Parsing methods...      [***]                                                                      (5.4s @ 1.35GB)
[5/7] Inlining methods...     [****]                                                                     (2.7s @ 2.04GB)
[6/7] Compiling methods...    [******]                                                                  (37.2s @ 2.65GB)
[7/7] Creating image...
                                                                                  (0.0s @ 2.42GB)
Error: Classes that should be initialized at run time got initialized during image building:
 oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource the class was requested to be initialized at run time (from feature io.quarkus.runner.Feature.beforeAnalysis with 'BlockSource$ThreadedCachingBlockSource.class'). oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource caused initialization of this class with the following trace: 
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource.<clinit>(BlockSource.java:288)
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaserListener.handleNotification(BlockSource.java:435)
	at sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:155)
	at sun.management.MemoryImpl.createNotification(MemoryImpl.java:161)
	at sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:326)
	at sun.management.Sensor.trigger(Sensor.java:134)

oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser the class was requested to be initialized at run time (from feature io.quarkus.runner.Feature.beforeAnalysis with 'BlockSource$ThreadedCachingBlockSource$BlockReleaser.class'). oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource caused initialization of this class with the following trace: 
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.<clinit>(BlockSource.java:313)
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource.<clinit>(BlockSource.java:454)


com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
 oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource the class was requested to be initialized at run time (from feature io.quarkus.runner.Feature.beforeAnalysis with 'BlockSource$ThreadedCachingBlockSource.class'). oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource caused initialization of this class with the following trace: 
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource.<clinit>(BlockSource.java:288)
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaserListener.handleNotification(BlockSource.java:435)
	at sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:155)
	at sun.management.MemoryImpl.createNotification(MemoryImpl.java:161)
	at sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:326)
	at sun.management.Sensor.trigger(Sensor.java:134)

oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser the class was requested to be initialized at run time (from feature io.quarkus.runner.Feature.beforeAnalysis with 'BlockSource$ThreadedCachingBlockSource$BlockReleaser.class'). oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource caused initialization of this class with the following trace: 
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.<clinit>(BlockSource.java:313)
	at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource.<clinit>(BlockSource.java:454)


	at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:73)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ProvenSafeClassInitializationSupport.checkDelayedInitialization(ProvenSafeClassInitializationSupport.java:273)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationFeature.afterImageWrite(ClassInitializationFeature.java:244)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$doRun$8(NativeImageGenerator.java:724)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:85)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:724)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:535)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:403)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:580)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)
------------------------------------------------------------------------------------------------------------------------
                       13.6s (9.0% of total time) in 166 GCs | Peak RSS: 4.96GB | CPU load: 6.69
------------------------------------------------------------------------------------------------------------------------

@zakkak does it ring any bell

[yrodiere]

It's possible that the Elasticsearch REST client brings in a dependency that the Oracle JDBC driver depends on optionally, and when the Oracle JDBC driver detects it in the classpath, it behaves differently for some reason. We had that kind of things happen in the past (with XML parsers, I think?).

Determining what the problematic dependency is will require some analysis of what's going on; though. I guess the Oracle JDBC driver is still closed-source, possibly obfuscated?

[xrobert35]

@gsmet sorry i didn't know about quarkus-elasticsearch-java-client I thought you were speaking about the normal one.
Example up to date. But yes same error

[zakkak]

@zakkak does it ring any bell

No. It looks like when using quarkus-elasticsearch-java-client, sun.management.Sensor.trigger becomes reachable and that causes the build-time initialization of oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser which we want to be initialized at run time:

quarkus/extensions/jdbc/jdbc-oracle/deployment/src/main/java/io/quarkus/jdbc/oracle/deployment/OracleMetadataOverrides.java

Lines 98 to 99 in ccf3df0

	runtimeInitialized.produce(new RuntimeInitializedClassBuildItem(
	"oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser"));

which for some reason we also register for runtime initialization in:

quarkus/extensions/jdbc/jdbc-oracle/deployment/src/main/java/io/quarkus/jdbc/oracle/deployment/OracleNativeImage.java

Line 41 in ccf3df0

.produce(new RuntimeInitializedClassBuildItem("oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource"));

My first try would be to register oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser for runtime reinitialization instead of runtime initialization. If this doesn't work I think the next step would be to figure out why/how sun.management.Sensor.trigger is becoming reachable and see whether we can avoid it.

[xrobert35]

Is there something I can try at my level ? Is there some kind of parameter like --reinitialize-at-run-time ? , can't figure out

[yrodiere]

Is there something I can try at my level ? Is there some kind of parameter like --reinitialize-at-run-time ? , can't figure out

I don't think you can since we also need to remove something in Quarkus.

I'll try to prepare a PR following @zakkak's instructions and will test against your reproducer, but if that doesn't work I'll probably be out of my depth and request help from someone :)

[zakkak]

FTR I provide a bit more context about why reinitialization fixes this in #31770 (comment)