UnsupportedClassVersionError: Preview features seem to not be enabled in fallback despite passing --enable-preview to native-image

[pjonsson]

Describe the issue
When passing --enable-preview to native-image, the fallback still seems to run without the parameter because I get an exception for faulty class versions, as shown in the logs below.

The code being compiled is heavily reliant on reflection-like mechanisms for historical reasons.

Steps to reproduce the issue
On Linux with bash:

1. git clone --depth 1 https://github.com/contiki-ng/contiki-ng
2. cd contiki-ng/tools
3. rmdir cooja
4. git clone --branch native-image-enable-preview https://github.com/pjonsson/cooja
5. cd ..
6. SRCPATH=$(pwd)
7. docker pull contiker/contiki-ng:latest
8. docker run --rm --mount type=bind,source=$SRCPATH,destination=/home/user/contiki-ng -e DISPLAY=:0 -v /tmp/.X11-unix:/tmp/.X11-unix -v /dev/bus/usb:/dev/bus/usb -ti contiker/contiki-ng
9. (Now inside container, prompt looks similar to user@2f67272ea2e9:~/contiki-ng)
10. cd tests/07-simulation-base
11. ../../tools/cooja/gradlew --info -p ../../tools/cooja nativeRun

The display forwarding from docker to the host must work to reproduce the issue, headless mode takes completely different code paths.

Describe GraalVM and your environment:

• GraalVM version: Hit [GR-51156] Return value of "java.util.jar.JarFile.getEntry(String)" is null #8096 with the latest snapshot build, so this is with the GraalVM 21.0.1 release.
• JDK major version: 21
• OS: Ubuntu 22.04 docker image inside Ubuntu 22.04
• Architecture: AMD64

More details
Excerpt from Gradle build output when passing --info:

Successfully started process 'command '/usr/local/bin/native-image''
[native-image-plugin] GraalVM Toolchain detection is enabled
[native-image-plugin] GraalVM uses toolchain detection. Selected:
[native-image-plugin]    - language version: 21
[native-image-plugin]    - vendor: Oracle
[native-image-plugin]    - runtime version: 21.0.1+12-jvmci-23.1-b19
[native-image-plugin] Native Image executable path: /usr/local/lib/svm/bin/native-image
Starting process 'command '/usr/local/bin/native-image''. Working directory: /home/user/contiki-ng/tools/cooja/build/native/nativeCompile Command: /usr/local/bin/native-image -cp /home/user/contiki-ng/tools/cooja/build/libs/cooja-full.jar -o /home/user/contiki-ng/tools/cooja/build/native/nativeCompile/cooja -H:ConfigurationFileDirectories=/home/user/contiki-ng/tools/cooja/build/native/generated/generateResourcesConfigFile,/home/user/.gradle/native-build-tools/repositories/e6cb96d180d876be3270cd292aa5952a3c312ef0/exploded/ch.qos.logback/logback-classic/1.4.1,/home/user/contiki-ng/tools/cooja/java --enable-preview --strict-image-heap --report-unsupported-elements-at-runtime --initialize-at-build-time=org.slf4j.LoggerFactory,ch.qos.logback --trace-class-initialization=ch.qos.logback.classic.Logger --trace-object-instantiation=ch.qos.logback.core.AsyncAppenderBase$Worker --enable-native-access=ALL-UNNAMED -J-Xmx10G org.contikios.cooja.Main
Successfully started process 'command '/usr/local/bin/native-image''
================================================================================

So native-image is invoked with --enable-preview. The log from running looks as expected, besides the "EDT thread call failed" and the exception:

> Task :nativeRun
Caching disabled for task ':nativeRun' because:
  Build cache is disabled
Task ':nativeRun' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
Starting process 'command '/home/user/contiki-ng/tools/cooja/build/native/nativeCompile/cooja''. Working directory: /home/user/contiki-ng/tools/cooja Command: /home/user/contiki-ng/tools/cooja/build/native/nativeCompile/cooja /home/user/contiki-ng/tests/07-simulation-base/23-rpl-tsch-z1.csc
Successfully started process 'command '/home/user/contiki-ng/tools/cooja/build/native/nativeCompile/cooja''
INFO  [main] [Cooja.java:1402] - EDT thread call failed
java.lang.reflect.InvocationTargetException: null
        at java.desktop/java.awt.EventQueue.invokeAndWait(EventQueue.java:1371)
        at java.desktop/java.awt.EventQueue.invokeAndWait(EventQueue.java:1346)
        at org.contikios.cooja.Cooja$RunnableInEDT.invokeAndWait(Cooja.java:1398)
        at org.contikios.cooja.Cooja.makeCooja(Cooja.java:172)
        at org.contikios.cooja.Cooja.go(Cooja.java:1135)
        at org.contikios.cooja.Main.main(Main.java:312)
Caused by: java.lang.UnsupportedClassVersionError: Preview features are not enabled for org/contikios/cooja/contikimote/ContikiMoteType (class file version 65.65535). Try running with '--enable-preview'
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
        at org.contikios.cooja.Cooja.registerClasses(Cooja.java:416)
        at org.contikios.cooja.Cooja.parseProjectConfig(Cooja.java:407)
        at org.contikios.cooja.GUI.parseProjectConfig(GUI.java:1138)
        at org.contikios.cooja.Cooja.<init>(Cooja.java:217)
        at org.contikios.cooja.Cooja$1.work(Cooja.java:167)
        at org.contikios.cooja.Cooja$1.work(Cooja.java:162)
        at org.contikios.cooja.Cooja$RunnableInEDT.lambda$invokeAndWait$0(Cooja.java:1398)
        at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:308)
        at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:773)
        at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:720)
        at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:714)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
        at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
        at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
        at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
        at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
        at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
        at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
        at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
        at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
INFO  [main] [Cooja.java:1145] - Loading /home/user/contiki-ng/tests/07-simulation-base/23-rpl-tsch-z1.csc random seed: null
INFO  [AWT-EventQueue-0] [LogScriptEngine.java:231] - Script timeout in 360000 ms

[fniephaus]

the fallback still seems to run without the parameter because I get an exception for faulty class versions

What do you mean by fallback? Does Native Image generate a fallback image for your application? What error do you get when you build with --no-fallback?

[pjonsson]

the fallback still seems to run without the parameter because I get an exception for faulty class versions

What do you mean by fallback? Does Native Image generate a fallback image for your application? What error do you get when you build with --no-fallback?

Building with fallback = false in build.gradle gives a few pages of errors about initializing various parts of slf4j, output pasted below.

Even if that worked, I would still end up in SymbolLookup.libraryLookup().find() in the end, and that isn't supported by native-image as far as I know.

Error: Unsupported features in 2 methods
Detailed message:
Error: An object of type 'org.slf4j.helpers.NOP_FallbackServiceProvider' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.
------------------------------------------------------------------------------------------------------------------------
This is not allowed for correctness reasons: All objects that are stored in the image heap must be initialized at build time.

You now have two options to resolve this:

1) If it is intended that objects of type 'org.slf4j.helpers.NOP_FallbackServiceProvider' are persisted in the image heap, add 

    '--initialize-at-build-time=org.slf4j.helpers.NOP_FallbackServiceProvider'

to the native-image arguments. Note that initializing new types can store additional objects to the heap. It is advised to check the static fields of 'org.slf4j.helpers.NOP_FallbackServiceProvider' to see if they are safe for build-time initialization,  and that they do not contain any sensitive data that should not become part of the image.

2) If these objects should not be stored in the image heap, you can use 

    '--trace-object-instantiation=org.slf4j.helpers.NOP_FallbackServiceProvider'

to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 

    '--initialize-at-run-time=<culprit>'

to prevent the instantiation of the object.

If you are seeing this message after enabling '--strict-image-heap', this means that some objects ended up in the image heap without their type being marked with --initialize-at-build-time.
To fix this, include '--initialize-at-build-time=org.slf4j.helpers.NOP_FallbackServiceProvider' in your configuration. If the classes do not originate from your code, it is advised to update all library or framework dependencies to the latest version before addressing this error.
Please address this problem to be prepared for future releases of GraalVM.

The following detailed trace displays from which field in the code the object was reached.
Trace: Object was reached by
  trying to constant fold static field org.slf4j.LoggerFactory.NOP_FALLBACK_SERVICE_PROVIDER
    at org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:498)
  parsing method org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:486) reachable via the parsing context
    at static root method.(Unknown Source)

Error: An object of type 'org.slf4j.helpers.SubstituteServiceProvider' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.
This is not allowed for correctness reasons: All objects that are stored in the image heap must be initialized at build time.

You now have two options to resolve this:

1) If it is intended that objects of type 'org.slf4j.helpers.SubstituteServiceProvider' are persisted in the image heap, add 

    '--initialize-at-build-time=org.slf4j.helpers.SubstituteServiceProvider'

to the native-image arguments. Note that initializing new types can store additional objects to the heap. It is advised to check the static fields of 'org.slf4j.helpers.SubstituteServiceProvider' to see if they are safe for build-time initialization,  and that they do not contain any sensitive data that should not become part of the image.
                       4.4s (17.2% of total time) in 109 GCs | 
Peak RSS: 4.48GB | CPU load: 11.47
2) If these objects should not be stored in the image heap, you can use 

    '--trace-object-instantiation=org.slf4j.helpers.SubstituteServiceProvider'
========================================================================================================================

Finished generating 'cooja' in 24.8s.to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 


    '--initialize-at-run-time=<culprit>'

to prevent the instantiation of the object.

If you are seeing this message after enabling '--strict-image-heap', this means that some objects ended up in the image heap without their type being marked with --initialize-at-build-time.
To fix this, include '--initialize-at-build-time=org.slf4j.helpers.SubstituteServiceProvider' in your configuration. If the classes do not originate from your code, it is advised to update all library or framework dependencies to the latest version before addressing this error.
Please address this problem to be prepared for future releases of GraalVM.

The following detailed trace displays from which field in the code the object was reached.
Trace: Object was reached by
  trying to constant fold static field org.slf4j.LoggerFactory.SUBST_PROVIDER
    at org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:504)
  parsing method org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:486) reachable via the parsing context
    at static root method.(Unknown Source)



> Task :nativeCompile FAILED

[pjonsson]

@fniephaus Sorry, I might have misinterpreted your comment, are you saying --enable-preview is not supported when a fallback image is built?

[fniephaus]

Building with fallback = false in build.gradle gives a few pages of errors about initializing various parts of slf4j, output pasted below.

Ok, so this means that your app cannot be turned into a native image. Instead, a fallback image is created, which is essentially a thin launcher that runs your app on HotSpot. I'm not sure this is what you want?

Sorry, I might have misinterpreted your comment, are you saying --enable-preview is not supported when a fallback image is built?

When --enable-preview is passed into a build, preview features should be available, even in the fallback image (as you said). So this could be a bug in the generation of fallback images.

Even if that worked, I would still end up in SymbolLookup.libraryLookup().find() in the end, and that isn't supported by native-image as far as I know.

According to this, I think you are right.

[pjonsson]

Ok, so this means that your app cannot be turned into a native image. Instead, a fallback image is created, which is essentially a thin launcher that runs your app on HotSpot. I'm not sure this is what you want?

My (unfounded) impression was that native-image would make a binary that had as much as possible as machine code, and then insert code to switch to executing on the JVM for some suitably selected part of the program. I realize the hand waving aspect of the terms "as much as possible" and "suitably selected".

So you are right, a fallback image is not really what I want, and I wasn't expecting the "suitably selected" part of the program to be 100%. But being able to run ./gradlew nativeRun would make it possible to merge various bits and pieces (build support, etc) to main, which would make it easier for others to contribute, so having this issue solved would improve the situation.

Our build.gradle is fairly standard as far as I know, so my guess is that others who use preview (and incubator?) features could also be affected by this issue.

[fniephaus]

My (unfounded) impression was that native-image would make a binary that had as much as possible as machine code, and then insert code to switch to executing on the JVM for some suitably selected part of the program. I realize the hand waving aspect of the terms "as much as possible" and "suitably selected".

We are working on some ideas that might allow to do this in the future, but currently, a fallback image simply calls out to HotSpot. Here's the relevant code in case you are interested:

graal/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/FallbackExecutor.java

Lines 46 to 55 in c069de7

	* This class is used to generate fallback images in case we are unable to build standalone images.
	*
	* A fallback image is a trivial standalone image that delegates execution of the application that
	* should originally be built to calling the Java executable with the original image classpath and
	* mainClass. System-properties specified during the original image-build get passed to the Java
	* executable that the FallbackExecutor uses to run the application.
	*
	* Control gets transferred to the Java executable by using {code}ProcessProperties.exec(){code}.
	* This ensures that the fallback image behaves as if the original application was started as
	* regular Java application.

Our build.gradle is fairly standard as far as I know, so my guess is that others who use preview (and incubator?) features could also be affected by this issue.

Sure, that can very well be the case. We are still working on the Foreign Function & Memory API for Native Image.

[fniephaus]

FYI, we just added #8113 for the FFM API in Native Image to the roadmap (in case you'd like to stay updated).