Support running JUnit 5 tests as native image

[sdeleuze]

@aclement has recently demonstrated it is possible to run a subset of Spring Framework unit tests as native images. The purpose of this issue is to start from that POC to experiment and identify the next steps toward a more robust and mature arrangement that would allow to run a subset of Spring Framework and Spring Boot JUnit 5 tests as a native image in order to:

• Identify in an automated fashion what part of Spring ecosystem works as native image and what part does not
• Provide regression testing for each update of Spring Framework, Spring Boot and GraalVM native.
• Provide a way for end user Spring Boot applications to be tested
• Provide a way to test GraalVM native support for libraries that support it natively like Netty or Tomcat

Goal of this issue:

• Provide a way to run these tests via command line/Gradle/Maven
• Provide a way to identify tests that will be ran as native images and those who will not because for example by design they can't due to native images limitations (configuration whitelisting or backlisting, support for conditional execution via annotations and ExecutionCondition).
• Is JUnit console launcher this right way to support it or not?
• What kind of output are we expecting? Could we generate similar output than JUnit 5 Maven/Gradle regular OpenJDK plugins?
• Create GraalVM native support for JUnit 5 via a feature or JSON configuration files (the related module can live in spring-graal-native for now)
• Allow to run tests locally (using java and native-image currently in the path, or in a container)
• Explore with @bsideup the opportunity of leveraging Testcontainers for testing in containers
• Check if that's interesting to use GraalVM capability to create shared libraries + calling them from Java code
• Provide a dedicated wiki page.

Non-goal of this issue:

• Integration in IDEs
• JUnit 4 support

[sbrannen]

Current work on the POC can be seen in the following feature branch: https://github.com/sbrannen/spring-framework/commits/graalvm-testing

Status Quo:

For spring-core, we are successfully executing 2301 out of 3272 tests in that module (so, approximately 70%).

At the moment we are often excluding entire test classes if a single contained test method cannot execute within a native image. Later iterations will improve on this to increase test coverage.

Work will continue solely with spring-core until we have automated the process (i.e., moved away from custom shell scripts).

[sbrannen]

Note that Log4J will eventually provide out-of-the-box GraalVM native image configuration.

Consult the Generate reflect-config.json for GraalVM during annotation processing issue for details.

See also: oracle/graal#808

[sbrannen]

Update

Current work can be viewed here: spring-projects/spring-framework@master...sbrannen:graalvm-testing

For spring-core, we are successfully executing 2661 out of 3272 tests in that module (so, approximately 81%). The percentage of tests executed within the native image will likely not increase significantly at this point.

The list of tests to execute within the native image is no longer hand crafted. Instead, the new TestClassTrackingTestExecutionListener generates a list of test classes that were executed using the standard Gradle test task with certain tags excluded. Thus, the resulting list has already been filtered when we execute the tests with the GraalVM agent and within the compiled native image.

In addition, we no longer need to manually download the junit-platform-console-standalone-1.6.0.jar since we now have a testRuntime dependency on org.junit.platform:junit-platform-console in the Gradle build.

[sbrannen]

The gradle-graal plugin cannot be used for recent GraalVM versions. See palantir/gradle-graal#239 for details.

[sbrannen]

Update

As of the latest changes in the feature branch, unit testing within the spring-core module is now automated with custom Gradle tasks.

Example

Executing the following within the spring-core folder:

../gradlew --offline --no-build-cache -DnativeImageTesting=true clean testInNativeImage

Results in output similar to the following:

[spring-core (graalvm-testing)]$ ../gradle --offline --no-build-cache -DnativeImageTesting=true clean testInNativeImage

> Task :spring-core:testWithGraalVmAgent
WARNING: Unsupported JNI function DefineClass used to load class org/graalvm/compiler/hotspot/management/libgraal/runtime/SVMHotSpotGraalRuntimeMBean
WARNING: Unsupported JNI function DefineClass used to load class org/graalvm/compiler/hotspot/management/libgraal/runtime/SVMHotSpotGraalRuntimeMBean$Factory
WARNING: Unsupported JNI function DefineClass used to load class org/graalvm/compiler/hotspot/management/libgraal/runtime/SVMHotSpotGraalRuntimeMBean$PushBackIterator
WARNING: Unsupported JNI function DefineClass used to load class org/graalvm/compiler/hotspot/management/libgraal/runtime/HotSpotToSVMCalls
WARNING: Unsupported JNI function DefineClass used to load class org/graalvm/compiler/hotspot/management/libgraal/runtime/SVMToHotSpotEntryPoints
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated

Thanks for using JUnit! Support its development at https://junit.org/sponsoring


Test run finished after 13142 ms
[       210 containers found      ]
[         6 containers skipped    ]
[       204 containers started    ]
[         0 containers aborted    ]
[       204 containers successful ]
[         0 containers failed     ]
[      2734 tests found           ]
[        73 tests skipped         ]
[      2661 tests started         ]
[         0 tests aborted         ]
[      2661 tests successful      ]
[         0 tests failed          ]


> Task :spring-core:buildNativeImageForTests
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]    classlist:  14,095.53 ms,  2.69 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]        (cap):   1,889.06 ms,  3.00 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]        setup:   3,611.13 ms,  3.00 GB
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated
warning: unknown locality of class Lkotlin/coroutines/CoroutineContext$plus$1;, assuming class is not local. To remove the warning report an issue to the library or language author. The issue is caused by Lkotlin/coroutines/CoroutineContext$plus$1; which is not following the naming convention.
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]   (typeflow):  53,445.58 ms,  7.62 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]    (objects):  59,257.92 ms,  7.62 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]   (features):   9,201.08 ms,  7.62 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]     analysis: 129,154.07 ms,  7.62 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]     (clinit):   2,111.14 ms,  7.62 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]     universe:   4,538.83 ms,  7.62 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]      (parse):   4,555.33 ms,  7.66 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]     (inline):   4,106.52 ms,  6.93 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]    (compile):  29,682.39 ms,  6.64 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]      compile:  41,989.92 ms,  6.64 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]        image:   8,514.74 ms,  6.58 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]        write:   2,054.82 ms,  6.58 GB
[/Users/developer/source/spring-framework/spring-core/build/native-image-tests.bin:85418]      [total]: 204,388.32 ms,  6.58 GB

> Task :spring-core:testInNativeImage
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated

Thanks for using JUnit! Support its development at https://junit.org/sponsoring


Test run finished after 7166 ms
[       204 containers found      ]
[         0 containers skipped    ]
[       204 containers started    ]
[         0 containers aborted    ]
[       204 containers successful ]
[         0 containers failed     ]
[      2699 tests found           ]
[        38 tests skipped         ]
[      2661 tests started         ]
[         0 tests aborted         ]
[      2661 tests successful      ]
[         0 tests failed          ]


BUILD SUCCESSFUL in 4m 11s
26 actionable tasks: 20 executed, 6 up-to-date
A build scan will not be published due to this build running offline.

[sbrannen]

With the latest commits to the feature branch, tests in spring-beans now execute within a native image.

> Task :spring-beans:testInNativeImage
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated
ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.oracle.truffle.js.scriptengine.GraalJSEngineFactory could not be instantiated
15:15:07.560 [main] WARN  o.s.b.f.c.YamlMapFactoryBean - Could not load map from non-existent: planned
15:15:07.562 [main] WARN  o.s.b.f.c.YamlMapFactoryBean - Could not load map from file [/Users/sbrannen/source/spring-framework/spring-beans/non-exsitent-file.yml]: non-exsitent-file.yml
15:15:07.566 [main] WARN  o.s.b.f.c.YamlPropertiesFactoryBean - Could not load map from class path resource [no-such-file.yml]: class path resource [no-such-file.yml] cannot be opened because it does not exist

Thanks for using JUnit! Support its development at https://junit.org/sponsoring


Test run finished after 966 ms
[       101 containers found      ]
[         2 containers skipped    ]
[        99 containers started    ]
[         1 containers aborted    ]
[        98 containers successful ]
[         0 containers failed     ]
[      1096 tests found           ]
[        68 tests skipped         ]
[      1025 tests started         ]
[         0 tests aborted         ]
[      1025 tests successful      ]
[         0 tests failed          ]

That equates to 1025 out of 1484 tests (i.e., 69%).

[sbrannen]

The gradle-graal plugin cannot be used for recent GraalVM versions. See palantir/gradle-graal#239 for details.

This has been fixed in https://github.com/palantir/gradle-graal/releases/tag/0.6.1