Incorrect / incomplete documentation regarding junit-platform-launcher incompatibilities

[Marcono1234]

Describe the bug
The documentation at https://graalvm.github.io/native-build-tools/latest/maven-plugin.html#testing-support-version-compatibility currently says that for old Surefire plugin versions it is necessary to add an explicit junit-platform-launcher dependency, but for newer versions this is not necessary.

However, this information might be incorrect or incomplete. It seems this Native Maven Plugin has itself a dependency on junit-platform-launcher and is accessing its classes through org.graalvm.junit.platform.JUnitPlatformFeature#discoverTestsAndRegisterTestClassesForReflection, without any involvement of Surefire (?).

For example it seems with JUnit 5.12.0 and native-maven-plugin 0.10.5 and 0.10.6 the step "Running in 'test listener' mode ..." fails with:

[junit-platform-native] Running in 'test listener' mode using files matching pattern [junit-platform-unique-ids*] found in folder [...\gson\test-graal-native-image\target\test-ids] and its subfolders.

Error: Feature defined by org.graalvm.junit.platform.JUnitPlatformFeature unexpectedly failed with a(n) org.junit.platform.commons.JUnitException. Please report this problem to the authors of org.graalvm.junit.platform.JUnitPlatformFeature.
com.oracle.svm.core.util.UserError$UserException: Feature defined by org.graalvm.junit.platform.JUnitPlatformFeature unexpectedly failed with a(n) org.junit.platform.commons.JUnitException. Please report this problem to the authors of org.graalvm.junit.platform.JUnitPlatformFeature.
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:87)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.handleFeatureError(FeatureHandler.java:293)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:92)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:774)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:593)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:551)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
Caused by: org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-jupiter' failed to discover tests
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:160)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverSafely(EngineDiscoveryOrchestrator.java:132)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:107)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:78)
        at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:99)
        at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:77)
        at org.junit.platform.launcher.core.DelegatingLauncher.discover(DelegatingLauncher.java:42)
        at org.junit.platform.launcher.core.SessionPerRequestLauncher.discover(SessionPerRequestLauncher.java:56)
        at org.graalvm.junit.platform.JUnitPlatformFeature.discoverTestsAndRegisterTestClassesForReflection(JUnitPlatformFeature.java:135)
        at org.graalvm.junit.platform.JUnitPlatformFeature.beforeAnalysis(JUnitPlatformFeature.java:94)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$9(NativeImageGenerator.java:774)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
        ... 7 more
Caused by: org.junit.platform.commons.JUnitException: OutputDirectoryProvider not available; probably due to unaligned versions of the junit-platform-engine and junit-platform-launcher jars on the classpath/module path.
        at org.junit.platform.engine.EngineDiscoveryRequest.getOutputDirectoryProvider(EngineDiscoveryRequest.java:94)
        at org.junit.jupiter.engine.JupiterTestEngine.discover(JupiterTestEngine.java:67)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:152)
        ... 18 more

(requires -H:+ReportExceptionStackTraces to see the stack trace, see #541 (comment))

To Reproduce

1. Clone google/gson@832382f
2. Set up GraalVM on PATH and JAVA_HOME
3. Run

   mvn test --activate-profiles native-image-test --projects test-graal-native-image --also-make

If you want I can also try to create a simplified example.

Expected behavior

• Ideally the JUnitException should not occur in the first place
  The Native Maven Plugin should on its own determine the necessary junit-platform-launcher version.
• If that is not possible, please adjust the documentation to mention that it might also be necessary to add a junit-platform-launcher dependency to solve incompatibilities with the Native Maven Plugin, and not only Surefire
• Consider enabling -H:+ReportExceptionStackTraces by default
  Without stack traces troubleshooting the original error message is impossible, and finding out about the -H:+ReportExceptionStackTraces argument might also not be easy if it is not clear to the user what and where to look for.

Logs
see stack trace above

System Info (please complete the following information):

• OS: Windows 10
• GraalVM Version: 21.0 CE
• Java Version: 21 (Oracle GraalVM 21.0.6+8.1)
• Plugin version: native-maven-plugin:0.10.6

Additional context
none

[Marcono1234]

Maybe related to:

• Classpath is duplicated in native tests with incompatible versions, preventing upgrade to newer JUnit versions #305

[sbrannen]

Indeed, this is effectively a duplicate of #305 and #541; however, this issue provides further insight about the (current) need to include a dependency on junit-platform-launcher.

See also: #541 (comment)

So, I think we should leave this issue open until the team has decided to address #305.