`JUnitPlatformFeature` unexpectedly failed with a `JUnitException`

[danthe1st]

Before reporting

• This repository should be used to report issues on the Maven or Gradle plugins for GraalVM.
• Please report issues which are specific to the Spring Framework or the Micronaut framework to their specific repositories.
• Do not report issues with building your specific application, e.g errors which happen at image build time like classes initialized at build time, or missing classes as run time: those are not related to the plugins but problems with configuration. You can refer to the GraalVM native image documentation for available options and the plugins documentation for how to use them with the plugin.

Describe the bug

When overriding an @Test method and without marking the overridden method with @Test, compilation of the test fails with Error: Feature defined by org.graalvm.junit.platform.JUnitPlatformFeature unexpectedly failed with a(n) org.junit.platform.commons.JUnitException. Please report this problem to the authors of org.graalvm.junit.platform.JUnitPlatformFeature..

To Reproduce

• Create a Maven Project with JUnit and the Native Maven Plugin
• Create a test class with an @Test method
• Create another class extending that class and overriding the method
  • Do not mark the overridden method in the subclass with @Test

<dependencies>
	<dependency>
		<groupId>org.junit.jupiter</groupId>
		<artifactId>junit-jupiter-api</artifactId>
		<version>5.10.1</version>
	</dependency>
</dependencies>
<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.11.0</version>
			<configuration>
				<release>17</release>
			</configuration>
		</plugin>
		<plugin>
			<groupId>org.graalvm.buildtools</groupId>
			<artifactId>native-maven-plugin</artifactId>
			<version>0.9.28</version>
			<extensions>true</extensions>
			<executions>
				<execution>
					<id>build-native</id>
					<goals>
						<goal>compile-no-fork</goal>
					</goals>
					<phase>package</phase>
				</execution>
				<execution>
					<id>test-native</id>
					<goals>
						<goal>test</goal>
					</goals>
					<phase>test</phase>
				</execution>
			</executions>
		</plugin>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>3.2.2</version>
		</plugin>
	</plugins>
</build>

mvn test

import org.junit.jupiter.api.Test;

class SuperTest {
	@Test
	public void test() throws Exception {

	}
}

class SubTest extends SuperTest {
	@Override
	public void test() throws Exception {
	}
}

See this minimal reproducer (CI log here)

Expected behavior

Native compilation of tests succeed and the tests run successfully.

Logs

[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.github.danthe1st.junitfeaturefailure.SubTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 s -- in io.github.danthe1st.junitfeaturefailure.SubTest
[INFO] Running io.github.danthe1st.junitfeaturefailure.SuperTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in io.github.danthe1st.junitfeaturefailure.SuperTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- native-maven-plugin:0.9.28:test (test-native) @ junit-feature-failure ---
[INFO] ====================
[INFO] Initializing project: junit-feature-failure
[INFO] ====================
[INFO] Found GraalVM installation from GRAALVM_HOME variable.
[INFO] Executing: /opt/hostedtoolcache/graalvm-community-jdk-21.0.1_linux-x64_bin/21.0.1/x64/graalvm-community-openjdk-21.0.1+12.1/bin/native-image -cp /home/runner/work/junit-feature-failure/junit-feature-failure/target/test-classes:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.10.1/junit-jupiter-api-5.10.1.jar:/home/runner/.m2/repository/org/opentest4j/opentest4j/1.3.0/opentest4j-1.3.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-commons/1.10.1/junit-platform-commons-1.10.1.jar:/home/runner/.m2/repository/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar:/home/runner/.m2/repository/org/graalvm/buildtools/native-maven-plugin/0.9.28/native-maven-plugin-0.9.28.jar:/home/runner/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.28/junit-platform-native-0.9.28.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-console/1.10.0/junit-platform-console-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-reporting/1.10.0/junit-platform-reporting-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-launcher/1.10.0/junit-platform-launcher-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-engine/1.10.0/junit-platform-engine-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-commons/1.10.0/junit-platform-commons-1.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter/5.10.0/junit-jupiter-5.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.10.0/junit-jupiter-api-5.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.10.0/junit-jupiter-params-5.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.10.0/junit-jupiter-engine-5.10.0.jar:/home/runner/.m2/repository/org/graalvm/buildtools/utils/0.9.28/utils-0.9.28.jar:/home/runner/.m2/repository/org/graalvm/buildtools/graalvm-reachability-metadata/0.9.28/graalvm-reachability-metadata-0.9.28.jar:/home/runner/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.28/junit-platform-native-0.9.28.jar --no-fallback -o /home/runner/work/junit-feature-failure/junit-feature-failure/target/native-tests -Djunit.platform.listeners.uid.tracking.output.dir=/home/runner/work/junit-feature-failure/junit-feature-failure/target/test-ids --features=org.graalvm.junit.platform.JUnitPlatformFeature org.graalvm.junit.platform.NativeImageJUnitLauncher
========================================================================================================================
GraalVM Native Image: Generating 'native-tests' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                    (3.9s @ 0.07GB)
 Java version: 21.0.1+12, vendor version: GraalVM CE 21.0.1+12.1
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (linux, x86_64, 11.4.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 2 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - org.graalvm.junit.platform.JUnitPlatformFeature
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 10.80GB of memory (69.2% of 15.61GB system memory, determined at start)
 - 4 thread(s) (100.0% of 4 available processor(s), determined at start)
[junit-platform-native] Running in 'test listener' mode using files matching pattern [junit-platform-unique-ids*] found in folder [/home/runner/work/junit-feature-failure/junit-feature-failure/target/test-ids] and its subfolders.

Error: Feature defined by org.graalvm.junit.platform.JUnitPlatformFeature unexpectedly failed with a(n) org.junit.platform.commons.JUnitException. Please report this problem to the authors of org.graalvm.junit.platform.JUnitPlatformFeature.
------------------------------------------------------------------------------------------------------------------------
                        0.3s (5.6% of total time) in 12 GCs | Peak RSS: 0.44GB | CPU load: 2.65
========================================================================================================================
Finished generating 'native-tests' in 4.1s.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13.508 s
[INFO] Finished at: 2023-11-25T22:19:26Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.graalvm.buildtools:native-maven-plugin:0.9.28:test (test-native) on project junit-feature-failure: Execution of /opt/hostedtoolcache/graalvm-community-jdk-21.0.1_linux-x64_bin/21.0.1/x64/graalvm-community-openjdk-21.0.1+12.1/bin/native-image -cp /home/runner/work/junit-feature-failure/junit-feature-failure/target/test-classes:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.10.1/junit-jupiter-api-5.10.1.jar:/home/runner/.m2/repository/org/opentest4j/opentest4j/1.3.0/opentest4j-1.3.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-commons/1.10.1/junit-platform-commons-1.10.1.jar:/home/runner/.m2/repository/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar:/home/runner/.m2/repository/org/graalvm/buildtools/native-maven-plugin/0.9.28/native-maven-plugin-0.9.28.jar:/home/runner/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.28/junit-platform-native-0.9.28.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-console/1.10.0/junit-platform-console-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-reporting/1.10.0/junit-platform-reporting-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-launcher/1.10.0/junit-platform-launcher-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-engine/1.10.0/junit-platform-engine-1.10.0.jar:/home/runner/.m2/repository/org/junit/platform/junit-platform-commons/1.10.0/junit-platform-commons-1.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter/5.10.0/junit-jupiter-5.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.10.0/junit-jupiter-api-5.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.10.0/junit-jupiter-params-5.10.0.jar:/home/runner/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.10.0/junit-jupiter-engine-5.10.0.jar:/home/runner/.m2/repository/org/graalvm/buildtools/utils/0.9.28/utils-0.9.28.jar:/home/runner/.m2/repository/org/graalvm/buildtools/graalvm-reachability-metadata/0.9.28/graalvm-reachability-metadata-0.9.28.jar:/home/runner/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.28/junit-platform-native-0.9.28.jar --no-fallback -o /home/runner/work/junit-feature-failure/junit-feature-failure/target/native-tests -Djunit.platform.listeners.uid.tracking.output.dir=/home/runner/work/junit-feature-failure/junit-feature-failure/target/test-ids --features=org.graalvm.junit.platform.JUnitPlatformFeature org.graalvm.junit.platform.NativeImageJUnitLauncher returned non-zero result -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error: Process completed with exit code 1.

Full logs here

System Info (please complete the following information):

• OS: Linux
• GraalVM Version: graalvm-community-jdk-21.0.1_linux-x64_bin
• Java Version 21
• Plugin version native-maven-plugin:0.9.28

Additional context

Full reproducer here

[danthe1st]

I am reporting this here since JUnitPlatformFeature seems to be part of this repository.

[danthe1st]

@dnestoro Is there a chance at getting this triaged?

[melix]

Possibly fixed by #565 although I have not checked.

[danthe1st]

Possibly fixed by #565 although I have not checked.

It seems to still be the case in 0.10.0. I have updated the reproducer. nvm that PR isn't merged yet

[danthe1st]

Now that 0.10.1 has been released with #545, I can still reproduce this (GH Actions)
@melix

[danthe1st]

Still happens with 0.10.2.

[stevenschlansker]

We're seeing this exception on 0.10.3 as well.

[danthe1st]

@stevenschlansker As a workaround, you can mark the overridden method with @Test.

[stevenschlansker]

@danthe1st , thanks for the suggestion but ... which test case? The error message does not print out which test case causes the failure, and we have thousands of tests :( I suppose I can narrow the search to a test class that extends another, but still. If this issue really won't be fixed I will eventually invest the time to look at doing so... but given the number of users reporting this, I will hope for a fix in the next release for now :)

[danthe1st]

@stevenschlansker
Yeah I had the same issue. Essentially look at overridden test methods that aren't annotated with @Test. You can also write some scripts for automating that.