[SUREFIRE-2252] - Fix JUnit5 reporting when tests are executed in parallel

[ArvindJoshi-okta]

Problem

• When parallel tests are executed in JUnit5, the reporting (console and xml) have some issues
• The tests are executed but the reporting combines tests into other classes showing incorrect results (as seen below)
• Sample code is attached to the JIRA https://issues.apache.org/jira/browse/SUREFIRE-2252

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.ClassBIT
[INFO] Running org.example.ClassAIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 s – in org.example.ClassAIT
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 s – in org.example.ClassBIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Fix

• Issue was in JUnitPlatformProvider where the execute method was sending in all the classes at the same time in class selector list, hence combining the results
• Fix was to move the list creation inside the iteration loop
• Added relevant unit and integration tests

Checklist

Following this checklist to help us incorporate your
contribution quickly and easily:

• Make sure there is a JIRA issue filed
  for the change (usually before you start working on it). Trivial changes like typos do not
  require a JIRA issue. Your pull request should address just this issue, without
  pulling in other changes.
• Each commit in the pull request should have a meaningful subject line and body.
• Format the pull request title like [SUREFIRE-XXX] - Fixes bug in ApproximateQuantiles,
  where you replace SUREFIRE-XXX with the appropriate JIRA issue. Best practice
  is to use the JIRA issue title in the pull request title and in the first line of the
  commit message.
• Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• Run mvn clean install to make sure basic checks pass. A more thorough check will
  be performed on your pull request automatically.
• You have run the integration tests successfully (mvn -Prun-its clean install).

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

• I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004

• In any other case, please file an Apache Individual Contributor License Agreement.

[michael-o]

There are test failures, please take a look.

[ArvindJoshi-okta]

There are test failures, please take a look.

Looks like the POM issue I was referring to in the JIRA, on what value should I add to test my current logic/version.

org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.maven.plugins:maven-surefire-plugin:3.3.2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.maven.plugins:maven-surefire-plugin:jar:3.3.2-SNAPSHOT

[ArvindJoshi-okta]

There are test failures, please take a look.

Looks like the POM issue I was referring to in the JIRA, on what value should I add to test my current logic/version.

org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.maven.plugins:maven-surefire-plugin:3.3.2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.maven.plugins:maven-surefire-plugin:jar:3.3.2-SNAPSHOT

Will try to use <version>${surefire.version}</version> in my test pom.xml and see if that gets picked up automatically.

[ArvindJoshi-okta]

There are test failures, please take a look.

@michael-o Could you pls review now? All tests are passing.

[michael-o]

How this has been collapsed and both cases are identical. Can you explain why this new code is correct for both? I honest cannot assess whether this is wrong or correct. What should happen in your opinion in the eager case?

[ArvindJoshi-okta]

Thinking about this a bit, I think I am misunderstanding what the eager reading actually means. Could you please clarify its intention? How does an end user set it and where?

In my case here, what is happening is, in a parallel run, the test runs are combined due to eager reading, the classes get executed together and hence tests get reported as a bundle (I'll look into the reporting aspect, maybe that's the issue and not the execution). But in general, I felt there is no need for eager reading for a parallel run. Maybe we need to separate the 2 conditions?

[michael-o]

Thinking about this a bit, I think I am misunderstanding what the eager reading actually means. Could you please clarify its intention? How does an end user set it and where?

Unfortunately, I cannot answer these question because I have no idea.

In my case here, what is happening is, in a parallel run, the test runs are combined due to eager reading, the classes get executed together and hence tests get reported as a bundle (I'll look into the reporting aspect, maybe that's the issue and not the execution). But in general, I felt there is no need for eager reading for a parallel run. Maybe we need to separate the 2 conditions?

Is this similar: https://issues.apache.org/jira/browse/SUREFIRE-2195?

[ArvindJoshi-okta]

Yes, 2195 seems like the same issue

[michael-o]

I honestly don't know how to proceed because I do not understand the implications :-(

[slawekjaranowski]

With this change I see in logs:

[INFO] Running pkg.domain.BxTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s -- in pkg.domain.BxTest
[INFO] pkg.domain.BxTest.test -- Time elapsed: 0.007 s
[INFO] Running pkg.domain.AxTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s -- in pkg.domain.AxTest
[INFO] pkg.domain.AxTest.test -- Time elapsed: 0 s

but without:

[INFO] Running pkg.domain.AxTest
[INFO] Running pkg.domain.BxTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s -- in pkg.domain.AxTest
[INFO] pkg.domain.BxTest.test -- Time elapsed: 0.007 s
[INFO] pkg.domain.AxTest.test -- Time elapsed: 0.007 s
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s -- in pkg.domain.BxTest

so look like tests are executed sequentially not in parallel

So for me problem is something else ...