Support parallelization in junit-vintage-engine

[YongGoose]

Overview

Resolves #2229

This PR implements parallel execution support for the JUnit Vintage engine, addressing issue #2229.

• Implemented concurrent execution of test descriptors when parallel execution is enabled.
• Updated the JUnit Vintage engine to read parallel configuration parameters. (planned)

Open Question

I haven’t yet worked on the part that handles reading configuration parameters, such as whether parallel execution should be enabled or the number of threads to be used.
Is there a specific configuration format or approach you would recommend for this?

---

I hereby agree to the terms of the JUnit Contributor License Agreement.

---

Definition of Done

• There are no TODOs left in the code
• Method preconditions are checked and documented in the method's Javadoc
• Coding conventions (e.g. for logging) have been followed
• Change is covered by automated tests including corner cases, errors, and exception handling
• Public API has Javadoc and @API annotations
• Change is documented in the User Guide and Release Notes

[marcphilipp]

Thanks for submitting the draft! I left a bunch of comments. 🙂

[YongGoose]

@marcphilipp

If the parallelization work is completed, I would appreciate it if you could take a look at the Open Question in the PR description 😀

[marcphilipp]

@YongGoose The last two weeks were very busy. I should be able to get to your PRs early next week, though. Cheers!

[marcphilipp]

I haven’t yet worked on the part that handles reading configuration parameters, such as whether parallel execution should be enabled or the number of threads to be used.
Is there a specific configuration format or approach you would recommend for this?

I think it should be similar to the configuration parameters in Jupiter. Thus, I'd suggest we use junit.vintage.execution.parallel.enabled and junit.vintage.execution.parallel.pool-size. WDYT?

[YongGoose]

I think it should be similar to the configuration parameters in Jupiter. Thus, I'd suggest we use junit.vintage.execution.parallel.enabled and junit.vintage.execution.parallel.pool-size. WDYT?

That's a great idea! I made this change in e0312cc

[marcphilipp]

As a next step, could you please add some tests?

[YongGoose]

@marcphilipp
While implementing the test, I came across a question and decided to leave a comment :)

In this test case, the return value of VintageEngineDescriptor.getModifiableChildren is the SuccessfulParallelTestCase class.

• VintageTestEngine

private boolean executeInParallel(VintageEngineDescriptor engineDescriptor,
		EngineExecutionListener engineExecutionListener, ExecutionRequest request) {
	ExecutorService executorService = Executors.newFixedThreadPool(getThreadPoolSize(request));
	RunnerExecutor runnerExecutor = new RunnerExecutor(engineExecutionListener);

	List<CompletableFuture<Void>> futures = new ArrayList<>();
	for (Iterator<TestDescriptor> iterator = engineDescriptor.getModifiableChildren().iterator(); iterator.hasNext();) {
		TestDescriptor descriptor = iterator.next();
		CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
			runnerExecutor.execute((RunnerTestDescriptor) descriptor);
		}, executorService);

		futures.add(future);
		iterator.remove();
	}

• TestCode

@Test
void successfulParallelTest(TestReporter reporter) {
	var events = executeInParallelSuccessfully(3, SuccessfulParallelTestCase.class).list();

	var startedTimestamps = getTimestampsFor(events, event(test(), started()));
	var finishedTimestamps = getTimestampsFor(events, event(test(), finishedSuccessfully()));

	reporter.publishEntry("startedTimestamps", startedTimestamps.toString());
	reporter.publishEntry("finishedTimestamps", finishedTimestamps.toString());

	assertThat(startedTimestamps).hasSize(3);
	assertThat(finishedTimestamps).hasSize(3);
}

• SuccessfulParallelTestCase

@RunWith(Enclosed.class)
public class JUnit4ParallelTestCase {

	public static class SuccessfulParallelTestCase {
		static AtomicInteger sharedResource;
		static CountDownLatch countDownLatch;

		@BeforeClass
		public static void initialize() {
			sharedResource = new AtomicInteger();
			countDownLatch = new CountDownLatch(3);
		}

		@Test
		public void firstTest() throws Exception {
			incrementAndBlock(sharedResource, countDownLatch);
		}

		@Test
		public void secondTest() throws Exception {
			incrementAndBlock(sharedResource, countDownLatch);
		}

		@Test
		public void thirdTest() throws Exception {
			incrementAndBlock(sharedResource, countDownLatch);
		}
	}
    ...

Question

• If the TestDescriptor has child TestDescriptors as in the case above, could it be improved more efficiently through recursion?

If a test class contains a large number of test methods, I expect that recursion could make the process more efficient.

I believe the commit will make it easier to understand. 1484347

---

While implementing recursion, I noticed something unusual.

When fetching child TestDescriptor objects using getChildren from a TestDescriptor, it returns instances of VintageTestDescriptor.
As a result, I found that casting them to RunnerTestDescriptor is not possible.

If that’s the case, it seems unavoidable to modify the internal code to implement parallel execution of child TestDescriptor objects through recursion.

Alternatively, we could proceed with the current approach instead of recursion, without modifying the internal code. WDYT?

[marcphilipp]

Question

• If the TestDescriptor has child TestDescriptors as in the case above, could it be improved more efficiently through recursion?

If a test class contains a large number of test methods, I expect that recursion could make the process more efficient.

Do you mean that running the children of a RunnerTestDescriptor concurrently instead of just the top-level RunnerTestDescriptors could speed up the overall execution? If so, I agree with that. However, that's not generally possible as the Runner orchestrates the execution of its children, not the VintageTestEngine.

JUnit 4 actually has a feature that allows running methods in parallel. It works for all subclasses of ParentRunner by setting a RunnerScheduler. We could use that to inject our own scheduler that uses the same ExecutorService. However, I think we should do that in a separate PR and focus on running test classes in parallel in this PR. Thus, you'll have to adjust the tests you've added to not check that methods can be run in parallel but classes.

[YongGoose]

Do you mean that running the children of a RunnerTestDescriptor concurrently instead of just the top-level RunnerTestDescriptors could speed up the overall execution?

Exactly!

JUnit 4 actually has a feature that allows running methods in parallel. It works for all subclasses of ParentRunner by setting a RunnerScheduler. We could use that to inject our own scheduler that uses the same ExecutorService. However, I think we should do that in a separate PR and focus on running test classes in parallel in this PR.

I'll work on it in the next PR along with the issue once this PR is complete!

Thus, you'll have to adjust the tests you've added to not check that methods can be run in parallel but classes.

I was actually considering this issue, so thank you for the detailed suggestion!👍🏻
I'll proceed with the approach you recommended.

[YongGoose]

Thus, you'll have to adjust the tests you've added to not check that methods can be run in parallel but classes.

I implemented it by extracting and comparing thread names when executing test classes. b196388

I understand that in JUnit-Jupiter, threads can be recorded and verified using ThreadReporter.
Therefore, I would like to modularize the code for recording threads.

• If you have any suggestions on how to achieve this, please let me know :)

[marcphilipp]

Please also add an entry to the release notes and a section to the User Guide (as a new section in the "Migrating from JUnit 4" chapter)

[YongGoose]

I pushed a commit that does the following:

• Add an assertion that all start timestamps are <= all finish timestamps to verify test classes are executed in parallel
• Remove the test for failures because I didn't think it provided value

WDYT?

Sounds like a great idea.
The code looks much cleaner now!!

Thank you🙂

[YongGoose]

Please also add an entry to the release notes and a section to the User Guide (as a new section in the "Migrating from JUnit 4" chapter)

I hope I did it well this time...

---

I have a question while writing the release notes.

Shouldn't the details of this PR also be included in the release notes?
Looking at the issue, it seems like it will be released in 5.12 M1, but I noticed it's not mentioned in the release notes, so I wanted to bring it up!

• Implement include and exclude filtering when calling registerAutoDetectedExtensions in MutableExtensionRegistry #4120

If it needs to be added, I will either create a new PR or include it in this one!

[YongGoose]

@marcphilipp

Additionally, once this PR is merged, I will create an issue regarding that comment and start working on it!

• Support parallelization in junit-vintage-engine #4135 (comment)

[marcphilipp]

Shouldn't the details of this PR also be included in the release notes? Looking at the issue, it seems like it will be released in 5.12 M1, but I noticed it's not mentioned in the release notes, so I wanted to bring it up!

• Implement include and exclude filtering when calling registerAutoDetectedExtensions in MutableExtensionRegistry #4120

If it needs to be added, I will either create a new PR or include it in this one!

Good catch! Indeed, that was an oversight on my part which I've now rectified in 4f926ed.

[marcphilipp]

Please also add an entry to the release notes and a section to the User Guide (as a new section in the "Migrating from JUnit 4" chapter)

I hope I did it well this time...

Thanks, you did! I polished your work a bit anyway because we're quite particular. Don't worry about it! 🙂

[marcphilipp]

@YongGoose Once again, thank you for your contribution! 🙇