Introduce support for filtering packages

[sbrannen]

Status Quo

It is currently possible to select a package -- for example, for classpath scanning. However, it is not possible to filter packages.

See use case described in #534 for additional input.

Related Issues

• Introduce DiscoverySelector for classpath resources #301
• Align gradle plugin and runner features with Launcher API #312
• Introduce support for excluding classes via filters #314
• Introduce support for selecting packages, classes, and methods in ConsoleLauncher and Gradle plugin #437
• Classpath scanning eagerly loads classes unnecessarily #474
• DiscoverySelectors#selectJavaPackage not restricted to the provided package name #534

Deliverables

• Introduce include and exclude filter support for packages in a new PackageFilter class (similar to ClassFilter).
• Introduce include and exclude filter support for packages in the Gradle JUnit Platform plugin.
• Introduce include and exclude filter support for packages in the JUnitPlatform runner (e.g., @IncludePackages and @ExcludePackages).

[bechte]

When specifying filters for packages we have different options how to evaluate the filter rules:

1. Use one filter that handles included and excluded packages (filter rules also apply for sub packages)
2. Use one filter that handles included and excluded packages (filter rules only apply for their specified package)
3. Use two different filters, one for included packages and one for excluded packages (filter rules also apply for sub packages)
4. Use two different filters, one for included packages and one for excluded packages (filter rules only apply for their specified package)

The advantage of having one filter over two different filters is, that it could evaluate all filter rules allowing to specify more granulate filter rules. The advantage of having two separate filters is, that the filters itself are simpler and better to maintain. Therefore, we prefer option 4 in favor to option 2 in case of simplicity.

We can discuss the other options using the following example and see have the filter rule are applied and what the outcome is.

Example

Given the following filter rules:

No	Package	Filter Rule
1	org.junit.sample	included
2	org.junit.sample.module1	excluded
3	org.junit.sample.module1.category1	included
4	org.junit.sample.module1.category2	excluded
5	org.junit.sample.module2	excluded
6	org.junit.sample.module3	included

Filtering with Option 1

Class	Matching Rules	Filter result	Included in test run?
org.junit.outside.SampleTests	-	undefined	❓
org.junit.sample.SampleTests	1	included	✔️
org.junit.sample.module1.SampleTests	1, 2	excluded	❌
org.junit.sample.module1.category1.SampleTests	1, 2, 3	included	✔️
org.junit.sample.module1.category2.SampleTests	1, 2, 4	excluded	❌
org.junit.sample.module1.category3.SampleTests	1, 2	excluded	❌
org.junit.sample.module2.SampleTests	1, 5	excluded	❌
org.junit.sample.module3.SampleTests	1, 6	included	✔️
org.junit.sample.module4.SampleTests	1	included	✔️

Note:

• If the Filter does not find any matching rule and has at least one rule, if is undefined if the class should be included or excluded!

Filtering with Option 2

Class	Matching Rules	Filter result	Included in test run?
org.junit.outside.SampleTests	-	undefined	❓
org.junit.sample.SampleTests	1	included	✔️
org.junit.sample.module1.SampleTests	2	excluded	❌
org.junit.sample.module1.category1.SampleTests	3	included	✔️
org.junit.sample.module1.category2.SampleTests	4	excluded	❌
org.junit.sample.module1.category3.SampleTests	-	undefined	❓
org.junit.sample.module2.SampleTests	5	excluded	❌
org.junit.sample.module3.SampleTests	6	included	✔️
org.junit.sample.module4.SampleTests	-	undefined	❓

Note:

• If the Filter does not find any matching rule and has at least one rule, if is undefined if the class should be included or excluded!

Filtering with Option 3

Class	Matching Rules	Filter result	Included in test run?
org.junit.outside.SampleTests	-	excluded	❌
org.junit.sample.SampleTests	1	included	✔️
org.junit.sample.module1.SampleTests	1, 2	excluded	❌
org.junit.sample.module1.category1.SampleTests	1, 2, 3	excluded	❌
org.junit.sample.module1.category2.SampleTests	1, 2, 4	excluded	❌
org.junit.sample.module1.category3.SampleTests	1, 2	excluded	❌
org.junit.sample.module2.SampleTests	1, 5	excluded	❌
org.junit.sample.module3.SampleTests	1, 6	included	✔️
org.junit.sample.module4.SampleTests	1	included	✔️

Note:

• If the IncludeFilter does not find any matching rule and has at least one rule, it will decline the class
• If the ExcludeFilter does not find any matching rule and has at least one rule, it will accept the class

Filtering with Option 4

Class	Matching Rules	Filter result	Included in test run?
org.junit.outside.SampleTests	-	excluded	❌
org.junit.sample.SampleTests	1	included	✔️
org.junit.sample.module1.SampleTests	2	excluded	❌
org.junit.sample.module1.category1.SampleTests	3	included	✔️
org.junit.sample.module1.category2.SampleTests	4	excluded	❌
org.junit.sample.module1.category3.SampleTests	-	excluded	❌
org.junit.sample.module2.SampleTests	5	excluded	❌
org.junit.sample.module3.SampleTests	6	included	✔️
org.junit.sample.module4.SampleTests	-	excluded	❌

Note:

• If the IncludeFilter does not find any matching rule and has at least one rule, it will decline the class
• If the ExcludeFilter does not find any matching rule and has at least one rule, it will accept the class

Quick option comparison

Class	Option 1	Option 2	Option 3	Option 4
org.junit.outside.SampleTests	❓	❓	❌	❌
org.junit.sample.SampleTests	✔️	✔️	✔️	✔️
org.junit.sample.module1.SampleTests	❌	❌	❌	❌
org.junit.sample.module1.category1.SampleTests	✔️	✔️	❌	✔️
org.junit.sample.module1.category2.SampleTests	❌	❌	❌	❌
org.junit.sample.module1.category3.SampleTests	❌	❓	❌	❌
org.junit.sample.module2.SampleTests	❌	❌	❌	❌
org.junit.sample.module3.SampleTests	✔️	✔️	✔️	✔️
org.junit.sample.module4.SampleTests	✔️	❓	✔️	❌

Conclusion

The most natural option would be Option 1. Its result seem to be the most obvious solution from a user perspective of the filtering. It allows to specify fine granulate rules and overriding on any level, e.g. org.junit.sample.module1.category1.SampleTests is included again. Still it requires a definition for cases that do not match any rule, e.g. org.junit.outside.SampleTests and it has a higher complexity than options 3 and 4.

Option 3 has a lower complexity and almost the same filter results. There is only one case org.junit.sample.module1.category1.SampleTests that differs. This is because the excluded and included are evaluated separately. Therefore, we cannot override a previously specified exclude rule. This also seems to be okay, if we say the rule is also being applied for sub packages.

Option 2 will lead to unexpected filter results if we define the outcome of the undefined cases to be included. Therefore, if should return exclude in these case, making the outcome of the filter evaluation exact equally with Option 4. We are in favour of option 4 in this case because of its simplicity over option 2.

Recommendation

One should go for Option 1 or Option 3, depending on whether or not one wants to be able to overrule previously defined exclude rules for sub packages. If JUnit should support this, then Option 1 is the best solution. If this is not required, Option 3 is the better solution, as it is much easier to implement, understand, and maintain.

[bechte]

Team decision: Follow the recommendation and start with option 3.

[bechte]

Merged into master with #570

[sbrannen]

Thanks, @bechte! 👍