Using test fragments for unit tests

[HeikoKlare]

Test fragments are great for defining unit tests for internals of a plug-in that are not intended to be accessible via the plug-in's public API (or public at all). Currently, I only find few test fragments in platform code:

• org.eclipse.e4.ui.workbench.addons.swt.test
• org.eclipse.test.performance.win32
• org.eclipse.tests.urischeme
• org.eclipse.ui.ide.application.tests
• org.eclipse.ui.monitoring.tests

I would like to ask if there are specific reasons for not having further test fragments. If I remember correct I once read a discussion about that topic, but unfortunately I can neither recap the contents nor find that discussion anymore.

A concrete reason for the question are our current efforts (in particular those of @akoch-yatta and colleagues at Yatta) to provide multi-monitor HiDPI support for SWT on Windows. Since there is no test fragment for SWT yet, only public classes and methods can be tested properly. Most of the added functionality will, however, not be public (and in particular not be part of the public API, as it is specific to Windows). We would like to provide the new functionality with proper architecture and encapsulation, but we are limited in terms of testability when not using test fragments. Either functionality cannot be tested in an isolated way as it is hidden, or it needs to have higher visibility than actually necessary just to make it addressable via tests.

If there are any reasons for not using test fragments or any other arguments on that topic, we would be interested in them. Otherwise we would like to provide a test fragment for SWT to enhance tests at the unit test level.

[merks]

In my opinion, you should not feel limited and should proceed following the approach you believe to be the best.

[laeubi]

Disclaimer

First of all the following is in no way normative for platform development and you should most probably follow the advice of @merks to use what works best for you but I still want to mention some things regarding "test fragments"

Testfragments

Actually test-fragments are a workaround for tooling limitations (in the past) and I would not recommend to use them for the following reasons:

1. The OSGi specification don't have such thing as a "test fragments" they are technically only regular fragments and have some implications, especially they break encapsulation, that means everything from your fragment (requirements, resources, package exports, ...) will become part of your bundle but only during the test! So maybe your test works but at runtime it fails because something was source accidentally only through your fragment.
2. There is no such thing as "internal API", what often is refereed to as this is actually an implementation detail and basing your test on implementation details will on the long run require you to rewrite your test every-time you change internal implementation with the risk that you cant use your test as a way to prevent regressions. That's also the reason why Tycho uses so many integration test in the form of maven projects because that's what is the user visible behavior and we almost never adjust them (except we change this user-visible-behavior) and if these test fail we know we have break something that must be fixed because it will affect users and not change the test because we refactored internal things!
3. If you really like to test internal algorithms (in contrast to behavior), you should neither use a fragment nor a "test bundle" but use a test source folder inside the same project, one example in platform can be found here and further information regarding Tycho here.

[HeikoKlare]

Thank you for that detailed feedback, @laeubi! That is very helpful. In particular, I agree that using test source folders inside a plug-in are the best-suited approach here and should be preferred over test fragments. I have tried that some years ago, but I had quite some pain of which I do not remember the reasons anymore (maybe it was something about the configuration of maven-surefire). That's why I am still used to test fragments. Since you have referred to some example and configuration information and since it is the approach I would prefer from a conceptual point of view, that resolves my question for now and I propose that we follow the approach of having a test source folder in the plug-in to be tested.

I would still like to add some comments/additions on your points for clarification.

Testing only at a single level of encapsulation

First, I want to add to my initial question that I do not ask for using test fragments as a replacement for ordinary test plug-ins. Ordinary plug-ins tests are of course the correct approach to test at the plug-in's level of encapsulation (which is one of the most important levels of abstraction in a plug-in/component-based software architecture). It is only about testing at this level of encapsulation being insufficient, as plug-ins usually have (or at least should have) their own internal microarchitecture requiring tests at appropriate levels of abstraction as well.

Testing a plug-in's microarchitecture

Yes, test fragments break the plug-in's encapsulation. However, as mentioned above, that is by intent, as a plug-in has it's own internal microarchitecture (with it's own modules, components or levels of abstraction) that requires appropriate tests depending on the internal structure. That may be the only point where we slightly disagree: not everything that is not part of a plug-in's required and provided interface is an implementation detail that is not required to be tested individually. You should test the interfaces at each level of abstraction that your system has (no matter if that is the plug-in level of your macroarchitecture or some further abstraction level within the microarchitecture of a plug-in). Only then you can change implementations and get test feedback at a proper interface level rather than at some universal plug-in encapsulation level.

As an example, you may have some complex algorithm whose plug-in's provided interface only has a single facade class with a method to execute that algorithm. But the algorithm itself is highly complex and modularized internally (each module having its own well-defined interface), so you want to test the internal modules (at the level of their own encapsulation) individually, so that you are able to alter the internal modules easily. Of course, the lower the abstraction level that you test, the more likely you will need to throw away or adapt tests. But the need to adjust tests is nothing bad, it only means that you changed some specification. And just because a specification may change, it should not be the reason for not testing an implementation against that specification.

[laeubi]

I agree that using test source folders inside a plug-in are the best-suited approach here and should be preferred over test fragments. I have tried that some years ago, but I had quite some pain of which I do not remember the reasons anymore (maybe it was something about the configuration of maven-surefire).

That's why I said its mostly a tooling issue and I tried in the past years to improve this (particular at Tycho) so if any issues arise please fell free to report that back.

One thing that is not completely solved from my point of view (even though it should not be an issue for the SWT case) is if your test requires additional libraries there are some workarounds but nothing really satisfying. I'm currently working on a solution, but also here feedback is welcome.

It is only about testing at this level of encapsulation being insufficient, as plug-ins usually have (or at least should have) their own internal microarchitecture requiring tests at appropriate levels of abstraction as well.

OSGi was made in mind with clear ways to "hide" as much as possible, so from OSGi point of view, the internal structure does not really matter it only matters what is exported (the contract) so if you test the contract and that is fulfilled (in OSGi there are so called Technical Compatibility Testkits) your bundle is fully tested by all consumer visible and observable aspects.

So just assume there is something not testable though public contracts, that usually means that either you bundle is doing stuff that is not needed or your contract / API is flawed or that it is simply not suitable for a "unit test" but has to be checked in a (differently contracted) context like an integration test.

But the need to adjust tests is nothing bad, it only means that you changed some specification. And just because a specification may change, it should not be the reason for not testing an implementation against that specification.

That's sadly something that's a bit flawed at platform as in most cases it mixes api/contract with concrete implementations. The idea of OSGi is that API/contract and implementation are two "things" so you should usually have (at laest) three artifacts for a specification (and that's how the OSGi specification itself is build):

1. An API contract (that's is a bundle defining the API interfaces and a specification further describing the behavior in more details) for example for the PDE declarative services annotations this is [org.osgi.service.component.annotations](https://mvnrepository.com/artifact/org.osgi/org.osgi.service.component.annotations/1.5.1) and Declarative Services Specification
2. A TCK that probes an implementation of this API contract according to the specification, this is for example org.osgi.test.cases.component.annotations
3. An implementation (in this case PDE) consuming the API and is tested against the TCK

This perfectly show that is would be dramatically flawed to require a "test fragment" that is specific to an implementation as then you would need a TCK for every other implementation and can't really compare them.

So in case the specification changes one would change (1 + 2) as for example here and then adjust (3) that it again complies to the TCK, but one won't ever change (1+2) because (3) changes!

Still as demonstrated it might be useful to test some aspects in isolation in form of small test cases that check a given aspect, but "test" fragments/bundles are from my point of view not really part of that. And whenever you feel the need to have an own level ob abstraction inside a bundle its maybe a good sign that your bundle is not in a good shape and should probably be the consumer and the abstraction being a separate API (with one or more implementations).

[HeikoKlare]

We are completely on the same line with respect to how to test a plug-in's behavior. Thank you also for detailing in which (separate) artifacts a specification conceptually has to split up. That's also exactly how we build our RCP-based product (and I agree that this separation is not present at many places in the Eclipse platform).

The only thing I do not really agree with is the assumption that an OSGi-based application has one single overall architecture, in which every relevent level of abstraction (that is tested on it's own) is defined by a single bundle:

And whenever you feel the need to have an own level ob abstraction inside a bundle its maybe a good sign that your bundle is not in a good shape and should probably be the consumer and the abstraction being a separate API (with one or more implementations).

So just assume there is something not testable though public contracts, that usually means that either you bundle is doing stuff that is not needed or your contract / API is flawed or that it is simply not suitable for a "unit test" but has to be checked in a (differently contracted) context like an integration test.

That does not reflect the necessity of complex products, in which for different "parts" (module/subsystems/...) of the system different architectures may be adequate, so that it may consist of a multitude of micro- and macroarchitecture, in which both an OSGi-based architecture may occur but there may also be other types of architecture with their own levels of abstraction in the overall system architecture. For example, if we reduced our product to only a plug-in-based architecture on OSGi without any further microarchitectures, so that every specification has to be done via an OSGi bundle, an implementation and according tests, we would end up in a convoluted set of 10.000s of plug-ins (other than the 2.000 plug-ins that we have right now). This would be technically infeasable and would also be inappropriate from an architectural point-of-view, as by flattening everything into a single layer of abstraction (given by OSGI bundles), you loose the structure that you gain by hierarchically combining modules/subsystems with their own architectures.

[carstenartur]

For me the quirks I need to get rid of warnings and issues when working on a eclipse version derived from jdt are huge.
Just look at the buddy policy.

https://github.com/carstenartur/eclipse.jdt.ui/blob/explicitencoding/org.eclipse.jdt.ui.tests.junit5/META-INF/MANIFEST.MF

And there are many more.

In my opinion it looks like jdt is not really taking care of osgi. Just to the point that jdt itself is somehow working. Who tries to reuse it stands in the rain.
You can argue that the positive thing is that of course in spite of the need of workarounds in fact different approaches work. So it is up to your point of view.

[Bananeweizen]

Historically speaking, JDT didn't differentiate main source from test source, as Maven or Gradle or... do. Therefore in the past having main source and test source in one plugin would easily lead to test only dependencies leaking into main code. However, that's a thing of the past and meanwhile you can (and should!) mark all the test source paths (including other classpath entries) as such. Therefore creating a bundle with main and test code following the Maven source folder conventions or similar is surprisingly easy nowadays. So that former benefit of fragments is gone.
Additionally, fragments have the problem that you cannot depend from fragment A to fragment B (but rather get a dependency to the host of B instead).

IMO these 2 points make fragments less attractive for testing nowadays. That said, at least my company bundle testing code is a wild mix of fragments, bundles, and source folders following or not following maven conventions. So it all works, and you can choose what suits your needs. :)

[HeikoKlare]

I just found that several tests that we need to provide will test code within a fragment (the SWT Win32 fragment). This makes test fragments unapplicable, as you have already pointed out that a fragment cannot depend on another fragment.

On the other hand, if you have multiple fragments that implement the same class and you write a test class that uses this class, you may end up with a test class that is compiled against the class (with the same FQN) from other fragments. At least this is what happens, e.g., regarding SWT's FontData, which is used from the GTK fragment, even though the test written against it is placed int the Win32 fragment.
I guess this is quite specific to SWT as the host has kind of dependencies to its fragments via its p2.inf files, but in my case it ends up having tests in one fragment that are compiled against classes from another fragment. I think I will have to make some adaptation to the Maven/Tycho dependencies here to avoid such improper dependencies.

[laeubi]

Yes SWT is quite special, but as SWT does not require OSGi to function it should work to put the test inside a src_test folder inside the fragment, then the test will be specific to the given os. As we compile on all platforms all fragments one would need an assumeThat(...) as a first thing to make sure it is only executed when run on the "real" platform.

For everything else that should be tested cross platform, a bundle that contain the tests and is executed on different platforms (like the github runners) should be suitable.

[HeikoKlare]

Yes, that's basically what I did. I have a test source folder inside the win32 fragment. The problem is that already the compilation fails if we access details of an OS-specific implementation, such as FontData.data, which is only present in the win32 implementation but not in the others (it's just an example, so it's not about whether one should access this exact implementation detail in a test or not). The compiler already fails because it tries to compile the test class (in the win32 fragment) against the FontData implementation from some other fragment. I am not really sure yet why that happens, but I will try to figure out.

[ERROR] D:\dev\eclipse\platform-only\git\eclipse.platform.swt\binaries\org.eclipse.swt.win32.win32.x86_64\Tests\org\eclipse\swt\internal\DefaultSWTFontRegistryTests.java:[58]
[ERROR]         int heightFontPrimary = fontPrimary.data.lfHeight;
[ERROR]                                             ^^^^
[ERROR] data cannot be resolved or is not a field

Edit: This does not seem to be a general problem, but only occurs when making a selective build with -pl option. When running a full build, compilation and test execution runs just fine.

[laeubi]

You at least need -am but be aware that gain SWT is special so even that would probably fail.