@MockBean fields are not reset for JUnit 5 @Nested tests

[slu-it]

Hello Spring Boot team!

Today a colleague of mine came across strange behaviour when writing some JUnit 5 tests. It seems that @MochBean annotated fields, or more specifically the actual mocks behind that, are not reset for @Test methods residing in @Nested inner classes.

Example:

@SpringBootTest
@ExtendWith(SpringExtension.class)
class FailingTests {

    @MockBean
    ServiceB serviceB;
    @Autowired
    ServiceA cut;

    @Nested
    class NestedTest {

        @Test
        void mockWasInvokedOnce() {
            cut.foo();
            verify(serviceB, times(1)).bar();
        }

        @Test
        void mockWasInvokedTwice() {
            cut.foo();
            cut.foo();
            verify(serviceB, times(2)).bar();
        }

    }

}

One of these tests will fail because the mock will have been invoked 3 times at that point.

Tested with Spring Boot 2.0.1.BUILD-SNAPSHOT and JUnit 5.1.0 on 2018-03-13 at around 8:00 PM. (also tested with 2.0.0.RELEASE earlier)

I created a simple example application (thx start.spring.io ;) ) to reproduce the issue: https://github.com/slu-it/bug-spring-mockbean-reset

I also tried to find the cause of the issue, but I might have underestimated the complexity of the spring-test and spring-boot-test libraries... As far as I can see, the application context returned from the TestContext instance for tests within the @Nested class does not include references to the ServiceB bean. (From debugging MockitoTestExecutionListener)

When I debug the same location for a test without the @Nested part, the returned context contains a clearly identifiable instance of ServiceB.

I think the correct behaviour would be to have a parent child relationships between application contexts in case of inner classes (like @Nested JUnit 5 test classes).

[slu-it]

As a side note: I think this might also be the reason why Spring REST Docs did not work with @Nested JUnit 5 tests last I checked (somewhere around M5). But I'll have to see, if this is still true.

[wilkinsona]

Thanks for the minimal sample, @slu-it.

The problem is that NestedTest is using a separate application context to the one that's created for FailingTests. This means that the before and after test method callbacks for NestedTest.mockWasInvokedOnce() and NestedTest.mockWasInvokedTwice() are called with the application context created for NestedTest. It doesn't contain any mocks so there's nothing to reset. There's no link that I can see between the two contexts (no parent-child relationship, for example) so there's also no obvious way for us to get from the context for NestedTest to the context for FailingTests.

I'm rather surprised that there's a second application context created for NestedTest. My hunch was that it would share the context that was created for FailingTests. There's no mention of @Nested in the relevant section of Spring Framework's reference documentation so I'm not sure what is the right way to use @Nested with Spring Framework's testing support.

@sbrannen can you help us to get to the bottom of this one please?

[wilkinsona]

A bit of hunting around led me to NestedTestsWithSpringAndJUnitJupiterTestCase.java . It's clear from those tests that the behaviour I described above is intentional and that my hunch was wrong. It's also taught me that in this case NestedTest should be annotated with @SpringBootTest. This gives it a chance to reuse the context that was created and cached for FailingTests. In fact, it would appear that it's only prevented from doing so due to there being no mock for serviceB in NestedTest. If FailingTests and NestedTest have the same mocks, the context is reused and the tests pass.

The change described above makes the test code look like this:

package example;

import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;

import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.test.context.junit.jupiter.SpringExtension;


@SpringBootTest
@ExtendWith(SpringExtension.class)
class FailingTests {

    @Autowired
    ServiceA cut;

    @MockBean
    ServiceB serviceB;

    @Nested
    @SpringBootTest
    class NestedTest {

        @MockBean
        ServiceB serviceB;

        @Test
        void mockWasInvokedOnce() {
            cut.foo();
            verify(serviceB, times(1)).bar();
        }

        @Test
        void mockWasInvokedTwice() {
            cut.foo();
            cut.foo();
            verify(serviceB, times(2)).bar();
        }

    }

}

I wonder if we can do something when calculating the context cache key for NestedTest that would allow us to reuse the context from FailingTests without having to declare the mock for serviceB in both places.

[slu-it]

In the meantime I checked whether or not the Spring REST Docs issue still exists. Tested it with Spring Boot 2.0.1-BUILD-SNAPSHOT in this repository: https://github.com/slu-it/bug-spring-restdocs-nested-tests
Failing with @Nested tests as well. Might be related, or it might just be coincidence.. I'll create a separate issue for that in the Spring REST Docs repository and link it here.

[wilkinsona]

Here's a possible fix. Can you take a look please, @philwebb?

With the fix in place, these failing tests are now actually working tests:

package example;

import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;

import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.test.context.junit.jupiter.SpringExtension;


@SpringBootTest
@ExtendWith(SpringExtension.class)
class FailingTests {

    @Autowired
    ServiceA cut;

    @MockBean
    ServiceB serviceB;

    @Nested
    @SpringBootTest
    class NestedTest {

        @Test
        void mockWasInvokedOnce() {
            cut.foo();
            verify(serviceB, times(1)).bar();
        }

        @Test
        void mockWasInvokedTwice() {
            cut.foo();
            cut.foo();
            verify(serviceB, times(2)).bar();
        }

    }

}

Adding a unit test for this is a challenge. As soon as I add an optional dependency on the JUnit Jupiter API to spring-boot-test, Eclipse defaults to using the JUnit 5 runner for all the tests in that project and you have to manually tell it to use JUnit 4 instead. I think that's too high a price to pay. We could add some use of @Nested to the JUnit 5 sample instead.

[slu-it]

@wilkinsona is the second @SpringBootTest on the @Nested still necessary?

[wilkinsona]

Yes. Without it, you get a second application context created for NestedTest. That second context is created as its configuration doesn't have anything to do with Spring Boot. Adding @SpringBootTest means that both NestedTest and FailingTests use SpringBootTestContextBootstrapper which finds example.Application as the basis for the test context's configuration in both cases.

[slu-it]

Ok, makes sense.

[sbrannen]

I'm rather surprised that there's a second application context created for NestedTest.

I'm actually very surprised by that. I thought that a @Nested would by default not have an ApplicationContext loaded for it.

In fact, I'm nearly 100% certain that an ApplicationContext will not be loaded for such a @Nested class when using the Spring TestContext Framework (TCF) without Spring Boot Test support.

So, perhaps it's just the magic of Spring Boot Test that is automagically finding the configuration for the test ApplicationContext.

But I'd have to debug that to see what's really going on.

My hunch was that it would share the context that was created for FailingTests.

Many people actually assume that would be the case, since the TCF supports configuration inheritance within a test class hierarchy.

However, Spring does not yet have any support for "inheriting" configuration from an enclosing class.

See SPR-15366 for details.

Unfortunately, resolving SPR-15366 is not so easy (if even feasible) since the entire annotation processing infrastructure within the core Spring Framework (and Spring portfolio projects for that matter) has zero support for searching for annotations on an enclosing class. 😞

There's no mention of @Nested in the relevant section of Spring Framework's reference documentation so I'm not sure what is the right way to use @nested with Spring Framework's testing support.

The "right way" for the time being is simply to redeclare all of the annotations declared on the enclosing class. A custom composed annotation can alleviate the pain here, but in general I admit that it is quite a nuisance.

[sbrannen]

Yes. Without it, you get a second application context created for NestedTest. That second context is created as its configuration doesn't have anything to do with Spring Boot.

Hmmm... what is the second context created from?

For example, what @Configuration class(es) is/are used to create that context?

[sbrannen]

Here's a possible fix. Can you take a look please, @philwebb?

With the fix in place, these failing tests are now actually working tests:

That of course sounds nice for now, but I'm wondering what the implications would be if SPR-15366 is resolved at a later date. Maybe the result of SPR-15366 would be an opt-in feature for turning on "inheritance from enclosing class" in order to remain backwards compatible.

Will have to ponder it... 😉

[sbrannen]

@wilkinsona,

Without it, you get a second application context created for NestedTest.

If you have personally confirmed that behavior with Boot, I would appreciate if you could add a corresponding comment to SPR-15366.

TIA