MeterBinder that uses Health infrastructure causes Rabbit metrics to be bound before meter registry customization has completed

[tde-desc]

Spring Boot version: 2.1.1
Micrometer version: 1.1.1

While defining a custom micrometer MeterRegistryCustomizer I had to face the following issue: The dependency chain of Spring Boot-HealthIndicator seems to interrupt the MeterRegistry bean creation at runtime. I will try to explain with the following setup. Given:

@Bean
public MicrometerConfiguration micrometerConfiguration() {
    return new MicrometerConfiguration();
}

@Bean
public HealthEndpointMeterBinder healthEndpointMetricRegistry(HealthEndpoint healthEndpoint) {
    return new HealthEndpointMeterBinder(healthEndpoint);
}

Are being defined as follows:

@RequiredArgsConstructor
public class MicrometerConfiguration implements MeterRegistryCustomizer {

    @Override
    public void customize(MeterRegistry registry) {
        // do something ...
    }
}

@RequiredArgsConstructor
public class HealthEndpointMeterBinder implements MeterBinder {
    private final HealthEndpoint healthEndpoint;

    @Override
    public void bindTo(MeterRegistry meterRegistry) {
        Gauge.builder("unhealthy", healthEndpoint,
                e -> e.health().getStatus() == Status.UP ? 0.0 : 1.0
        ).register(meterRegistry);
    }
}

In org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryPostProcessor#getConfigurer you can observe the process being interrupted which for example leads to org.springframework.boot.actuate.metrics.amqp.RabbitMetrics#bindTo being called before MicrometerConfiguration#customize. This results to not being able to register custom common tags for example.

I could solve this issue by defining the HealthIndicator dependency as ObjectProvider:

@RequiredArgsConstructor
public class HealthEndpointMeterBinder implements MeterBinder {
    private final ObjectProvider<HealthEndpoint> healthEndpoint;

    @Override
    public void bindTo(MeterRegistry meterRegistry) {
        Gauge.builder("unhealthy", healthEndpoint,
                e -> e.getIfAvailable().health().getStatus() == Status.UP ? 0.0 : 1.0
        ).register(meterRegistry);
    }
}

In my mind, injecting the HealthIndicator bean should not lead to the interruption of the MeterRegistry bean creation. I hope i could make myself clear ...

Regards
Dennis

[wilkinsona]

Interesting problem. It sounds like something is triggering premature initialisation but I cannot tell what it is from the description you've provided. Can you please provide a complete and minimal sample (something that we can unzip or git clone and run) that reproduces the behaviour you've described?

[tde-desc]

Sure:
springhealth.zip
springhealth_2.zip

Just start the application, call http://localhost:8080/actuator/metrics/rabbitmq.consumed and u will see the following:

{

    "name": "rabbitmq.consumed",
    "description": null,
    "baseUnit": null,
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 0.0
        }
    ],
    "availableTags": [
        {
            "tag": "name",
            "values": [
                "rabbit"
            ]
        }
    ]

}

If u swap the commented lines of code with the inlined code, u will see the following output:

{

    "name": "rabbitmq.consumed",
    "description": null,
    "baseUnit": null,
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 0.0
        }
    ],
    "availableTags": [
        {
            "tag": "serverrole",
            "values": [
                "someRole"
            ]
        },
        {
            "tag": "stage",
            "values": [
                "production"
            ]
        },
        {
            "tag": "name",
            "values": [
                "rabbit"
            ]
        }
    ]

}

The second output is the one I would've expected :-)

[wilkinsona]

Thanks for the sample, but it doesn't reproduce the problem. It also doesn't seem to include any of the code that you've mentioned above. What have I missed?

[tde-desc]

I'm sorry, I obviously uploaded the wrong archive. Here's the correct one:
springhealth_2.zip

I also updated my first feedback. Sorry for any inconvenience.

[wilkinsona]

Thanks for the updated sample. I've reproduced the problem.

Here's what's happening:

1. SimpleMeterRegistry is being created
2. MeterRegistryPostProcessor starts post-processing the registry
3. It asks for all the MeterBinder beans as an ordered list
4. Creation of your HealthEndpointMeterBinder bean begins
5. Creation of HealthEndpoint begins which triggers creation of all HealthIndicator beans
6. Creation of RabbitHealthIndicatorAutoConfiguration begins which triggers creation of all RabbitTemplate beans
7. Creation of the RabbitTemplate bean triggers creation of a RabbitConnectionFactory bean
8. RabbitConnectionFactoryMetricsPostProcessor post-processes the RabbitConnectionFactory and binds its metrics
9. MeterRegistryPostProcessor continues post-processing the registry and asks for all the MeterRegistryCustomizer beans.
10. Your MicrometerConfiguration configures the registry's common tags

The problem is the ordering of steps 8 and 10. We may be able to improve the situation by reworking the meter registry post-processing so that MeterRegistryCustomizer beans are applied before MeterBinder beans are retrieved from the bean factory.