feat: now possible to only output non-resource related metrics

[metacosm]

Fixes #1812.

[scrocquesel]

It seems that without per resource metrics we ended totally blind. Would it be possible to keep the metrics per resource kind ?

[metacosm]

It's still possible to get them but you would need to activate a flag in future versions, i.e. staring from v5 probably. This PR should keep the current behavior as-is.
Are per-resource metrics that interesting by themselves or are they symptomatic of the SDK needing maybe better, more structured logging and/or tracing, though?

[metacosm]

Also, we could indeed remove the name/namespace tags only

[scrocquesel]

Also, we could indeed remove the name/namespace tags only

+1
I guess the issue was to disable per-resource name/namespace due to label cardinality (which produces time series fragmentation) not because the event metrics by themself are not much interesting.
It still could be interesting to observe events flowing.

[metacosm]

Turns out that removing meters without delay doesn't work reliably due to the fact that cleaning operations open in different threads and heavily depend on how / when the API server responds to the SDK so I will remove the possibility of immediately removing the meters.

[metacosm]

@scrocquesel made the per-resource collection more granular and added docs, would appreciate a new review 😄

[scrocquesel]

Some few things.

Also want to share my thoughts on per schedule cleaner as I don't know if creating lots of timer has memory/perf issues. I think the objective is to keep stale meters until they have been collected one last time by the scrapper. Thus the configured delay should be a little more than the scrapping delay. We may remove meters by batch after they have spend delay seconds in a per generation collection. I guess with two generations, we can rotate every delay seconds where stale meters are recorded and clean them upon next switch.

Ideally, I would love to have some sort of a event souurce and being able to clean all stale meters when the event is raised. The event would be raised by the metric endpoint route handler after returning metrics to the scrapper. I checked quarkus route handler implementation but currenlty it is not possible to know when metrics are collected. If it make sense we may open an issue at quarkus. JOSDK could expose the a signal or method to be wired up in QOSDK.

[scrocquesel]

Shouldn't builder pattern be consistent so we have only one entry point to the configuration ? MicrometerMetrics.builder() and then be self documented with javadoc. By default, the builder is not configuring per resource and you can switch to the per resource builder if needed.

MicrometerMetrics.newMicrometerMetricsBuilder(new LoggingMeterRegistry())
    .collectingMetricsPerResource(perResourceBuilder ->
        perResourceBuilder.withCleanUpDelayInSeconds(60))
    .build();

[metacosm]

Having several entry points allows to provide often used configurations without users having to create them with the builder. It also allows to have stable API calls since these factory method implementations can be changed if we change the default behavior while the semantics of the method won't so I'd be in favor of keeping things like they are for now.

[metacosm]

Some few things.

Also want to share my thoughts on per schedule cleaner as I don't know if creating lots of timer has memory/perf issues. I think the objective is to keep stale meters until they have been collected one last time by the scrapper. Thus the configured delay should be a little more than the scrapping delay. We may remove meters by batch after they have spend delay seconds in a per generation collection. I guess with two generations, we can rotate every delay seconds where stale meters are recorded and clean them upon next switch.

I have to admit that I'm not too familiar with how the scrapping work and I'm not sure whether our implementation can actually be notified that scrapping occurred.

Ideally, I would love to have some sort of a event souurce and being able to clean all stale meters when the event is raised. The event would be raised by the metric endpoint route handler after returning metrics to the scrapper. I checked quarkus route handler implementation but currenlty it is not possible to know when metrics are collected. If it make sense we may open an issue at quarkus. JOSDK could expose the a signal or method to be wired up in QOSDK.

That's an interesting idea. I'm not sure how feasible that is, though.

[scrocquesel]

Looking at failedReconciliation, I wonder if we should also reduce cardinality on exception with a builder.includeExceptionTag().

[scrocquesel]

Find some discrepancies between doc and impl

[scrocquesel]

doc says it has resoure metadata. I guess it should reuse the tags list above. Same apply for the failure below

[metacosm]

I think we should leave it like this for this release to keep the current behavior. I'll fix the doc instead. I'm also unsure whether adding more metadata is useful for this metric (at least, the GVK can be determined by looking at the controller definition from its name).

[metacosm]

That said, the metadata is on the timers which doesn't really make sense either and should probably be removed there as well in the next major release.

[scrocquesel]

Make sense there is operator.sdk.reconciliations.failed if the user want to build alerts on failure for a specific GVK

[metacosm]

Looking at failedReconciliation, I wonder if we should also reduce cardinality on exception with a builder.includeExceptionTag().

I don't actually want to make things too configurable or too fine-grained with the default implementation. It shouldn't be too hard for people to use that as a starting point for their own implementation.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
11 Code Smells

32.8% Coverage
0.0% Duplication