Support property based MeterFilters

[philwebb]

micrometer-metrics/micrometer#366 / d034a7f

[jkschneider]

To help you better imagine the use case of using filters to control distribution statistics, consider timers that are instrumented in low-level core libraries (like the instrumentation that now exists in HikariCP and Rabbit). These authors can't make a general-purpose decision about whether it is useful for users of their library to see certain summary stats like client-side percentiles. For example:

1. Billy is using InfluxDB, which does not have a rich-enough query language to compute aggregable percentiles from histograms. Billy might choose to ship a Micrometer-computed P95, knowing that the usefulness of this is limited to the tag cardinality of the underlying metric. For example, if the timer is tagged with "host" and "result" and his organization only runs 10 service instances where there are 2 possible results, he could plausibly plot a non-aggregable P95 by simply plotting all the individual time series independently and monitoring for outliers.

2. Sue is using Prometheus and never wants to ship Micrometer-computed percentiles because she has access to histogram_quantile and so the usefulness of her plots doesn't need to be constrained by tag cardinality. She chooses to ship the histogram buckets.

3. Jane is using Wavefront and doesn't care about percentile distributions at all. She is comfortable monitoring the range between the distribution average (a measure of centrality that is neither better or worse than P50 in the general case) and max. Wavefront publishes a blog about their new aggregable percentiles feature and Jane gets P99 on key latency metrics via an additional property.

4. Organization has adopted a "instrument all the things" approach but are worried about the scalability of their on-prem Prometheus cluster. If their Prometheus cluster ever gets stressed, they want to be able to drop summary stats on some less-critical latency metrics and perhaps disable whole swaths of other metrics. They want to accomplish both in a property-driven way such that they could ease pressure on their monitoring system across their whole stack with a combination of something like Archaius/Config Server.

There is no way for the Rabbit client author to satisfy all of these needs without shipping wasteful extra time series to each of our imaginary users. Our recommendation for library authors is always just instrument with plain timers and let application developers decide at the last minute which summary statistics they want via filters.

[snicoll]

@jkschneider does it have to be property-based though? An application developer may craft the stats that they want based on what is available using a DSL or something? The number of options is dense and the fact that we had to restructure the namespace is a sign that perhaps a DSL would be a better fit?

[jkschneider]

Ideally it's property based, yes. We want folks to be able to leverage config server to dynamically influence enablement/fidelity of certain metrics in the long term.

[jkschneider]

the fact that we had to restructure the namespace is a sign that perhaps a DSL would be a better fit?

Perhaps it is just a sign that the original structure was a poor design. After all, I came up with it ;)

[philwebb]

See also this comment when looking at the implementation.

[philwebb]

@jkschneider I've been thinking some more about this and I'm wondering if something like we do with OAuth2ClientProperties might work? What if we split the properties into two parts? A named definition and then a reference to it?

Something like

metrics.config.light.percentiles=1,2,3
metrics.config.light.somethingelse=forsure

metrics.config.heavy.percentiles=10,20,30
metrics.config.light.somethingelse=noway

metrics.apply.net.foo.bar=light
metrics.apply.net.foo.baz=light
metrics.apply.net.foo=heavy

Would that extra level of indirection be useful?

We could even provide built in configs for common scenarios.