Runtime metrics for Metricbeat

[odacremolbap]

Describe the enhancement:

There is a good chance that every piece of software is instrumented with similar metrics regarding runtime.

This enhancement discussion should come up with a proposal on:

• common metrics that we expect to find for all runtimes
• how to map those values in an standardized way into events
• how to map values that are not standadized but part of the runtime, into events

This has been already discuseed for go here: #11694

[odacremolbap]

As an starting point, at the Go thread that originated this discussion we came to a point where we seemed to agree on having metrics grouped under runtime

We didn't expressed explicit agreement on this, but summarizing here:

• runtime metrics would fit into categories runtime.memory and runtime.process
• there are probably categories that don't apply to all runtimes but are ok to standardize like garbage collection, that could be hosted under category runtime.gc
• there might also be custom categories that depend only on the runtime. Routines are specific to go, these could be represented either under runtime.process.go or runtime.go

We should at least abstract common fields to those that are common to be instrumented at JMX, WMI, and go

[odacremolbap]

@webmat @exekias @ruflin

[ruflin]

This issue might be related: elastic/apm-agent-go#504 Would be good to standardise on the naming (cc @axw )

[ruflin]

I like the above proposal. For language specific metrics I'm on the fence which option is better. My initial instinct was going with runtime.{language} to not repeat the language multiple times but then if someone looks for all memory infos, he would not see in autocomplete the go memory info he goes to runtime.memory.*.

An other way to look at it would be to have language specific things on the top level like go.* as for each language there will be more specific things than just runtime I assume.

[axw]

Naive questions: what's the value of common metric naming? When would you query only those common metrics across arbitrary languages?

Even though each language will have stack/heap/gc/etc., they can vary wildly or subtly, which may mean they have different semantics. Also, given that they vary, I expect it would generally be more useful to view them in language-specific dashboards, e.g. for Java applications you probably want to see the GC metrics broken down by generation, but this doesn't make sense for Go.

The APM Go agent currently records metrics under the top-level golang. namespace, with names and values matching those produced by the Metricbeat Golang heap module. Beyond these, I added "golang.goroutines".

I like the suggested runtime.{language} namespacing, but we'd have to go through some deprecation/upgrade process to move off the existing metric names. What about languages where there are multiple major runtime implementations, like Java? HotSpot, IBM, Zing, etc. They have different GC implementations, among other things, so it may be useful to capture metrics specific to the runtime implementation.

[felixbarny]

They have different GC implementations, among other things, so it may be useful to capture metrics specific to the runtime implementation.

Even Hotspot has different flavors of GC algorithms. Most commonly, you also have different algorithms for the new and the old generation of the heap. The way I did it in the Java agent was to have a name label for the GC algorithm name: https://www.elastic.co/guide/en/apm/agent/java/current/metrics.html#metrics-jvm

[ruflin]

@axw It's a good question around common metric naming. If we don't expect to have 1 visualisation which should be capable of showing metrics from different runtimes, then this is not needed. Based on your comments, I assume we don't.

What we should standardise on is per language at least the metrics are the same, so if the metrics for go come from the apm agent or metricbeat, they can be visualised the same.

[axw]

What we should standardise on is per language at least the metrics are the same, so if the metrics for go come from the apm agent or metricbeat, they can be visualised the same.

Agreed.

The APM Java agent's current metrics are documented at https://www.elastic.co/guide/en/apm/agent/java/current/metrics.html#metrics-jvm. Since they're already released, I think it would be best to use those. As mentioned above, the Go agent is using what Metricbeat defines + golang.goroutines.

We'll need to think about how to keep them in sync. One option would be to nominate one of them (language-specific agent or Metricbeat) as the canonical definition of the metrics. I think APM might make sense as it's likely to have access to increasingly more runtime metrics over time.

[ruflin]

+1 on using the jvm metric names you linked above. They look well defined.

What about choosing a third place? ECS today does has almost no metrics, but I would say such standardised metrics could also fit there in the future. What about starting an "experimental" part in ECS where we put these and reference it from? Beats already depends on ECS and would hopefully get these for free then.

Having said that, I'm good with starting pulling it from APM agents. Some metrics I would also like to see standardised are the system metrics like CPU, Memory, Disk etc. Unfortunately these are in Metricbeat. Perhaps a third repo is the best place to get started and the port it later to ECS?