feat: Usable out of the box metrics

[MarcoPolo]

resource manager should expose metrics around the current resource usage. And aggregating where necessary.

It doesn't make sense to record every peer's resource usage, but aggregates would still be very valuable (avg, p90, p99, max, min) to operators.

In most use cases the end-user will want this information, and we already record this so we should expose.

[marten-seemann]

Should this be part of libp2p/go-libp2p#1356? Unfortunately, the API we were going to use is still in Alpha.

[MarcoPolo]

It could be, but this is tightly scoped and I don't think it should be blocked on 1356 or on waiting for opentelemetry metrics to come out of alpha.

[BigLep]

I know there is work underway on this. I'm putting in my notes for the high-level dashboard I think would be useful for an operator to see:

(mocked from here)

The system/transient graphs straight copy/pastes from the observable. The other scops are showing an aggregation. At the minimum when it comes to aggregation I would want to see:

• n - the number of values that were aggregated. It should be graphed on a separate axis (left axis). For example, with peer scope, at the time of metric datapoint generation, it would denote how many peers were in that metric interval.
• p50 - at the time of datapoint generation, would show the median value. It should be on a separate axis (right axis)
• p90 - show the 90th percentil
• p100 - show the max value

We could obviously have other percentiles too.
If percentiles are too expensive to calculate because of the sorting required we could get away with average and max for now.

I could imagine an operator seeing that dashboard above, and if things look fishy, can then do the tracing/observable route for deeper dive to analyze.

[BigLep]

Once this feature is developed, we also need to make sure user's can discover it easily enough. There is a docs issue to give visibility to it here: libp2p/docs#158