Switch to OTel metrics

[christos68k]

Summary

This PR adds OTel metrics instrumentation and cleans up the relevant reporter interfaces. My initial implementation kept the MetricsReporter abstraction but I then decided to remove all the previous metric reporting logic (that was not OTel-compliant) in order to simplify the code and keep OTel metrics instrumentation in a single place.

This is an initial attempt at introducing OTel instrumentation, deeper architectural restructuring could be made but I think this is a good first step. Note that I'm not introducing a meter provider for OTLP metrics: if the agent is running as an OTel collector receiver, the expectation is that the global meter provider that the OTel collector configures will be used. If the agent is running standalone and reporting via OTLP, we could introduce a meter provider and an OTLP metrics exporter in a follow-up PR (assuming we think this is needed).

Review commit-by-commit might be easier. The last commit will be removed before merging and contains a meter provider, stdout exporter for testing. We'd follow a similar route if we wanted to add an OTLP metrics exporter to the OTLP reporter.

TODO:

• Exporter example for testing (stdout)
• Mark unused metrics as obsolete

[christos68k]

I'm using the name field from metrics.json here, another option would be to use field which is "namespaced" and maybe easier to parse. Both options abide by the OTel instrument naming requirements.

[christos68k]

I pushed #350 which should fix the ARM64 test failures.

[christos68k]

I measured elapsed time for this loop and it's in the tens-of-microseconds range, which means that performance is not an issue.

[florianl]

Should there be a way so that this can be disabled by configuration?

[christos68k]

Configuration normally takes place at the meter provider level which is either part of OTel collector (if agent runs inside OTel collector) or if we introduce a meter provider and OTLP metrics exporter in the otlp_reporter this is where we'd add it.

If a meter provider is not configured, every metering operation is a NOP.

[florianl]

To be able to cancel the operation and not get blocked, it might be best to propagate the context to the function and us it.

Suggested change

	ctx := context.Background()
	func report(ctx context.Context)

[christos68k]

If we add context it will propagate out of report and into every Add/AddSlice operation we do in the agent, which is what I was trying to avoid as it could get ugly (e.g. ProcessManager is caching AddSlice). Based on the initial research that I did, it didn't seem possible that a metering operation would block (this is also incompatible with metering being perfomant enough to be inserted in hot loops) so that would make propagating context for this reason - avoid blocking - unnecessary. I'll dig some more.

[florianl]

Could we simplify this loop and just do a for _, metric := range metricsBuffer { instead?

[christos68k]

metricsBuffer is statically sized by worst-case (IDMax) and the number of metrics it actually contains is given by nMetrics, so we can't iterate over the entire slice (we could check against Metric{} or Metric.ID == 0 during the iteration but I think the current approach is cleaner).

[rockdaboot]

I roughly agree with Florian here.
for _, metric := range metricsBuffer[0:nMetrics] { would do it. Avoiding i here increases readability - otherwise when reading the code the first time, one tries to find if i is used in the loop body, which can be avoided.