feat(cmd): initial integration of OpenTelemetry with OTLP exporter

[Wondertan]

• Initial integration of OpenTelemetry with JaegerOTLP exporter
• Adds new flags switches for both tracing and metrics with endpoint customization
  • --tracing and --tracing.endpoint
  • --metrics and --metrics.endpoint
• Simple tracing coverage for ipld.Retriever
• Simple metric for current store height

P.S. OTLP integration could be separated from introduced metrics and tracing coverage, but it is the bare minimum needed to manually verify that the integration works as expected by looking at actual data reported by the local nodes.

TODO:

• Ensure different node types advertise as different services
• Use OTLP instead of Jaeger
• Integrate OTLP for metrics
• Store height metric

As always, recommend looking at the PR commit by commit(CBC)

Substitutes #810
Closes #934

[codecov-commenter]

Codecov Report

Merging #907 (255ed8c) into main (0c2daea) will decrease coverage by 0.23%.
The diff coverage is 41.24%.

@@            Coverage Diff             @@
##             main     #907      +/-   ##
==========================================
- Coverage   58.59%   58.35%   -0.24%     
==========================================
  Files         130      132       +2     
  Lines        7798     7937     +139     
==========================================
+ Hits         4569     4632      +63     
- Misses       2755     2831      +76     
  Partials      474      474              

Impacted Files	Coverage Δ
cmd/celestia/full.go	42.37% <0.00%> (ø)
cmd/celestia/light.go	42.37% <0.00%> (ø)
cmd/env.go	76.47% <ø> (ø)
header/metrics.go	0.00% <0.00%> (ø)
node/services/service.go	80.66% <0.00%> (-1.22%)	⬇️
node/settings.go	31.37% <0.00%> (-4.19%)	⬇️
cmd/flags_misc.go	42.59% <36.11%> (-11.80%)	⬇️
cmd/celestia/bridge.go	63.26% <100.00%> (ø)
ipld/retriever.go	93.58% <100.00%> (+3.19%)	⬆️
ipld/retriever_byzantine.go	69.23% <100.00%> (ø)
... and 16 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

[rootulp]

Cool! Is it possible for me to see this in action? Is there an OTEL collector that has metrics / traces emitted to it?

[Wondertan]

@rootulp, you can try running collector locally with default endpoints/port while passing to Light or Full node new flags described on the PR desc. And you should see the height of the node being reported, etc. To see traces you would need to run Full node

[renaynay]

Looks good! It would be nice to get a demo for this some time soon. One question, tracing happens regardless of whether tracing is enabled, correct? --tracing being enabled just adds the default (or custom) exporter endpoint. So if --tracing isn't passed, but tracing still occurs, does it impact performance?

[Wondertan]

One question, tracing happens regardless of whether tracing is enabled, correct? --tracing being enabled just adds the default (or custom) exporter endpoint. So if --tracing isn't passed, but tracing still occurs, does it impact performance?

That's a good question, and I am glad you asked it. I had the same concern and even looked deeper at Otel implementation to find out. My conclusion is that it won't noticeably affect performance. The default tracing exporter does nothing besides keeping information about spans to be delegated if tracing is turned on in runtime(see #937).

The same is true for metrics. The default exporter does nothing besides logic to delegate to some custom exporter that can be enabled in runtime. I could potentially make MonitorHead always called without adding any options for the node to enable it, similar to tracing, but I am still unsure about this. Mainly the confusion here comes from how to marry the global meter pattern with an instance of Store, etc, and the current result looks clean.

[rootulp]

Screenshot of a Grafana dashboard featuring the head metric (implemented in this PR). This is for a light node I'm running on AWS which is emitting metrics to an OTEL Collector deployed on the same instance. The OTEL Collector is exporting metrics to Prometheus that is hosted by Grafana Cloud. Setup instructions are in #922

tagging @renaynay because I'm happy to hop on a call and demo this or set it up for you.

[renaynay]

@rootulp would love to do a demo on how to set it up, thanks!

[renaynay]

👍🏻