feat(cli): enable traces export via tracing-otlp cli arg

[lean-apple]

Closes #18241.

Add new CLI flag tracing-otlp to export opentelemetry spans either to stdout or to an OTLP HTTP endpoint.

• Stdout export: --tracing-otlp
• OTLP endpoint export: --tracing-otlp=http://localhost:4318/v1/traces

Here how I tested :

Example of very simple otel-collector configuration's file, limited for traces export :

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

and then I put up the collector using this config :

docker run -p 4318:4318 \
  -v $(pwd)/test-otlp.yaml:/etc/otel-collector-config.yaml \
  otel/opentelemetry-collector:latest \
  --config=/etc/otel-collector-config.yaml

The build for the node should be done with otlp feature for example cargo build --release -p op-reth --features reth-optimism-cli/otlp.

and then ./target/release/op-reth --tracing-otlp=http://localhost:4318/v1/traces node --dev

or

cargo run --release -p op-reth --features reth-optimism-cli/otlp -- --tracing-otlp=http://localhost:4318/v1/traces node --dev

Depending on the type provided to --tracing-otlp, either it will just print the spans, either you can see them being redirected to the otlp-collector.
cc @Rjected

[mattsse]

this should be super useful

we maybe also want to add this to rpc as well?

i dont know much about otlp, maybe @onbjerg could also take a look here?

[shekhirin]

@lean-apple is this ready for review and can be moved out of a draft?

[lean-apple]

Thanks for your review @Rjected :), Addressed most of the comments, open to remove the stdout option if you think it would be useless to keep it.

Let me know if you could try and have some other configuration idea/fix in mind!

[Rjected]

thanks! LGTM

Dan reviewed the PR.

[Zygimantass]

Not sure if this should be a part of another PR or this one, but in this current state, the traces / spans aren't super useful as an operator.

Running in dev mode and using Jaeger, I can see that all traces contain only one span, meaning every event happening on the node is "disparate" and unconnected to other events. How I imagine it should (?) be is that every block (or another unit of work) has a trace that is propagated through all events down stream: block building, state root calculation, executing the block, etc.

Another piece of feedback is that none of the spans / traces have any context attached to them, meaning that I can't see whether execute_block was for block 0 or 100.

EDIT:

there's a actually a hacky way to get the block number, but you have to do it through the attached logs, instead of the actual span tags

[Zygimantass]

I'm also only getting 6 types of events (looking at the Reth codebase, there should definitely be more than that?)

[Zygimantass]

I think this might cause the exporter to only export reth::cli related spans, instead of exporting everything globally

[Rjected]

just changed this to reth, not super confident on this but I think this just determines the service name?

[lean-apple]

Yes it's the service-name, maybe we can this also as configurable, with reth by default

[lean-apple]

Running in dev mode and using Jaeger, I can see that all traces contain only one span, meaning every event happening on the node is "disparate" and unconnected to other events. How I imagine it should (?) be is that every block (or another unit of work) has a trace that is propagated through all events down stream: block building, state root calculation, executing the block, etc.

@Zygimantass I'm not sure right now there is a full propagation of spans and if the tracing context is kept so, which would help to have less disparate events on jaeger and enable to follow to full workflow.