Add Tracing For Additional Observability

[seanmalloy]

What happens?

Currently the prometheus_bigquery_remote_storage_adapter application only supports logs and prometheus metrics for observability.

What were you expecting to happen?

I would like to be able to use tracing to monitor/observe prometheus_bigquery_remote_storage_adapter when running in a production environment.

Steps to reproduce:

Run prometheus_bigquery_remote_storage_adapter and notice that only logs and prometheus metrics are provided.

Any errors, stacktrace, logs?

N/A

Environment:

• Runtime version(Java, Go, Python, etc): N/A
• Desktop OS/version: N/A

Additional comments:

From my perspective it seems like OpenTelemetry would provide a nice vendor neutral option for enabling tracing.

[wdoogz]

@seanmalloy were you thinking about offering tracing as a flag in our code when the application is started it'll begin sending the traces to a OTel collector (such as maybe another sidecar running within the same pod as the Prom BQ remote storage adapter), which will send it to whatever backend a user wishes?

[seanmalloy]

@seanmalloy were you thinking about offering tracing as a flag in our code when the application is started it'll begin sending the traces to a OTel collector (such as maybe another sidecar running within the same pod as the Prom BQ remote storage adapter), which will send it to whatever backend a user wishes?

@wdoogz my simple answer is yes I agree with you.

I'm not an OpenTelemetry expert, but I think we would want to keep this feature as vendor agnostic as possible, so using the OpenTelemetry collector seems like a good idea.

https://opentelemetry.io/docs/collector/

Also, having a flag to enable/disable tracing seems like a good idea.