Add OpenTelemetry tracing

[electrum]

Description

Add initial, experimental support for distributed tracing using OpenTelemetry. This adds traces for queries with spans for analysis, planning, optimization, split generation, tasks, remote tasks, local planning, metadata, HTTP client requests, JAX-RS endpoints, etc.

Connectors can participate in tracing using the OpenTelemetry API. As a proof-of-concept, TrinoFileSystem and the S3 file system code are instrumented for the Hive, Iceberg, Delta Lake and Hudi connectors.

Usage

Enable tracing in Trino and configure the OLTP/gRPC endpoint:

tracing.enabled=true
tracing.exporter.endpoint=http://localhost:4317

For easy testing on a single machine, such as when running IcebergMinioQueryRunnerMain, you can run Jaeger in Docker:

docker run -e COLLECTOR_OTLP_ENABLED=true -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest

Open http://localhost:16686 in your browser to view the Jaeger UI.

Additional context and related issues

• Depends on Add OpenTelemetry tracing airlift/airlift#1055
• Depends on Add tracing support airlift/discovery#49
• Fixes Add more granular query tracing events #1826
• Fixes Add hooks to call an open tracing api to feed telemetry data to various tracing systems #4207

Release notes

(x) Release notes are required, with the following suggested text:

# General
* Add experimental support for tracing using [OpenTelemetry](https://opentelemetry.io/).
  Enable tracing by setting `tracing.enabled=true` and configure the 
  [OLTP/gRPC endpoint](https://opentelemetry.io/docs/reference/specification/protocol/otlp/)
  endpoint by setting `tracing.exporter.endpoint` to override
  the default of `http://localhost:4317` if needed. The functionality, span names, attributes, etc.,
  are experimental and subject to change. ({issue}`16950`)

[sajjoseph]

Long overdue feature. Thanks @electrum for making it happen.

[wendigo]

Is there a kill switch to disable tracing? What's the added overhead?

[electrum]

@wendigo It is only enabled if you do tracing.enabled. When not enabled, it will use the OpenTelemetry no-op implementation. We shouldn't be tracing at the granularity where the overhead would be significant when not enabled.

[bitsondatadev]

This is incredible!!!!

[colebow]

@electrum - do we want to document this, do you have that covered, or should we wait until it's not experimental?

[lambrospetrou]

I looked through the code, but cannot find anything.
Is there a way we can specify the tracing.exporter.endpoint override using an environment variable instead of the config file?
In our case, we use k8s, and we need to dynamically retrieve the host IP of the OTLP exporter endpoint and then pass it to Trino as environment variable.