Open Telemetry custom header configuration

[apexskier]

Our observability platform (New Relic) supports Open Telemetry, and so does Hydra. I was expected to be able to set Hydra up to use Open Telemetry, but have run into a few issues.

First one was #3421.

The second is that I currently need to connect directly to New Relic and pass their API key using a header (https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/opentelemetry-setup/). (There's a chance we'll move to a collector set up, which avoid needing hydra to directly pass the api key.)

This should be possible by setting the environment variable OTEL_EXPORTER_OTLP_HEADERS="api-key=<mykey>". However, there are a few issues with that:

First, the otel go SDK doesn't support a lot of environment variables yet (see issues linked in open-telemetry/opentelemetry-go#1698). This can be worked around with otlptracehttp.WithHeaders, but that requires editing code in github.com/ory/x/otelx/otlp.go (I've verified this locally). To do this would require code changes in that repo, then updating hydra with the ory/x dependency, then adding the configuration to hydra (similar to #3421).

It would definitely be nice to support this, and other otel configuration items. Ideally this would be provided by the otel go sdk, but short of that, it's possible to add additional configuration items through the ory/x sdk.

[aeneasr]

Thank you for the thoughtful discussion! I think the best path forward would to merge the env var change upstream to otel, as that would benefit the whole otel community! :)

[apexskier]

Update: I'm not sure what's changed, but my use case appears to be working on master (but not on v2.0.3).

TRACING_PROVIDERS_OTLP_SAMPLING_SAMPLING_RATIO=1
TRACING_PROVIDER=otel
TRACING_PROVIDERS_OTLP_SERVER_URL=otlp.nr-data.net:4317
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net:4317
OTEL_EXPORTER_OTLP_HEADERS="api-key=..."
OTEL_SERVICE_NAME="oidc-server.local"
TRACING_SERVICE_NAME="oidc-server.local"

Using this configuration.

Just waiting on a new release!