OTEP: Pluggable Authentication Interface for OTLP Exporters in SDKs#4500
OTEP: Pluggable Authentication Interface for OTLP Exporters in SDKs#4500srprash wants to merge 10 commits intoopen-telemetry:mainfrom
Conversation
|
|
||
| ### Proposed Authenticator Interface | ||
|
|
||
| Each SDK would define an Authenticator interface (or class, depending on the language and the exporter client), such as: |
There was a problem hiding this comment.
Do we want this to be provided by the SDK or the OTLP exporter?
There was a problem hiding this comment.
Uh, OTLP exporters I believe. Sorry, what's the difference between the two? Isn't exporter a part of the SDK?
There was a problem hiding this comment.
I think there could be two positions:
- This authenticator interface will be provided by the OTLP exporter. In other words, if I use the OpenTelemetry SDK with let's say Zipkin exporter, I shouldn't even see/know this interface.
- This interface will be a shared infrastructure provided by the SDK, any exporter can choose to use it if they want. In addition, OTLP exporter must support it.
There was a problem hiding this comment.
I see. I think the actual authenticator interface will be tied to the http/grpc client in that language. For example in python, the interface can be def authenticate(self, request: requests.PreparedRequest) which is from the requests library.
This interface may not apply to non-OTLP exporters if they are not using the same client library. Option 2 could be confusing to the users of such exporters.
There was a problem hiding this comment.
I think making the interface part of OTLP exporters is a good idea, despite auth being a more general concept because the function signatures in the interface will be tied to http/grpc client implementation (if this is not true, then perhaps we can have a single interface).
I also think it might be a good idea to name this interface something like OtlpAuthenticator and IIUC, it would have different authenticate methods for http and grpc - and the SDK will call the correct method depending on the config.
|
|
||
| ### Implementation Suggestion | ||
|
|
||
| The initial implementation can be language-specific and start with HTTP-based exporters, which are simpler to modify. For example, in Python, the OTLP HTTP exporter can accept a new authenticator parameter, and modify its send() method to invoke `authenticator.authenticate(request)` before dispatch. |
There was a problem hiding this comment.
Can you clarify how this would work ? Like would you modify the http exporter code
There was a problem hiding this comment.
Yes. The OTLP HTTP exporters need to change to accommodate this functionality. In python, the _export() method is duplicated across all three exporters, so we can consider breaking it out into a common class.
In my mind, the updated _export() method would look like this once the payload has been serialized and compression applied.
req = Request(
method='POST',
url=self._endpoint,
data=data
)
prepped = self._session.prepare_request(req)
# call to authenticate the request.
# authenticate can add auth headers to this prepared request and do anything that's needed.
self._authenticator.authenticate(prepped)
resp = self._session.send(
request=prepped,
verify=self._certificate_file,
timeout=self._timeout,
cert=self._client_cert
)This uses a prepared request, a pattern which I believe should be available in other languages as well.
There was a problem hiding this comment.
I think my issues with this is it requires us (Google) to build something custom to get auth working for otel, when we already have overrides of the existing requests.Session, and grpc.ChannelCredential objects which the OTLP exporters already accept as params and which can handle custom auth. Why build something new when the existing OSS libraries being used already have ways that allow for customizing auth ?
| - Exporters should treat authentication failures as transient errors and apply standard retry logic. | ||
| - Encourage authenticators to expose metrics or logs to help users diagnose failures. | ||
|
|
||
| ## Prior art and alternatives |
There was a problem hiding this comment.
What about other designs (more generic that would handle more than just a custom authenticator) and features which are already in the existing OTLP exporter? E.g. for Go we have recently added the possibility to add a custom http.Client (open-telemetry/opentelemetry-go#6688) which allows using HTTP clients that handle authentication such as https://pkg.go.dev/golang.org/x/oauth2/clientcredentials#Config.Client.
Maybe we should just have some high-level requirement that the OTLP exporters should be able to be configured with different authentication schemes and workflows
There was a problem hiding this comment.
What about other designs (more generic that would handle more than just a custom authenticator) and features which are already in the existing OTLP exporter?
I would like to keep this OTEP focused on authentication only. Increasing the scope to having a more generic design for auth and beyond will surely increase OTel's surface area for potential problems and upkeep. I don't know if the community will be open for that.
There was a problem hiding this comment.
My opinion is the opposite. Authentication is just one of use cases when the users need a possibilities to customize (or pass their own) HTTP/gRPC clients used by the OTLP exporters.
Some people may be worried that it would couple the exporter to an HTTP/gRPC client library but if there is any problem that would require changing it then we can always make a major bump of given OTLP exporter.
There was a problem hiding this comment.
I believe this proposal is just about adding an interface that would by default be no-op, unless the user provides an authenticator to the SDK.
So IIUC, the coupling would be between the authenticator and the HTTP/gRPC clients. So like for the Go example, the object passed to WithHttpClient would implement this OtlpAuthenticator interface.
There was a problem hiding this comment.
@pellared while I agree that auth is just one of the use-cases, i'm not sure if we will be able to make progress on the OTEP if the scope is too large. It may just be that Go's HTTP "authenticator" interface needs to be overly broad (e.g. provides the entire HTTP client). From what i've seen of Java's auth plugin, it would be ideal for them to have "header-providers" functions or similar instead. I would prefer to not specify the interfaces too precisely, but to ensure it can support a list of auth mechanisms instead. As long as they can all be loaded from the same file-based config, I don't know if the interfaces themselves matter too much.
There was a problem hiding this comment.
I think that in order to go forward, this OTEP (and spec) should be more focused on specifying "capabilities" (the potential or ability of a component to perform specific tasks) rather than "interfaces" (means by which different components interact and exchange information).
|
@DylanRussell @pellared I believe you bring up a common concern where some of the SDKs already have a way (although not a direct one) to achieve authentication either via custom sessions, clients, or something else that can be plugged into the OTLP exporters. Also, we aim to keep the proposed change backwards compatible. So existing users of Google's custom session can continue to use it or switch to using the new authentication plugin if Google decides to build one in the future. |
Could you also cover the SDKs that already have a way (although not a direct one) to achieve authentication? It would be good to call out whether they should also implement something. It would be also good if other SDKs can also provide similar mechanism instead of providing a feature that only covers authentication. |
I think it is a good idea to mandate that each exporter provide a way that vendors/users can use to customize auth, but is it critical that they all do it the same way, or is it OK for each to come up with it's own slightly different mechanism ? For example for python, the HTTP exporter could just be updated to accept a For gRPC there is a standard auth plugin that exists across languages: https://grpc.io/docs/guides/auth/#extending-grpc-to-support-other-authentication-mechanisms . It is not something that each language makes configurable yet (see open-telemetry/opentelemetry-cpp#3403), so maybe what this OTEP should say for gRPC is that ChannelCredentials must be something that can be based into the gRPC exporters, and that we need to figure out how it can be supplied when auto instrumentation is used ? |
|
|
||
| ```python | ||
| class Authenticator: | ||
| def authenticate(self, request: requests.PreparedRequest) -> None: |
There was a problem hiding this comment.
as discussed during Spec call, please take a look at the HTTP auth flow: https://datatracker.ietf.org/doc/html/rfc7235, https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Authentication#the_general_http_authentication_framework
It typically looks like this:
- optional, but highly desirable: populate pre-cached token on the request
- issue a request
- receive 401 response with
WWW-Authenticate(Proxy-Authenticate, etc) header specifying additional authentication parameters - get auth token using parameters returned in response headers
- re-issue a request with updated headers including new token
- optionally: cache the token to reuse on consequent requests
It does not seem like this API can support a typical auth scheme like this.
There was a problem hiding this comment.
In addition, there might need to be asynchronous requests (e.g. to refresh a token in a cookie) that need to be sent through the same requests session. Authentication can't just be handled on a per-request basis.
|
This is introducing a very large security vulnerability target. We are going to maintain clients with elevated permissions to sensitive endpoints. Does OpenTelemetry want to focus its efforts on maintaining this complex and highly sensitive piece of code? Can we rely on existing solutions within languages instead? At a minimum there needs to be effort and documented thought put into the security implications of this proposal. |
|
|
||
| - Multiple Signals, One Authenticator: If users share a single authenticator across trace, log, and metric exporters, SDKs must not cache or mutate state inside the exporters in ways that interfere with this usage. | ||
|
|
||
| ### Implementation Suggestion |
There was a problem hiding this comment.
How would this work in languages like Go and Rust that require all code to be linked at compile-time? Would you force users to recompile their code to add the new authenticator, like they already have to do today with new exporters?
|
|
||
| ```python | ||
| class Authenticator: | ||
| def authenticate(self, request: requests.PreparedRequest) -> None: |
There was a problem hiding this comment.
In addition, there might need to be asynchronous requests (e.g. to refresh a token in a cookie) that need to be sent through the same requests session. Authentication can't just be handled on a per-request basis.
|
|
||
| While the OpenTelemetry Collector supports pluggable `auth` extensions (e.g., [sigv4auth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/sigv4authextension), [oauth2clientauth](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/oauth2clientauthextension)), the SDKs do not offer a standardized way to add authentication to OTLP exporters. Today, SDK users resort to authentication workarounds available within some SDKs (like [header supplier in Java](https://opentelemetry.io/docs/languages/java/sdk/#authentication), and [custom requests session in Python](https://github.com/open-telemetry/opentelemetry-python/issues/4459#issuecomment-2711675191)) but these are limited in capabilities and are not standard across all languages. Otherwise SDK users must either fork exporter implementations or build custom wrappers to inject authentication logic—leading to fragmentation and duplicated effort. | ||
|
|
||
| This OTEP proposes adding a standard pluggable authentication interface to OTLP exporters in SDKs. It enables users to inject custom authenticators (e.g., for SigV4 or OAuth2) without altering exporter internals. This improves portability, simplifies integration with secured backends, and brings SDKs closer to feature parity with the Collector—empowering secure, direct export from applications without requiring a Collector. |
There was a problem hiding this comment.
What is the advantage to defining a new authentication interface over defining a new exporter that wraps the existing OTLP exporter? This says "[the] user[…] must[…]fork exporter implementations", but if the existing OTLP exporter already provides options to provide a custom requests/gRPC session, why couldn't an authenticator just be exposed as a new exporter that wraps that?
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
|
Hey. Sorry folks. I got sidelined with other work. I still do want to work on this and hopefully will be able to make time in this week or next. Please reopen this PR. TY. |
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
|
Not stale. :) |
|
|
||
| Each OTLP exporter (for traces, metrics, and logs) exposes a new optional configuration parameter `authenticator` which allows users to plug in a custom authenticator implementation. | ||
|
|
||
| An authenticator is a user-defined component that modifies outgoing requests to add the appropriate authentication metadata, such as headers or tokens. This plugin model is conceptually similar to the `auth` extensions used in the OpenTelemetry Collector, but scoped for SDK usage. |
There was a problem hiding this comment.
I thing we should consider HTTP and gRPC auth providers separately. gRPC has very clearly defined support for auth, with most languages already having well-defined interfaces for doing auth with gRPC.
HTTP on the other hand will need something different, but I think that is already captured with other discussions here.
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
|
Hi all. Thanks for the comments on this proposal. Really helpful. Looking at this with fresh eyes, I see that:
Let me know if this makes sense and if I have missed something. I can work on updating the proposal if we agree on the above points. |
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
|
Hey folks. I think the simplest capability that could address all the auth related needs is that SDKs allow users to inject custom HTTP or gRPC clients to the OTLP exporters. Go and Python (in a way) already do this but it may not be feasible for languages such as Java where clients are abstracted away and cannot be exposed. |
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
|
@srprash, is your plan to have this PR closed or maybe you want to come up with some OTEP or specification changes (e.g. by adding some notes) that summarizes the outcome around the discussion we all had: #4500 (comment)? |
|
This PR was marked stale due to lack of activity. It will be closed in 7 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
10 participants
Hi all,
Following discussions in recent OpenTelemetry SIG meetings regarding the need for supporting authentication plugins in SDK exporters, it was agreed that a formal proposal should be developed to standardize this capability across languages.
This requirement has been brought up in the past via multiple issues in defferent forms:
This PR introduces an OTEP that proposes a pluggable authenticator interface for OTLP exporters within OpenTelemetry SDKs. The goal is to enable users to authenticate telemetry export requests without requiring custom exporter implementations.
I’m opening this PR to initiate a broader discussion within the community and to collaboratively shape the direction of this feature.
I’ll update the description with additional context or background links as the discussion evolves. Looking forward to your feedback and contributions.