OpenTelemetry, Summaries and TDigests

[pedrolamarao]

I am raising this topic as a Github Issue for lack of a better place. If there is a more appropriate forum to raise such topics, please advise.

I have recently adopted OpenTelemetry as standard interface for integrating my components and telemetry infrastructure with great success, except for one point: computing quantiles with precision.

OpenTelemetry Metrics initial stable release offers instruments with type counter, gauge and histogram. It did at some point also offer type summary, which was all about quantiles, but those were removed before the first stable release. This act was captured in the merge request below. In my understanding of the issue, there were concerns over the "summability" of summaries.
open-telemetry/opentelemetry-proto#199

At the moment I read that discussion, and formed my understanding of the problem, immediately T-Digests came to mind. I opened a discussion topic at the OpenTelemetry Specification forum to point how appropriate T-Digests seem to solve the concerns raised for summary instruments.
open-telemetry/opentelemetry-specification#3118

Both T-Digest and OpenTelemetry have proven excellent components and I would love to see them working together!

[tdunning]

You are correct that t-digests are summable in just the way that open telemetry seems to need.

…

On Tue, Feb 7, 2023 at 3:37 PM Pedro Lamarão ***@***.***> wrote: I am raising this topic as a Github Issue for lack of a better place. If this there is a more appropriate forum to raise such topics, please advise. I have recently adopted OpenTelemetry as standard interface for integrating my components and telemetry infrastructure with great success, except for one point: computing quantiles with precision. OpenTelemetry Metrics initial stable release offers instruments with type counter, gauge and histogram. It did at some point also offer type summary, which was all about quantiles, but those were removed before the first stable release. This act was captured in the merge request below. In my understanding of the issue, there were concerns over the "summability" of summaries. open-telemetry/opentelemetry-proto#199 <open-telemetry/opentelemetry-proto#199> At the moment I read that discussion, and formed my understanding of the problem, immediately T-Digests came to mind. I opened a discussion topic at the OpenTelemetry Specification forum to point how appropriate T-Digests seem to solve the concerns raised for summary instruments. open-telemetry/opentelemetry-specification#3118 <open-telemetry/opentelemetry-specification#3118> Both T-Digest and OpenTelemetry have proven excellent components and I would love to see them working together! — Reply to this email directly, view it on GitHub <#206>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5E6UGKONJM4IXEW6W7YLWWLFCJANCNFSM6AAAAAAUURDGWI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[pedrolamarao]

Here is a link for the data model specification for OpenTelemetry Metrics:
https://opentelemetry.io/docs/reference/specification/metrics/data-model/

I am requesting guidance from the OpenTelemetry Java community on how to add new experimental instruments.

I am not familiar with T-Digest serialization format. Where can I learn more? It will be necessary to map this format onto JSON and Protobuf to fully support the OpenTelemetry Protocol -- which in turn allows bridging with the world via OpenTelemetry Collector.

[tdunning]

The easiest path would be to define a native protobuf serializer. Happy to work with you on this.

…

On Wed, Feb 8, 2023 at 8:55 AM Pedro Lamarão ***@***.***> wrote: Here is a link for the data model specification for OpenTelemetry Metrics: https://opentelemetry.io/docs/reference/specification/metrics/data-model/ I am requesting guidance from the OpenTelemetry Java community on how to add new experimental instruments. I am not familiar with T-Digest serialization format. Where can I learn more? I will be necessary to map this format onto JSON and Protobuf to fully support the OpenTelemetry Protocol -- which in turn allows bridging with the world via OpenTelemetry Collector. — Reply to this email directly, view it on GitHub <#206 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5E6UYFJRLO2BOJ4MAB4LWWO6V7ANCNFSM6AAAAAAUURDGWI> . You are receiving this because you commented.Message ID: ***@***.***>

[pedrolamarao]

I'll fork the project once more and take a look at the code to see if I can get up to speed.
@tdunning could you please take a look at the discussion below on OpenTelemetry Java Github forum?
There was a question about serialization which I am not ready to answer.
open-telemetry/opentelemetry-java#5186

[tdunning]

Actually, I would like to keep t-digest itself without any dependencies. So let's create a new package with the protobuf serializer.

…

On Thu, Feb 9, 2023 at 7:54 AM Pedro Lamarão ***@***.***> wrote: I'll fork the project once more and take a look at the code to see if I can get up to speed. @tdunning <https://github.com/tdunning> could you please take a look at the discussion below on OpenTelemetry Java Github forum? There was a question about serialization which I am not ready to answer. open-telemetry/opentelemetry-java#5186 <open-telemetry/opentelemetry-java#5186> — Reply to this email directly, view it on GitHub <#206 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5E6RY7RSVBOE7SNNGEVDWWUHMTANCNFSM6AAAAAAUURDGWI> . You are receiving this because you were mentioned.Message ID: ***@***.***>