Introducing Application Telemetry Schema in OpenTelemetry - Vision and Roadmap

[lquerel]

[Easy to read version with images: https://github.com/lquerel/oteps/blob/app-telemetry-schema-vision-roadmap/text/0243-app-telemetry-schema-vision-roadmap.md]

Unlike the traditional data ecosystem (OLTP and OLAP), the world of telemetry generally does not rely on the concept of a schema. Instrumentation is deeply embedded in the code of applications and libraries, making it difficult to discover all the possible telemetry signals an application can emit. This gap prevents or limits the development of CI/CD tools for checking, reporting, documenting, and generating artifacts from telemetry signals specific to an application. This document presents a long-term vision aimed at enabling the OpenTelemetry project to address this issue and extend its impact to a broader ecosystem. It proposes extending the initiatives of Telemetry Schema and Semantic Conventions to include concepts of Application Telemetry Schema and Resolved Telemetry Schema. A series of OTEPs and Tools will be proposed in this overarching document to detail each aspect of this vision.

Similar (but proprietary) initiative from Facebook: Positional Paper: Schema-First Application Telemetry

EDIT 1: The OTel Weaver project (a PoC implementation of this OTEP and some of the others mentioned in the roadmap) is now available here.

EDIT 2: The Slack channel #otel-weaver is dedicated to this OTEP and the associated OTel Weaver project.

[yurishkuro]

Easier to read version with images: https://github.com/lquerel/oteps/blob/app-telemetry-schema-vision-roadmap/text/0240-app-telemetry-schema-vision-roadmap.md

[tigrannajaryan]

@lquerel thank you for working on this, I think schemas are very important for Otel's evolution.

I already had a chance to discuss the roadmap and I support it. The details of each steps of course are going to be important and will need their own OTEPs or spec PRs.

[pyohannes]

This is great, thanks @lquerel for this bold and ambitious proposal.

[jmacd]

I support this vision and roadmap and look forward to reviewing future OTEPs.

[lquerel]

@yurishkuro @tigrannajaryan @jmacd @pyohannes @jsuereth @lmolkova

I have added the following notes to the Telemetry Schema v1.2 section to clarify a few important concepts that were omitted in the first version of this document. I don't believe there is anything controversial in these additions, but please read them and let me know if you have any disagreements or need further clarification.

Note 1: Each signal definition, where possible, reuses the existing syntax and semantics defined by the semantic conventions. Each signal definition is also identified by a unique name (or ID), making schemas easy to traverse, validate, and diff.

Note 2: This hierarchy of telemetry schemas helps large organizations in collaborating on the Application Telemetry Schema. It enables different aspects of an Application Telemetry Schema to be managed by various teams.

Note 3: For all the elements that make up the Application Telemetry Schema, a general mechanism of annotation or tagging will be integrated in order to attach additional traits, characteristics, or constraints, allowing vendors and companies to extend the definition of concepts defined by OpenTelemetry.

Note 4: Annotations or Tags can also be employed to modify schemas for diverse audiences. For example, the public version of a schema can exclude all signals or other metadata labeled as private. Similarly, elements can be designated as exclusively available for beta testers. These annotations can also identify attributes as PII (Personally Identifiable Information), and privacy policy enforcement can be implemented at various levels (e.g., in the generated client SDK or in a proxy).

EDIT: added another note on the relationship between Semantic Layers, Telemetry Schemas, and Observability Query Assistant.

Note 5: This recent paper from data.world, along with the MetricFlow framework which underpins the dbt Semantic Layer, highlights the significance of adopting a schema-first approach in data modeling, especially for Generative AI-based question answering systems. Tools like Observability Query Assistants (e.g. Elastic AI Assistant and Honeycomb Query Assistant) are likely to become increasingly prevalent and efficient in the near future, thanks to the adoption of a schema-first approach.

[mtwo]

@jmacd can you also add the 'triaged' label?

[tigrannajaryan]

I had some old unpublished comments remaining from reviewing the previous iteration of the OTEP, which accidentally got published when I approved this OTEP. Ignore the comments if you receive them in emails, they are no longer relevant. I also deleted the comments from this PR to avoid confusion.

[jmacd]

This content is substantially similar to the copy I reviewed last time, with less content so that we can discuss/debate more in future OTEPs. Thanks @lquerel!

[tigrannajaryan]

@open-telemetry/specs-approvers please review.

[tigrannajaryan]

@open-telemetry/specs-approvers this is a major OTEP that will impact many parts of Otel. We have enough approvals to merge it. Last call to review it.

[tigrannajaryan]

Thank you @lquerel for the OTEP!