Add schema URL support

[tarekgh]

open-telemetry/opentelemetry-dotnet#2417
https://github.com/open-telemetry/opentelemetry-specification/pull/1666/files

Telemetry sources such as instrumented applications and consumers of telemetry such as observability backends sometimes make implicit assumptions about the emitted telemetry. They assume that the telemetry will contain certain attributes or otherwise have a certain shape and composition of data. Essentially there is a coupling between 3 parties: OpenTelemetry semantic conventions, telemetry sources and telemetry consumers. The coupling complicates the independent evolution of these 3 parties. The 3 parties should be able to evolve independently over time and Telemetry Schemas are central to how we make this possible.

Proposal

namespace System.Diagnostics
{
    public sealed class ActivitySource : IDisposable
    {
        public ActivitySource(string name, string? version = "") { ... }
+       public ActivitySource(string name, string? version, string? telemetrySchemaUrl) { ... }
+       public string? TelemetrySchemaUrl { get; }
    }
}

namespace System.Diagnostics.Metrics
{
    public class Meter : IDisposable
    {
        public Meter(string name) { }
        public Meter(string name, string? version)  { }
+       public Meter(string name, string? version, string? telemetrySchemaUrl)  { }
+       public string? TelemetrySchemaUrl { get; }
    }
}

Usage

    private static ActivitySource s_source = new ActivitySource("MyLibrary.MyFeature", "2.0", "https://opentelemetry.io/schemas/1.2.0");
    private static Meter s_meter = new Meter("MyLibrary.MyFeature", "2.0", "https://opentelemetry.io/schemas/1.2.0");

[ghost]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-activity
See info in area-owners.md if you want to be subscribed.

Issue Details

---

open-telemetry/opentelemetry-dotnet#2417
https://github.com/open-telemetry/opentelemetry-specification/pull/1666/files

Author:	tarekgh
Assignees:	-
Labels:	feature request, area-System.Diagnostics.Activity
Milestone:	7.0.0

[tarekgh]

We are going to wait for finalizing the semantic convention discussion on OpenTelemetry side to know if there will be any effect on this current proposal. We expect to have the feedback by the end of March or early April.

CC @cijothomas @reyang @denisivan0v

[CodeBlanch]

Just looking at open-telemetry/opentelemetry-dotnet#3445. There are cases where instrumentation does not own the ActivitySource being used (ex AspNetCore, HttpClient). In those cases it might be helpful to make some of this data settable?

namespace System.Diagnostics
{
    public sealed class ActivitySource : IDisposable
    {
        public ActivitySource(string name, string? version = "") { ... }
+       public ActivitySource(string name, string? version, string? telemetrySchemaUrl) { ... }
        public string? Version { 
          get;
+         set;
        }
+       public string? TelemetrySchemaUrl { get; set; }
    }
}

That would allow (for example) OpenTelemetry instrumentation augmenting the sources to pass along its version + schema url for the schema being augmented.

Breaks if users have multiple augmenting libraries/schemas, but may not be a concern?

[danelson]

@tarekgh I am curious if this was revisited at all with some of the changes made for .NET 8 (e.g. https://github.com/lmolkova/semantic-conventions/blob/dotnet8-metrics/docs/dotnet/dotnet-http-metrics.md). There was a comment that mentioned staying compatible with OTel but I am a bit concerned that this will be hard for consumers to manage if there are additional changes without the existence of schema URL. Do you have any idea when I might be able to expect this change?

Speaking from someone whose organization sticks to mainly LTS releases, if this does not make .NET 8 then it becomes hard to manage across various language SDKs.

[tarekgh]

@danelson the semantic convention discussion came very late in the 8.0 after passing the point of adding a new API. This is something we can consider for our next release.

CC @reyang @noahfalk @CodeBlanch

[cijothomas]

if this does not make .NET 8 then it becomes hard to manage across various language SDKs.

I doubt if this is going to be part of .NET 8 as it is too late. The earliest, hence, would be .NET 9. And I don' think it is relevant that your org recommends LTS only, as this change will be part of DiagnosticSource package, which is out-of-band, and is not tired to the actual .NET runtime.

I believe the main reason why OTel .NET has not pushed for this to land in .NET 8 was that, the use cases of the schema url are still an experimental spec:
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/telemetry-stability.md

(the schema_url part of tracer/meter itself is stable long ago, the above is just the spec about the schemas/ and its transformations etc.)

[pellared]

Does it mean that we won't get any stable OpenTelemetry .NET instrumentation library before .NET 9 release?

Take notice that the schema URL support is required by OpenTelemetry specification since 2021-06-07 (sic!).

[cijothomas]

Does it mean that we won't get any stable OpenTelemetry .NET instrumentation library before .NET 9 release?

I don't think so. As the instrumentation libraries are expected to move to stable in couple months from what i heard (though there are lot of pending issues)

The spec which talks about schema_url usages (transformation etc.)m fixed telemetry schema etc. are still experimental, with no sign of moving to stable. Even if they are needed, it should be possible to not change any schema OR wait until schema_url is available before making any changes. The schema_url could come from .NET itself OR we might leverage Meter attributes.

[pellared]

OpenTelemetry Semantic Conventions for HTTP Spans are stable.

[cijothomas]

OpenTelemetry Semantic Conventions for HTTP Spans are stable.

But that is not relevant to this issue right?

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/telemetry-stability.md is still experimental, though that does not have to prevent us from adding schema_url support.

[lmolkova]

@tarekgh is it in scope for .NET 9? thanks!

[tarekgh]

@lmolkova no this is not in 9.0.

The fixed schema is the only stable part of the specs which doesn't require URL.

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/telemetry-stability.md#fixed-schema-telemetry-producers.

CC @reyang

[lmolkova]

So none of the .NET instrumentation can change after they are declared stable until at least .NET 10? Sounds like we're going to break this rule a lot.

[lmolkova]

Adding some context on how SchemaURL is useful:

SchemaURL looks like this: https://opentelemetry.io/schemas/1.29.0 and has two major parts:

• opentelemetry.io represents the authority - who owns the schema. I can easily envision https://aka.ms/otel/azure/sdk/schemas/1.42.0 and would publish one for Azure SDK in the future.
• 1.29.0 represents SemVer version. We should be able to evolve it and add stability suffix

The schema URL is useful even without the content behind it - it can be treated as an opaque string, or string with specific format containing authority and version.
The content provided under this URL is evolving, it has a huge potential, but de-facto is not used today.

---

1. As a user I want to know what schema certain library/app/component emits. It maybe OTel semconv,
   custom schema created by my company, etc. It helps me with:

   • knowing the source of telemetry schema (otel, my company, someone else)
   • knowing the version when debugging telemetry-related issues.
   • knowing when things are updated and get notified (e.g. via an alert) so I can adjust my dashboards and queries

2. As a client library or a service that emits telemetry, I need a way to tell my users which telemetry schema my component follows.
   I can document it, but auto-updates happen and it could be hard to determine what version is used in runtime.
   If I publish my own conventions, I'm making it explicit to my users.

   I can additionally communicate stability level of the conventions by including GA/preview in the SemVer version (https://opentelemetry.io/schemas/1.29.0-dev).
   I can document that users can rely on this field when writing queries/alerts if it indicates stable version and following SemVer compatibility.

3. As a telemetry backend (processor, exporter) I may need to map otel attributes from otel schema to my own. To perform such mapping, in a general case, I need to understand if the data follows the schema I support. If it does, I can write mappings for
   specific version and/or optimize them based on it.

4. As a future-proof and very advanced telemetry backend, I want to use the schema definition provided on this URL to upgrade/downgrade that telemetry to the version I work best with. This is still in flight.

---

What happens without SchemaURL:

• As a user, I need to look into each of my libs docs (for current version) to learn which conventions they emit.

• If telemetry changes (e.g. new attribute pops up) it's hard for me to understand if it has changed because my app started doing something new or lib changed and now follows updated telemetry schema

• As a backend, if I want to support multiple unstable versions, I have to support all combinations of attributes/metrics in each versions. I have no means to fail fast and drop/optimize things I don't support yet or anymore.

  Imagine you're writing a backend and receive a request. You probably can serve it, but you need to explore the body to guess which API version client was following.
  It all works reasonably well until you need to upgrade, route based on version, apply different logic, etc.

We have a lot of breaking changes in experimental semconv - there is a big need for schema transformation that depends on schema url - people are building it.

It's expected that we'll evolve areas of OTel Semconv like GenAI after they stabilize and have v2+ in years to come. Upgrade without versioning is hard or impossible. Versioning helps backends to handle it almost gracefully

---

Schema URL is

• stable in OTel spec
• it's specified when tracer/meter/logger is created
• passed over otlp in instrumentation scope
• supported by Go, Java, JS, Python, C++ OTel SDKs
• we set it in Azure SDKs in other languages and have to use custom attribute in .NET. This is part of our contract with AzMon on versioning.