Baggage not included w/ OpenTelemetry

[bmcclory]

Describe the bug

OpenTelemetry .NET recommends:

The recommended way to add Baggage is to use the Baggage.SetBaggage() API. OpenTelemetry users should not use the Activity.AddBaggage method.

I'm new to OpenTelemetry and don't grok why they needed this specialized Baggage API and couldn't use the Activity API. But this Serilog enricher does not find baggage that is set in the OpenTelemetry API.

Not sure if this a use case this Serilog enricher intends to support. Or maybe OpenTelemetry should be copying its Baggage data onto the Activity context (where it can be found by this library), and I should raise the issue over there? It's a little confusing!

Steps to reproduce

Log.Logger = new LoggerConfiguration()
  .Enrich.WithSpan(new SpanOptions { IncludeBaggage = true });
  .WriteTo.Console(outputTemplate: "{Properties}");
  .CreateLogger();
  
Activity.SetBaggage("ActivityBaggage", "Test");
Baggage.SetBaggage("OtelBaggage", "Test");

// Only includes 'ActivityBaggage' in the log output
Log.Information("Hello!");

Expected behaviour

Serilog Span enricher with IncludeBaggage option will include baggage set by the OpenTelemetry Baggage API.

[RehanSaeed]

My understanding was that the OTEL API was just a wrapper around Activity but with the correct names. Is that incorrect?

[bmcclory]

Looks like that's not not the case. There's not much documentation around it but:

open-telemetry/opentelemetry-dotnet#1842

and

https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api/README.md#baggage-api

The recommended way to add Baggage is to use the Baggage.SetBaggage() API. OpenTelemetry users should not use the Activity.AddBaggage method.

I think the idea here is that in OTEL API Baggage isn't tied to an Activity (Span) context; it's a separate/standalone thing with its own propagator and whatnot.

It's all pretty confusing, because OTEL Baggage is propagated using standard W3C headers (as long you're using the OTEL propagators). And a receiving app using ASP.NET Core's built in diagnostics (without OTEL) will see those headers and set them on Activity.Baggage, where this Serilog Enricher will find them.

But the limitation there is that OTEL Baggage isn't copied into Activity.Baggage until that propagation has occurred. So for an app that does Baggage.SetBaggage(), this Serilog Enricher won't see that data and include it in the logs.

Maybe this is an issue for OTEL .NET to address -- maybe Baggage.SetBaggage() should be copying values into Activity.Baggage() But it's really hard to know what's most correct here...

[RehanSaeed]

The key lines in the second link:

It is important to note that Baggage is not automatically attached to any telemetry. User can explicitly read Baggage and use it to enrich metrics, logs and traces. An example of doing this for traces is shown here.

It sounds like you have to copy baggage manually?

[bmcclory]

I think the idea there is that OTEL telemetry data (Traces, Metrics, Logs) is not automatically enriched with Baggage. If you want it, you have to be explicit.

This also underscores how, in the OTEL specification, Baggage is a standalone thing and not tied to Traces/Activities, which is probably why OTEL .NET is trying to move away from Activity.Baggage.

Since this project is a Serilog Enricher, that's why I raised the issue here. But I realize Serilog != OTEL Logging, and I don't know what it'd look like to bring those things together. For now, I can certainly add a tiny OtelBaggageEnricher to Serilog to solve this problem.

[RehanSaeed]

This is certainly confusing. I'm not sure this project should do anything about this though. We'd also need to add a reference to the OTEL package if we decide to. Should we close for now?

[bmcclory]

Confusing indeed. Especially when OTEL acts as if Baggage is a separate thing from Trace, but recommends using "W3C format for distributed trace" to propagate it. But these things are in various levels of draft/experimental/beta, so probably premature to decide how Serilog (and this Enricher) should participate in that world, and maybe another package someday to avoid the extra dependency.

I'll go ahead and close. Hope this issue / bread crumb helps someone else!