Baggage value certain characters will crash application or encode chars

[Mario-Hofstaetter]

Bug Report

      <PackageReference Include="OpenTelemetry.Api" Version="1.0.0-rc1.1" />
      <PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="1.0.0-rc1.1" />
      <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc1.1" />
      <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc1.1" />
      <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc1.1" />

• Runtime version: netcoreapp3.1

Symptom

When adding Baggage to an Activity using certain chars will encode them or crash the receiving application.
The baggage value is not correctly received in child-service.

Char	Result
\	%2B
@	%2540
,	%252C
&	%2526
(	Will crash the ASP.NET Core Service
(only some samples)

There is no mentioning of a limited allowed charset in specification/overview.md#baggage ?

What is the expected behavior?

Original baggage value should be preserved exactly.

What is the actual behavior?

Some characters are only encoded.

Using round brackets will crash the ASP.NET Core WEB API Service.

Using .AddBaggage("demo.foobar5", "!x_x,x-x&x(x");:

[2020-11-22 15:23:48Z ERR] Connection id "0HM4EVAE30FTQ", Request id "0HM4EVAE30FTQ:00000001": 
An unhandled exception was thrown by the application. System.FormatException: The format of
value '!x_x%2Cx-x%26x(x' is invalid.
   at System.Net.Http.Headers.NameValueHeaderValue.CheckValueFormat(String value)
   at System.Net.Http.DiagnosticsHandler.InjectHeaders(Activity currentActivity, HttpRequestMessage request)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at FrontendService.Controllers.WeatherFrontendController.GetWeatherForecast() in D:xxxxxxxxxxxxx:line 109

Reproduce

Activity.Current?
	.AddBaggage("demo.foobar1", "demouser")
	.AddBaggage("demo.foobar2", "demo user")
	.AddBaggage("demo.foobar3", @"domain\demouser")
	.AddBaggage("demo.foobar4", "[email protected]")
        .AddBaggage("demo.foobar5", "!x_x,x-x&x(x"); // Will cause exception in receiving service

In child-service:

if (Activity.Current?.Baggage != null)
{
	foreach (var bag in Activity.Current.Baggage)
	{
		_logger.LogInformation("Found Baggage {BaggageKey}={BaggageValue}", bag.Key, bag.Value);

		// Use baggage data in tag which will be stored in jaeger
		Activity.Current.SetTag($"ax.debug.{bag.Key}", bag.Value);
	}
}

Log output of receiving service

Found Baggage demo.foobar4=user%2540example.com 
Found Baggage demo.foobar3=domain%255Cdemouser 
Found Baggage demo.foobar2=demo%2Buser 
Found Baggage demo.foobar1=demouser 

Resulting display in jaeger 1.21.0:

[Mario-Hofstaetter]

Any update on this? I have not yet re-tested it with the 1.0 release.

I have overheard this comment: #1842 (comment)
So maybe my reported issue above has nothing to do with opentelemetry after all?

[rgmills]

@Mario-Hofstaetter I ended up finding dotnet/runtime#36908 which is the root cause, so this is really unrelated to OTEL. Unfortunately, it doesn't seem like much one can do other than not using Activity.Baggage especially with the fun part of anything in Activity.Baggage propagating automatically to any external service you call.

[Mario-Hofstaetter]

@rgmills Thank you for the response. What about using the OpenTelemetry Baggage API? As mentioned in #1842 (comment)
That should be fine then?

I have to dig into the docs again, we have postponed implementation of Opentelemetry Tracing since I reported the issue ...

[rgmills]

That seems to be the correct route since the DiagnosticListener in the runtime is automatically going to add anything on Activity.Baggage. I haven't tried yet, yesterday we just switched out baggage to tag to get OTEL enabled again. Today, I'll get something into prod with the OTEL Baggage API.

[kipwoker]

Got similar behavior when tried to pass baggage between services.
Set baggage like this:

Baggage.Current.SetBaggage("baggage_key_test", "baggage value test 123");

Get baggage:

var value = Baggage.Current.GetBaggage("baggage_key_test");
// baggage%2Bvalue%2Btest%2B123

I see the encoding in inject method, but there is no decoding in the extract method.

So probably it could be easily fixed if we add decoding into the extract method.