OTel impact Application Insight SDK sampling

[fkpwolf]

Bug Report

<PackageReference Update="Microsoft.ApplicationInsights.AspNetCore" Version="2.20.0" />
<PackageReference Update="Azure.Monitor.OpenTelemetry.Exporter" Version="1.0.0-beta.7" />
<PackageReference Update="OpenTelemetry" Version="1.4.0-rc.3" />
<PackageReference Update="OpenTelemetry.Exporter.Console" Version="1.4.0-rc.3" />
<PackageReference Update="OpenTelemetry.Exporter.Geneva" Version="1.4.0-rc.3" />
<PackageReference Update="OpenTelemetry.Extensions.Hosting" Version="1.4.0-rc.3" />
<PackageReference Update="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.12" />
<PackageReference Update="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.12" />

Runtime version: net6.0

Symptom

We use Application Insight SDK and OTel SDK side by side and enabled sampling on AI SDK. It is observed that sampling of Application Insight is impacted by OTel.

The reason we use both of SDK is that we still have some AI related code like samping rule and legacy metrics and have to keep them before migrate to OTel completely. We also checked code to make sure OTel will not export telemetry data to Application Insight.

What is the expected behavior?

Appliction Insight samping works as before.

What is the actual behavior?

The itemcount of Appliction Insight is much higher(x10). Itemcount indicates sampling ratio. If we remove OTel, itemcount goes to normal.

Reproduce

  // services
  services.AddApplicationInsightsTelemetry()
      .Configure<TelemetryConfiguration>(config =>
      {
          config.TelemetryInitializers.Add(new ExcludeTelemetryFromSamplingTelemetryInitializer());
      });

  // Add OTel Logging
  var instrumentationKey = this.Configuration["APPINSIGHTS_INSTRUMENTATIONKEY"];
  services.AddSingleton<ILoggerProvider, OpenTelemetryLoggerProvider>()
      .Configure<OpenTelemetryLoggerOptions>((openTelemetryLoggerOptions) =>
      {
          openTelemetryLoggerOptions.IncludeFormattedMessage = true;
          openTelemetryLoggerOptions.SetResourceBuilder(OpenTelemetryResourceBuilder);
          openTelemetryLoggerOptions.AddGenevaLogExporter(exporterOptions =>
          {
              if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
              {
                  exporterOptions.ConnectionString = "EtwSession=OpenTelemetry";
              }
              else
              {
                  exporterOptions.ConnectionString = "Endpoint=unix:/var/run/mdsd/default_fluent.socket";
              }
          });
      });

  // Add OTel tracing and metrics
  services.AddOpenTelemetry().WithTracing(builder =>
  {
      builder
      .SetResourceBuilder(OpenTelemetryResourceBuilder)
      .AddAspNetCoreInstrumentation(o => o.RecordException = true) // Auto-instrument http incoming requests
      .AddHttpClientInstrumentation() // Auto-instrument http outgoing dependencies like calls triggered by HttpClient
      .AddGenevaTraceExporter(exporterOptions =>
      {
          if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
          {
              exporterOptions.ConnectionString = "EtwSession=OpenTelemetry";
          }
          else
          {
              exporterOptions.ConnectionString = "Endpoint=unix:/var/run/mdsd/default_fluent.socket";
          }
      });
  }).WithMetrics(builder =>
  {
      builder
      .SetResourceBuilder(OpenTelemetryResourceBuilder)
      .AddAspNetCoreInstrumentation() // Auto-instrument http incoming requests
      .AddHttpClientInstrumentation() // Auto-instrument http outgoing dependencies like calls triggered by HttpClient
      .AddMeter(Metrics.MetricsMeterId)
      .AddGenevaMetricExporter(options =>
      {
          options.ConnectionString = $"Account={this.Configuration["GenevaMetricAccountName"]};Namespace={this.Configuration["GenevaMetricNamespace"]}";
      });
  }).StartWithHost();

[vishweshbankwar]

@fkpwolf Take a look at microsoft/ApplicationInsights-dotnet#2742. Closing this as it is not a bug with OTel.