Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions docs/trace/customizing-the-sdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,72 @@ probability configured via the `OTEL_TRACES_SAMPLER_ARG` environment variable.
Follow [this](../extending-the-sdk/README.md#sampler) document
to learn about writing custom samplers.

#### Troubleshooting: spans dropped due to an unsampled parent

The default sampler is `ParentBased(root=AlwaysOn)`. As the name implies, when a
span has a parent, the `ParentBased` sampler honors the parent's sampling
decision: if the parent was **not** sampled (recorded), the child is dropped as
well. This is the behavior required by the
[specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#parentbased)
and is usually what you want, but it can be surprising when the parent activity
is one you did not create yourself.

One scenario where this can occur is an ASP.NET Core application that emits custom
spans **without** adding the ASP.NET Core instrumentation to your application:

```csharp
var activitySource = new ActivitySource("MyCompany.MyProduct");

builder.Services.AddOpenTelemetry()
.WithTracing(tracing => tracing
.AddSource("MyCompany.MyProduct")
.AddConsoleExporter());

app.MapGet("/hello", () =>
{
using var activity = activitySource.StartActivity("Hello");
return "Hello, World!";
});
```

ASP.NET Core creates a `Microsoft.AspNetCore.Hosting.HttpRequestIn` activity for
every request (it is used for request logging and context propagation, even when
no tracing is configured). Because the `Microsoft.AspNetCore.Hosting`
`ActivitySource` has **not** been added to the `TracerProvider`, the SDK never
samples that activity, so it is created with its `Recorded` flag set to `false`
and becomes the parent (`Activity.Current`) of the custom `Hello` activity. The
default `ParentBased` sampler then drops the custom activity because its parent
was not recorded.

You can confirm this is what is happening by enabling
[self-diagnostics](../../../src/OpenTelemetry/README.md#self-diagnostics) at the
`Verbose` level. The SDK emits an event explaining that the activity was dropped
because its local parent is not recorded, together with the name of the
`ActivitySource`.

To resolve this, pick whichever option best suites your needs:
Comment thread
martincostello marked this conversation as resolved.
Outdated

* **Record the parent activity (recommended for ASP.NET Core apps).** Add the
[ASP.NET Core
instrumentation package](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.AspNetCore)
(`AddAspNetCoreInstrumentation()`), or at minimum register the source with
`AddSource("Microsoft.AspNetCore.Hosting")`. The request activity then becomes

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it by just?

Suggested change
`AddSource("Microsoft.AspNetCore.Hosting")`. The request activity then becomes
`AddSource("Microsoft.AspNetCore")`. The request activity then becomes

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm checking in the ASP.NET Core source, but it's definitely Microsoft.AspNetCore.Hosting for metrics: source

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that for tracing it's Microsoft.AspNetCore: source

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course I should have just looked in the other repo first... code 😅

Comment thread
martincostello marked this conversation as resolved.
Outdated
a recorded root and the custom child is sampled normally.

* **Use a sampler that does not depend on the parent.** For example
`SetSampler(new AlwaysOnSampler())` records every activity from the registered
sources regardless of the parent's sampling decision. Note this also disables
the honoring of upstream (remote) sampling decisions, so it is generally only
appropriate for development or simple scenarios.

* **Make the custom activity a root span** by starting it with an explicit
default (empty) parent context so the parent-based decision does not apply:

```csharp
using var activity = activitySource.StartActivity(
"Hello", ActivityKind.Internal, parentContext: default);
```

## Context Propagation

The OpenTelemetry API exposes a method to obtain the default propagator which is
Expand Down
5 changes: 5 additions & 0 deletions src/OpenTelemetry/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ Notes](../../RELEASENOTES.md).
* Fixed a metric point reclaim data race on CPU ARM architectures.
([#7401](https://github.com/open-telemetry/opentelemetry-dotnet/pull/7401))

* Added a verbose `OpenTelemetry-Sdk` self-diagnostics event that is emitted
when an activity is dropped because its local (in-process) parent is not
recorded.
([#TODO](https://github.com/open-telemetry/opentelemetry-dotnet/issues/TODO))
Comment thread
martincostello marked this conversation as resolved.
Outdated

## 1.16.0

Released 2026-Jun-10
Expand Down
19 changes: 19 additions & 0 deletions src/OpenTelemetry/Internal/OpenTelemetrySdkEventSource.cs
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,21 @@ public void ExemplarReservoirException(Exception ex)
}
}

[NonEvent]
public void ActivityDroppedDueToUnsampledLocalParent(string activityName, string activitySourceName, in ActivityContext parentContext)
{
// Only emit for an in-process (local) parent which was not recorded.
// A remote parent's sampling decision is controlled by the caller and is
// expected to flow through, so it is not reported here.
if (parentContext.TraceId != default &&
!parentContext.IsRemote &&
(parentContext.TraceFlags & ActivityTraceFlags.Recorded) == 0 &&
this.IsEnabled(EventLevel.Verbose, EventKeywords.All))
{
this.ActivityDroppedDueToUnsampledLocalParent(activityName, activitySourceName);
}
}

[Event(4, Message = "Unknown error in SpanProcessor event '{0}': '{1}'.", Level = EventLevel.Error)]
public void SpanProcessorException(string evnt, string ex)
=> this.WriteEvent(4, evnt, ex);
Expand Down Expand Up @@ -325,6 +340,10 @@ public void MetricViewException(string source, string ex)
public void ExemplarReservoirException(string ex)
=> this.WriteEvent(57, ex);

[Event(58, Message = "Activity '{0}' from source '{1}' was dropped because its local parent activity is not recorded (sampled). To record these activities, register the parent's instrumentation/ActivitySource or configure a non-parent-based sampler such as the AlwaysOnSampler.", Level = EventLevel.Verbose)]
public void ActivityDroppedDueToUnsampledLocalParent(string activityName, string activitySourceName)
=> this.WriteEvent(58, activityName, activitySourceName);

void IConfigurationExtensionsLogger.LogInvalidConfigurationValue(string key, string value)
=> this.InvalidConfigurationValue(key, value);

Expand Down
6 changes: 6 additions & 0 deletions src/OpenTelemetry/Trace/TracerProviderSdk.cs
Original file line number Diff line number Diff line change
Expand Up @@ -481,6 +481,11 @@ private static ActivitySamplingResult ComputeActivitySamplingResult(
SamplingDecision.Drop or _ => PropagateOrIgnoreData(ref options),
};

if (samplingResult.Decision == SamplingDecision.Drop)
{
OpenTelemetrySdkEventSource.Log.ActivityDroppedDueToUnsampledLocalParent(options.Name, options.Source.Name, options.Parent);
}
Comment thread
martincostello marked this conversation as resolved.
Outdated

if (activitySamplingResult > ActivitySamplingResult.PropagationData)
{
if (samplingResult.AttributesOrNull is { } attributes)
Expand Down Expand Up @@ -567,6 +572,7 @@ private void RunGetRequestedDataOtherSampler(Activity activity)
case SamplingDecision.Drop:
activity.IsAllDataRequested = false;
activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
OpenTelemetrySdkEventSource.Log.ActivityDroppedDueToUnsampledLocalParent(activity.DisplayName, activity.Source.Name, parentContext);
break;
Comment thread
martincostello marked this conversation as resolved.
case SamplingDecision.RecordOnly:
activity.IsAllDataRequested = true;
Expand Down
72 changes: 72 additions & 0 deletions test/OpenTelemetry.Tests/Trace/TracerProviderSdkTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using System.Diagnostics;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using OpenTelemetry.Internal;
using OpenTelemetry.Resources;
using OpenTelemetry.Resources.Tests;
using OpenTelemetry.Tests;
Expand Down Expand Up @@ -1349,6 +1350,77 @@ public void CheckActivityLinksAddedAfterActivityCreation()
Assert.Equal(link2.Context, exportedActivity.Links.ElementAt(1).Context);
}

[Fact]
public void SdkEmitsDiagnosticEventWhenActivityDroppedDueToUnsampledLocalParent()
{
using var source = new ActivitySource(Utils.GetCurrentMethodName());

// Default sampler is ParentBased(AlwaysOn).
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
.AddSource(source.Name)
.Build();

// Simulate the framework-created in-process parent activity. Its source is
// not registered with the provider, so it is never sampled (Recorded == false)
// but it still becomes Activity.Current.
using var parent = new Activity("ParentNotRecorded");
parent.Start();

Assert.False(parent.Recorded);
Assert.False(parent.Context.IsRemote);

// Create the event listener after the parent has started so its thread
// activity id is the one stamped on the diagnostic event.
using var eventListener = new TestEventListener(OpenTelemetrySdkEventSource.Log);

using (var child = source.StartActivity("Child"))
{
// The child is dropped because its local parent is not recorded.
Assert.Null(child);
}

parent.Stop();

var droppedEvents = eventListener.Messages.Where(
e => e.EventId == 58 &&
(e.Payload?.Count ?? 0) >= 2 &&
(e.Payload![0] as string) == "Child" &&
(e.Payload![1] as string) == source.Name);

Assert.Single(droppedEvents);
}

[Fact]
public void SdkDoesNotEmitUnsampledLocalParentEventForRemoteParent()
{
// A remote parent's sampling decision is controlled by the caller and is
// expected to flow through, so the diagnostic event should not be emitted.
using var source = new ActivitySource(Utils.GetCurrentMethodName());

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
.AddSource(source.Name)
.Build();

using var eventListener = new TestEventListener(OpenTelemetrySdkEventSource.Log);

var remoteParent = new ActivityContext(
ActivityTraceId.CreateRandom(),
ActivitySpanId.CreateRandom(),
ActivityTraceFlags.None,
isRemote: true);

using (var child = source.StartActivity("Child", ActivityKind.Internal, remoteParent))
{
// Dropped as PropagationData by the default ParentBased sampler.
Assert.False(child?.Recorded ?? false);
}

var droppedEvents = eventListener.Messages.Where(
e => e.EventId == 58 && (e.Payload?.Count ?? 0) >= 2 && (e.Payload![0] as string) == "Child");

Assert.Empty(droppedEvents);
}

public void Dispose()
=> GC.SuppressFinalize(this);

Expand Down