Skip to content
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5b27a7a
Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss)
mrsharm Feb 23, 2026
e88afde
Sharpen eval.yaml: add custom-spans-invisible pain-point and name-mat…
mrsharm Feb 25, 2026
d75cb0f
Migrate configuring-opentelemetry-dotnet to plugins/ directory structure
mrsharm Mar 6, 2026
b666ed1
Move OpenTelemetry skill to new dotnet-aspnet plugin, address review …
Mar 28, 2026
c0ed61b
Configure metrics OTLP exporter endpoint explicitly to match tracing …
Mar 28, 2026
a08539a
Address round 2 review: logging endpoint, marketplace, RecordExceptio…
Mar 28, 2026
7f0300e
Clarify logging package source and add ActivitySource registration re…
Mar 28, 2026
b574a12
Add @dotnet/aspnet team to skill CODEOWNERS entries
Mar 28, 2026
f8c8ebb
Add dotnet-aspnet to README table, make propagation ActivitySource st…
Mar 28, 2026
72a9606
Add expect_activation:false, IMeterFactory using, clarify OTLP export…
Mar 28, 2026
60f068e
Add OtlpExporter using directive, clarify OTLP package covers logging
Mar 28, 2026
622885c
Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval
Mar 28, 2026
9022b35
Clarify Jaeger is traces-only, document logging package provenance
Mar 28, 2026
706515a
Fix eval: increase timeout, reject tools, rewrite prompt as explanati…
Mar 28, 2026
826e637
Improve OTel skill evals and address review feedback
mrsharm Mar 31, 2026
f8ed61e
Add propagate-trace-context scenario to OTel evals (+25.6%)
mrsharm Mar 31, 2026
2ed24b9
Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by…
mrsharm Mar 31, 2026
f6954f6
Use @dotnet/aspnet team alias in CODEOWNERS for wider reach
mrsharm Mar 31, 2026
63611fe
Merge branch 'main' into musharm/configuring-opentelemetry-dotnet-skill
ViktorHofer Mar 31, 2026
e58d054
Addressed feedback
mrsharm Apr 6, 2026
d191a20
Addressed feedback
mrsharm Apr 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@
/tests/dotnet-ai/technology-selection/ @luisquintanilla @artl93

# dotnet-aspnet (ASP.NET Core web development)
/plugins/dotnet-aspnet/ @BrennanConroy @adityamandaleeka @halter73
/tests/dotnet-aspnet/ @BrennanConroy @adityamandaleeka @halter73
/plugins/dotnet-aspnet/ @dotnet/aspnet
/tests/dotnet-aspnet/ @dotnet/aspnet

# dotnet-data (data access, Entity Framework)
/plugins/dotnet-data/skills/optimizing-ef-core-queries/ @dotnet/efteam
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,312 @@
---
name: configuring-opentelemetry-dotnet
description: Configure OpenTelemetry distributed tracing, metrics, and logging in ASP.NET Core using the .NET OpenTelemetry SDK. Use when adding observability, setting up OTLP exporters, creating custom metrics/spans, or troubleshooting distributed trace correlation.
---

# Configuring OpenTelemetry in .NET

## When to Use

- Adding distributed tracing to an ASP.NET Core application
- Setting up OpenTelemetry exporters (OTLP is the primary protocol shown; Jaeger and Prometheus accept OTLP natively)
Comment thread
mrsharm marked this conversation as resolved.
Outdated
- Creating custom metrics or trace spans for business operations
Comment thread
danmoseley marked this conversation as resolved.
- Troubleshooting distributed trace context propagation across services

## When Not to Use

- The user wants application-level logging only (use ILogger, Serilog)
- The user is using Application Insights SDK directly (different API)
- The user needs APM with a commercial vendor's proprietary SDK

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| ASP.NET Core project | Yes | The application to instrument |
| Observability backend | No | Where to export: OTLP collector, Aspire dashboard, Jaeger (accepts OTLP natively) |

Comment thread
danmoseley marked this conversation as resolved.
Comment thread
danmoseley marked this conversation as resolved.
## Workflow

### Step 1: Install the correct packages

**There are many OpenTelemetry NuGet packages. Install exactly these:**

```bash
# Core SDK + ASP.NET Core instrumentation + logging integration
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

Comment thread
danmoseley marked this conversation as resolved.
# Exporter (pick one or more — also used by logging in Step 3)
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol # OTLP exporter for traces, metrics, AND logs
dotnet add package OpenTelemetry.Exporter.Console # Dev/debugging
Comment thread
mrsharm marked this conversation as resolved.
Outdated
```
Comment thread
danmoseley marked this conversation as resolved.
Comment thread
danmoseley marked this conversation as resolved.

**Do NOT install `OpenTelemetry` alone** — you need `OpenTelemetry.Extensions.Hosting` for proper DI integration.

#### Optional: additional auto-instrumentation packages

Install only the packages that match the libraries your application uses:

```bash
dotnet add package OpenTelemetry.Instrumentation.SqlClient # SQL Server queries
dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore # EF Core
dotnet add package OpenTelemetry.Instrumentation.GrpcNetClient # gRPC calls
dotnet add package OpenTelemetry.Instrumentation.Runtime # GC, thread pool metrics
```

### Step 2: Configure tracing and metrics in Program.cs

```csharp
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Logs;
using OpenTelemetry.Exporter; // for OtlpExportProtocol

var builder = WebApplication.CreateBuilder(args);

// Define the service resource (appears in all telemetry)
var serviceName = "MyOrderService";
var serviceVersion = "1.0.0";

builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource
.AddService(serviceName: serviceName, serviceVersion: serviceVersion))
.WithTracing(tracing => tracing
// Auto-instrumentation sources
.AddAspNetCoreInstrumentation(options =>
{
// Filter out health check endpoints from traces
options.Filter = httpContext =>
!httpContext.Request.Path.StartsWithSegments("/healthz");
})
.AddHttpClientInstrumentation(options =>
{
// Enrich outgoing HTTP spans with request/response details
options.RecordException = true;
})
// Optional: add SQL instrumentation if using SqlClient directly
// .AddSqlClientInstrumentation(options =>
// {
// options.SetDbStatementForText = true;
// options.RecordException = true;
// })
// Custom activity sources (for your own spans)
.AddSource("MyOrderService.Orders")
.AddSource("MyOrderService.Payments")
// Exporter
.AddOtlpExporter(options =>
Comment thread
mrsharm marked this conversation as resolved.
Outdated
{
options.Endpoint = new Uri("http://localhost:4317"); // gRPC endpoint
Comment thread
mrsharm marked this conversation as resolved.
Outdated
// For HTTP: options.Protocol = OtlpExportProtocol.HttpProtobuf;
// options.Endpoint = new Uri("http://localhost:4318/v1/traces");
}))
Comment thread
danmoseley marked this conversation as resolved.
Outdated
.WithMetrics(metrics => metrics
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
// Optional: .AddRuntimeInstrumentation() for GC and thread pool metrics
// (requires OpenTelemetry.Instrumentation.Runtime package)
// Custom meters
.AddMeter("MyOrderService.Metrics")
// Note: Jaeger is traces-only. For metrics, export to an OTel Collector or
// Prometheus-compatible backend. Here we use the same OTLP endpoint assuming
// an OTel Collector that routes traces and metrics to appropriate backends.
.AddOtlpExporter(options =>
{
options.Endpoint = new Uri("http://localhost:4317"); // gRPC endpoint (metrics)
}));
```

### Step 3: Add OpenTelemetry logging integration

No additional packages are needed — `AddOpenTelemetry()` comes from the `OpenTelemetry` package (a transitive dependency of `OpenTelemetry.Extensions.Hosting`), and `AddOtlpExporter()` comes from `OpenTelemetry.Exporter.OpenTelemetryProtocol`, both already installed in Step 1.

```csharp
// Connect ILogger to OpenTelemetry
builder.Logging.AddOpenTelemetry(logging =>
{
logging.IncludeScopes = true;
logging.IncludeFormattedMessage = true;
logging.ParseStateValues = true; // Preserve structured log attributes
Comment thread
mrsharm marked this conversation as resolved.
Outdated

// CRITICAL: Set the same service resource on the logging provider.
// builder.Services.AddOpenTelemetry().ConfigureResource() does NOT
// propagate to the logging provider — set it explicitly here.
Comment thread
mrsharm marked this conversation as resolved.
Outdated
logging.SetResourceBuilder(ResourceBuilder.CreateDefault()
.AddService(serviceName: serviceName, serviceVersion: serviceVersion));

logging.AddOtlpExporter(options =>
{
options.Endpoint = new Uri("http://localhost:4317");
});
});
```

**This correlates logs with traces automatically** — each log entry gets the current TraceId and SpanId.

### Step 4: Create custom spans (Activities) for business operations

```csharp
using System.Diagnostics;
using OpenTelemetry.Trace;

public class OrderService
{
// Create an ActivitySource matching what you registered in Step 2
private static readonly ActivitySource ActivitySource = new("MyOrderService.Orders");

public async Task<Order> ProcessOrderAsync(CreateOrderRequest request)
{
// Start a new span
using var activity = ActivitySource.StartActivity("ProcessOrder");

// Add attributes (tags) to the span
activity?.SetTag("order.customer_id", request.CustomerId);
activity?.SetTag("order.item_count", request.Items.Count);

try
{
// Child span for validation
using (var validationActivity = ActivitySource.StartActivity("ValidateOrder"))
{
await ValidateOrderAsync(request);
validationActivity?.SetTag("validation.result", "passed");
}

// Child span for payment
using (var paymentActivity = ActivitySource.StartActivity("ProcessPayment",
ActivityKind.Client)) // Client = outgoing call
{
paymentActivity?.SetTag("payment.method", request.PaymentMethod);
await ProcessPaymentAsync(request);
}

var order = new Order { Id = Guid.NewGuid(), CustomerId = request.CustomerId, Status = "Completed" };

activity?.SetTag("order.status", "completed");
activity?.SetStatus(ActivityStatusCode.Ok);

return order;
}
Comment thread
danmoseley marked this conversation as resolved.
catch (Exception ex)
{
// Record the exception on the span
activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
activity?.RecordException(ex);
Comment thread
danmoseley marked this conversation as resolved.
Outdated
Comment thread
mrsharm marked this conversation as resolved.
Outdated
throw;
}
Comment thread
danmoseley marked this conversation as resolved.
}
}
```

**Critical: `ActivitySource` name must match `AddSource("...")` in configuration.** Unmatched sources are silently ignored — this is the #1 debugging issue.

### Step 5: Create custom metrics

Use `IMeterFactory` (injected via DI) to create meters — this ensures proper lifetime management and testability.

```csharp
using System.Diagnostics;
using System.Diagnostics.Metrics;
using Microsoft.Extensions.Diagnostics.Metrics;
Comment thread
mrsharm marked this conversation as resolved.
Outdated

public class OrderMetrics
{
// Meter name must match AddMeter("...") in configuration
Comment thread
mrsharm marked this conversation as resolved.
Outdated
private readonly Counter<long> _ordersProcessed;
private readonly Histogram<double> _orderProcessingDuration;
private readonly UpDownCounter<int> _activeOrders;

public OrderMetrics(IMeterFactory meterFactory)
{
var meter = meterFactory.Create("MyOrderService.Metrics");

// Counter — use for things that only go up
_ordersProcessed = meter.CreateCounter<long>(
"orders.processed", "orders", "Total orders successfully processed");

// Histogram — use for measuring distributions (latency, sizes)
_orderProcessingDuration = meter.CreateHistogram<double>(
"orders.processing_duration", "ms", "Time to process an order");

// UpDownCounter — use for things that go up AND down
_activeOrders = meter.CreateUpDownCounter<int>(
"orders.active", "orders", "Currently processing orders");
}

public void RecordOrderProcessed(string region, double durationMs)
{
// Tags enable dimensional filtering (by region, status, etc.)
var tags = new TagList
{
{ "region", region },
{ "order.type", "standard" }
};
Comment thread
danmoseley marked this conversation as resolved.

_ordersProcessed.Add(1, tags);
_orderProcessingDuration.Record(durationMs, tags);
}
}
```

Register `OrderMetrics` in DI:

```csharp
builder.Services.AddSingleton<OrderMetrics>();
```

### Step 6: Configure context propagation for distributed scenarios

Trace context propagation is automatic for HTTP calls when using `AddHttpClientInstrumentation()`. For non-HTTP scenarios:

```csharp
using System;
using System.Collections.Generic;
using System.Diagnostics;
using OpenTelemetry.Context.Propagation;

// ActivitySource should be static — register via .AddSource("MyOrderService.Messaging") in Step 2
private static readonly ActivitySource MessageSource = new("MyOrderService.Messaging");

// Manual context propagation (e.g., across message queues)
// On the SENDING side:
var propagator = Propagators.DefaultTextMapPropagator;
var activityContext = Activity.Current?.Context ?? default;
var context = new PropagationContext(activityContext, Baggage.Current);
var carrier = new Dictionary<string, string>();

propagator.Inject(context, carrier, (dict, key, value) => dict[key] = value);
// Send carrier dictionary as message headers

// On the RECEIVING side:
var parentContext = propagator.Extract(default, carrier,
(dict, key) => dict.TryGetValue(key, out var value) ? new[] { value } : Array.Empty<string>());

Baggage.Current = parentContext.Baggage;
using var activity = MessageSource.StartActivity("ProcessMessage",
ActivityKind.Consumer,
parentContext.ActivityContext); // Links to parent trace!
```

## Validation

- [ ] Traces appear in the observability backend (Jaeger, Aspire dashboard, etc.)
- [ ] HTTP requests automatically create spans with correct verb, URL, status code
- [ ] Custom `ActivitySource` names match `AddSource()` registrations
- [ ] Custom `Meter` names match `AddMeter()` registrations
- [ ] Logs include TraceId and SpanId for correlation
- [ ] Health check endpoints are filtered from traces
- [ ] Exception details appear on error spans

## Common Pitfalls

| Pitfall | Solution |
|---------|----------|
| `ActivitySource.StartActivity` returns null | Source name doesn't match any `AddSource()` — names must match exactly |
| Traces not appearing in exporter | Check OTLP endpoint: gRPC uses port 4317, HTTP uses 4318 |
Comment thread
danmoseley marked this conversation as resolved.
| Missing HTTP client spans | Ensure `AddHttpClientInstrumentation()` is registered; it works for both `IHttpClientFactory`/DI and `new HttpClient()` (use `IHttpClientFactory` for lifetime management) |
| High cardinality tags | Don't use user IDs, request IDs, or UUIDs as metric tags — explodes storage |
| OTLP gRPC vs HTTP mismatch | Default is gRPC (port 4317); if collector only accepts HTTP, set `OtlpExportProtocol.HttpProtobuf` |
| Static `Meter` instead of `IMeterFactory` | Prefer `IMeterFactory` from DI for proper lifetime management and testability |
| `Meter` and `ActivitySource` not static | `ActivitySource` should be static; `Meter` should be created via `IMeterFactory` in DI |
Comment thread
mrsharm marked this conversation as resolved.
Outdated
62 changes: 62 additions & 0 deletions tests/dotnet-aspnet/configuring-opentelemetry-dotnet/eval.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
scenarios:
- name: "Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core"
prompt: |
I'm adding OpenTelemetry to my ASP.NET Core 8 API. I need:
1. Distributed tracing with OTLP export to a collector at localhost:4317
2. A custom span around my OrderService.ProcessOrder method
3. A custom counter metric for orders processed
Show me the complete Program.cs setup and the service class. Don't create files.
assertions:
- type: "output_matches"
pattern: "(OpenTelemetry\\.Extensions\\.Hosting|AddOpenTelemetry)"
- type: "output_matches"
pattern: "(ActivitySource|StartActivity)"
- type: "output_matches"
pattern: "(AddOtlpExporter|OtlpExporter)"
rubric:
- "Included OpenTelemetry.Extensions.Hosting and OpenTelemetry.Instrumentation.Http in the required NuGet packages"
- "Configured both .WithTracing() and .WithMetrics() in the same AddOpenTelemetry() call"
- "Registered custom ActivitySource names via AddSource() and showed that the ActivitySource name in the service class must match exactly"
- "Used IMeterFactory via dependency injection to create the Meter instead of a static Meter constructor"
reject_tools: ["bash", "edit"]
timeout: 180

- name: "Configure all three OpenTelemetry signals with correct OTLP endpoints"
prompt: |
I need to set up all three OpenTelemetry signals (traces, metrics, logs) in my
ASP.NET Core 8 app, all exporting via OTLP to my collector. I want logs to
include trace correlation and carry the same service name as my traces.
Comment thread
mrsharm marked this conversation as resolved.
Outdated
What NuGet packages do I need, and show me the complete Program.cs configuration?
Don't create files.
assertions:
- type: "output_matches"
pattern: "(AddOpenTelemetry|WithTracing|WithMetrics)"
- type: "output_matches"
pattern: "(AddOtlpExporter|OtlpExporter)"
- type: "output_matches"
pattern: "(Logging|AddOpenTelemetry|TraceId|SpanId)"
rubric:
Comment thread
mrsharm marked this conversation as resolved.
- "Configured all three signals: tracing (WithTracing), metrics (WithMetrics), and logging (builder.Logging.AddOpenTelemetry)"
- "Set explicit OTLP exporter endpoints consistently across all three signals instead of leaving some as defaults"
Comment thread
mrsharm marked this conversation as resolved.
Outdated
- "Set the service resource on the logging provider explicitly so logs carry the same service name as traces and metrics"
Comment thread
mrsharm marked this conversation as resolved.
Outdated
- "Listed the complete set of required NuGet packages including the OTLP exporter package"
Comment thread
mrsharm marked this conversation as resolved.
reject_tools: ["bash", "edit"]
timeout: 180

- name: "Propagate trace context across a message queue"
prompt: |
I have two ASP.NET Core 8 services communicating via RabbitMQ. Service A publishes
an order event and Service B consumes it. Traces show up fine within each service
but I can't see the end-to-end distributed trace across the queue boundary.
How do I propagate the trace context through message headers? Don't create files.
assertions:
- type: "output_matches"
pattern: "(Propagat|TextMapPropagator|Inject|Extract)"
- type: "output_matches"
pattern: "(ActivitySource|StartActivity)"
rubric:
- "Showed how to inject trace context into message headers on the sending side using a TextMapPropagator"
- "Showed how to extract trace context from message headers on the receiving side and link it as a parent"
- "Used the extracted context as the parent when starting a new Activity on the consumer so spans connect into one distributed trace"
reject_tools: ["bash", "edit"]
timeout: 180
Loading