-
Notifications
You must be signed in to change notification settings - Fork 363
Add configuring-opentelemetry-dotnet skill #268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
mrsharm
merged 21 commits into
dotnet:main
from
mrsharm:musharm/configuring-opentelemetry-dotnet-skill
Apr 7, 2026
Merged
Changes from 20 commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
5b27a7a
Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss)
mrsharm e88afde
Sharpen eval.yaml: add custom-spans-invisible pain-point and name-mat…
mrsharm d75cb0f
Migrate configuring-opentelemetry-dotnet to plugins/ directory structure
mrsharm b666ed1
Move OpenTelemetry skill to new dotnet-aspnet plugin, address review …
c0ed61b
Configure metrics OTLP exporter endpoint explicitly to match tracing …
a08539a
Address round 2 review: logging endpoint, marketplace, RecordExceptio…
7f0300e
Clarify logging package source and add ActivitySource registration re…
b574a12
Add @dotnet/aspnet team to skill CODEOWNERS entries
f8c8ebb
Add dotnet-aspnet to README table, make propagation ActivitySource st…
72a9606
Add expect_activation:false, IMeterFactory using, clarify OTLP export…
60f068e
Add OtlpExporter using directive, clarify OTLP package covers logging
622885c
Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval
9022b35
Clarify Jaeger is traces-only, document logging package provenance
706515a
Fix eval: increase timeout, reject tools, rewrite prompt as explanati…
826e637
Improve OTel skill evals and address review feedback
mrsharm f8ed61e
Add propagate-trace-context scenario to OTel evals (+25.6%)
mrsharm 2ed24b9
Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by…
mrsharm f6954f6
Use @dotnet/aspnet team alias in CODEOWNERS for wider reach
mrsharm 63611fe
Merge branch 'main' into musharm/configuring-opentelemetry-dotnet-skill
ViktorHofer e58d054
Addressed feedback
mrsharm d191a20
Addressed feedback
mrsharm File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
285 changes: 285 additions & 0 deletions
285
plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,285 @@ | ||
| --- | ||
| name: configuring-opentelemetry-dotnet | ||
| description: Configure OpenTelemetry distributed tracing, metrics, and logging in ASP.NET Core using the .NET OpenTelemetry SDK. Use when adding observability, setting up OTLP exporters, creating custom metrics/spans, or troubleshooting distributed trace correlation. | ||
| --- | ||
|
|
||
| # Configuring OpenTelemetry in .NET | ||
|
|
||
| ## When to Use | ||
|
|
||
| - Adding distributed tracing to an ASP.NET Core application | ||
| - Setting up OpenTelemetry exporters (OTLP is the primary protocol; Jaeger accepts OTLP natively; Prometheus OTLP ingestion requires explicit opt-in) | ||
| - Creating custom metrics or trace spans for business operations | ||
| - Troubleshooting distributed trace context propagation across services | ||
|
|
||
| ## When Not to Use | ||
|
|
||
| - The user wants application-level logging only (use ILogger, Serilog) | ||
| - The user is using Application Insights SDK directly (different API) | ||
| - The user needs APM with a commercial vendor's proprietary SDK | ||
|
|
||
| ## Inputs | ||
|
|
||
| | Input | Required | Description | | ||
| |-------|----------|-------------| | ||
| | ASP.NET Core project | Yes | The application to instrument | | ||
| | Observability backend | No | Where to export: OTLP collector, Aspire dashboard, Jaeger (accepts OTLP natively) | | ||
|
|
||
|
danmoseley marked this conversation as resolved.
danmoseley marked this conversation as resolved.
|
||
| ## Workflow | ||
|
|
||
| ### Step 1: Install the correct packages | ||
|
|
||
| **There are many OpenTelemetry NuGet packages. Install exactly these:** | ||
|
|
||
| ```bash | ||
| # Core SDK + ASP.NET Core instrumentation + logging integration | ||
| dotnet add package OpenTelemetry.Extensions.Hosting | ||
| dotnet add package OpenTelemetry.Instrumentation.AspNetCore | ||
| dotnet add package OpenTelemetry.Instrumentation.Http | ||
|
|
||
|
danmoseley marked this conversation as resolved.
|
||
| # Exporter (pick one or more — also used by logging in Step 3) | ||
| dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol # OTLP exporter for traces, metrics, AND logs | ||
| dotnet add package OpenTelemetry.Exporter.Console # Dev/debugging | ||
|
mrsharm marked this conversation as resolved.
Outdated
|
||
| ``` | ||
|
danmoseley marked this conversation as resolved.
danmoseley marked this conversation as resolved.
|
||
|
|
||
| **Do NOT install `OpenTelemetry` alone** — you need `OpenTelemetry.Extensions.Hosting` for proper DI integration. | ||
|
|
||
| #### Optional: additional auto-instrumentation packages | ||
|
|
||
| Install only the packages that match the libraries your application uses: | ||
|
|
||
| ```bash | ||
| dotnet add package OpenTelemetry.Instrumentation.SqlClient # SQL Server queries | ||
| dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore # EF Core | ||
| dotnet add package OpenTelemetry.Instrumentation.GrpcNetClient # gRPC calls | ||
| dotnet add package OpenTelemetry.Instrumentation.Runtime # GC, thread pool metrics | ||
| ``` | ||
|
|
||
| ### Step 2: Configure all signals in Program.cs | ||
|
|
||
| ```csharp | ||
| using OpenTelemetry.Resources; | ||
| using OpenTelemetry.Trace; | ||
| using OpenTelemetry.Metrics; | ||
| using OpenTelemetry.Logs; | ||
|
|
||
| var builder = WebApplication.CreateBuilder(args); | ||
|
|
||
| builder.Services.AddOpenTelemetry() | ||
| .ConfigureResource(resource => resource | ||
| .AddService(serviceName: builder.Environment.ApplicationName)) | ||
| .WithTracing(tracing => tracing | ||
| .AddAspNetCoreInstrumentation(options => | ||
| { | ||
| // Filter out health check endpoints from traces | ||
| options.Filter = httpContext => | ||
| !httpContext.Request.Path.StartsWithSegments("/healthz"); | ||
| }) | ||
| .AddHttpClientInstrumentation(options => | ||
| { | ||
| options.RecordException = true; | ||
| }) | ||
| // Optional: add SQL instrumentation if using SqlClient directly | ||
| // .AddSqlClientInstrumentation(options => | ||
| // { | ||
| // options.SetDbStatementForText = true; | ||
| // options.RecordException = true; | ||
| // }) | ||
| // Custom activity sources (must match ActivitySource names in your code) | ||
| .AddSource("MyApp.Orders") | ||
| .AddSource("MyApp.Payments")) | ||
|
mrsharm marked this conversation as resolved.
Outdated
|
||
| .WithMetrics(metrics => metrics | ||
| .AddAspNetCoreInstrumentation() | ||
| .AddHttpClientInstrumentation() | ||
| // Optional: .AddRuntimeInstrumentation() for GC and thread pool metrics | ||
| // (requires OpenTelemetry.Instrumentation.Runtime package) | ||
| // Custom meters (must match Meter names in your code) | ||
| .AddMeter("MyApp.Metrics")) | ||
| .WithLogging(logging => | ||
| { | ||
| logging.IncludeScopes = true; | ||
| logging.IncludeFormattedMessage = true; | ||
| }) | ||
| // Single OTLP exporter for all signals — reads OTEL_EXPORTER_OTLP_ENDPOINT | ||
| // env var (defaults to http://localhost:4317). Override via environment variable | ||
| // or appsettings.json configuration. | ||
| .UseOtlpExporter(); | ||
| ``` | ||
|
|
||
| ### Step 3: Understanding log–trace correlation | ||
|
|
||
| The `.WithLogging()` call in Step 2 integrates ILogger with OpenTelemetry: | ||
|
|
||
| - Each log entry automatically includes TraceId and SpanId for correlation with traces | ||
| - The service resource from `.ConfigureResource()` propagates to logs automatically | ||
| - `UseOtlpExporter()` applies to logs alongside traces and metrics | ||
| - No additional packages or separate `SetResourceBuilder` call needed | ||
|
|
||
| ### Step 4: Create custom spans (Activities) for business operations | ||
|
|
||
| ```csharp | ||
| using System.Diagnostics; | ||
| using Microsoft.Extensions.Logging; | ||
|
|
||
| public class OrderService | ||
| { | ||
| // Create an ActivitySource matching what you registered in Step 2 | ||
| private static readonly ActivitySource ActivitySource = new("MyApp.Orders"); | ||
| private readonly ILogger<OrderService> _logger; | ||
|
|
||
| public OrderService(ILogger<OrderService> logger) => _logger = logger; | ||
|
|
||
| public async Task<Order> ProcessOrderAsync(CreateOrderRequest request) | ||
| { | ||
| // Start a new span | ||
| using var activity = ActivitySource.StartActivity("ProcessOrder"); | ||
|
|
||
| // Add attributes (tags) to the span | ||
| activity?.SetTag("order.customer_id", request.CustomerId); | ||
| activity?.SetTag("order.item_count", request.Items.Count); | ||
|
|
||
| try | ||
| { | ||
| // Child span for validation | ||
| using (var validationActivity = ActivitySource.StartActivity("ValidateOrder")) | ||
| { | ||
| await ValidateOrderAsync(request); | ||
| validationActivity?.SetTag("validation.result", "passed"); | ||
| } | ||
|
|
||
| // Child span for payment | ||
| using (var paymentActivity = ActivitySource.StartActivity("ProcessPayment", | ||
| ActivityKind.Client)) // Client = outgoing call | ||
| { | ||
| paymentActivity?.SetTag("payment.method", request.PaymentMethod); | ||
| await ProcessPaymentAsync(request); | ||
| } | ||
|
|
||
| var order = new Order { Id = Guid.NewGuid(), CustomerId = request.CustomerId, Status = "Completed" }; | ||
|
|
||
| activity?.SetTag("order.status", "completed"); | ||
| activity?.SetStatus(ActivityStatusCode.Ok); | ||
|
|
||
| return order; | ||
| } | ||
|
danmoseley marked this conversation as resolved.
|
||
| catch (Exception ex) | ||
| { | ||
| activity?.SetStatus(ActivityStatusCode.Error, ex.Message); | ||
| // Log via ILogger — OpenTelemetry captures this with trace correlation. | ||
| // Prefer logging over activity.RecordException() as OTel is deprecating | ||
| // span events for exception recording in favor of log-based exceptions. | ||
| _logger.LogError(ex, "Order processing failed for customer {CustomerId}", request.CustomerId); | ||
| throw; | ||
| } | ||
|
danmoseley marked this conversation as resolved.
|
||
| } | ||
| } | ||
| ``` | ||
|
|
||
| **Critical: `ActivitySource` name must match `AddSource("...")` in configuration.** Unmatched sources are silently ignored — this is the #1 debugging issue. | ||
|
|
||
| ### Step 5: Create custom metrics | ||
|
|
||
| Use `IMeterFactory` (injected via DI) to create meters — this ensures proper lifetime management and testability. | ||
|
|
||
| ```csharp | ||
| using System.Diagnostics; | ||
| using System.Diagnostics.Metrics; | ||
|
|
||
| public class OrderMetrics | ||
| { | ||
| private readonly Counter<long> _ordersProcessed; | ||
| private readonly Histogram<double> _orderProcessingDuration; | ||
| private readonly UpDownCounter<int> _activeOrders; | ||
|
|
||
| public OrderMetrics(IMeterFactory meterFactory) | ||
| { | ||
| // Meter name must match AddMeter("...") in configuration | ||
| var meter = meterFactory.Create("MyApp.Metrics"); | ||
|
|
||
| // Counter — use for things that only go up | ||
| _ordersProcessed = meter.CreateCounter<long>( | ||
| "orders.processed", "orders", "Total orders successfully processed"); | ||
|
|
||
| // Histogram — use for measuring distributions (latency, sizes) | ||
| _orderProcessingDuration = meter.CreateHistogram<double>( | ||
| "orders.processing_duration", "ms", "Time to process an order"); | ||
|
|
||
| // UpDownCounter — use for things that go up AND down | ||
| _activeOrders = meter.CreateUpDownCounter<int>( | ||
| "orders.active", "orders", "Currently processing orders"); | ||
| } | ||
|
|
||
| public void RecordOrderProcessed(string region, double durationMs) | ||
| { | ||
| // Tags enable dimensional filtering (by region, status, etc.) | ||
| var tags = new TagList | ||
| { | ||
| { "region", region }, | ||
| { "order.type", "standard" } | ||
| }; | ||
|
danmoseley marked this conversation as resolved.
|
||
|
|
||
| _ordersProcessed.Add(1, tags); | ||
| _orderProcessingDuration.Record(durationMs, tags); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Register `OrderMetrics` in DI: | ||
|
|
||
| ```csharp | ||
| builder.Services.AddSingleton<OrderMetrics>(); | ||
| ``` | ||
|
|
||
| ### Step 6: Configure context propagation for distributed scenarios | ||
|
|
||
| Trace context propagation is automatic for HTTP calls when using `AddHttpClientInstrumentation()`. For non-HTTP scenarios: | ||
|
|
||
| ```csharp | ||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.Diagnostics; | ||
| using OpenTelemetry.Context.Propagation; | ||
|
|
||
| // ActivitySource should be static — register via .AddSource("MyApp.Messaging") in Step 2 | ||
| private static readonly ActivitySource MessageSource = new("MyApp.Messaging"); | ||
|
|
||
| // Manual context propagation (e.g., across message queues) | ||
| // On the SENDING side: | ||
| var propagator = Propagators.DefaultTextMapPropagator; | ||
| var activityContext = Activity.Current?.Context ?? default; | ||
| var context = new PropagationContext(activityContext, Baggage.Current); | ||
| var carrier = new Dictionary<string, string>(); | ||
|
|
||
| propagator.Inject(context, carrier, (dict, key, value) => dict[key] = value); | ||
| // Send carrier dictionary as message headers | ||
|
|
||
| // On the RECEIVING side: | ||
| var parentContext = propagator.Extract(default, carrier, | ||
| (dict, key) => dict.TryGetValue(key, out var value) ? new[] { value } : Array.Empty<string>()); | ||
|
|
||
| Baggage.Current = parentContext.Baggage; | ||
| using var activity = MessageSource.StartActivity("ProcessMessage", | ||
| ActivityKind.Consumer, | ||
| parentContext.ActivityContext); // Links to parent trace! | ||
| ``` | ||
|
|
||
| ## Validation | ||
|
|
||
| - [ ] Traces appear in the observability backend (Jaeger, Aspire dashboard, etc.) | ||
| - [ ] HTTP requests automatically create spans with correct verb, URL, status code | ||
| - [ ] Custom `ActivitySource` names match `AddSource()` registrations | ||
| - [ ] Custom `Meter` names match `AddMeter()` registrations | ||
| - [ ] Logs include TraceId and SpanId for correlation | ||
| - [ ] Health check endpoints are filtered from traces | ||
| - [ ] Exception details appear on error spans | ||
|
|
||
| ## Common Pitfalls | ||
|
|
||
| | Pitfall | Solution | | ||
| |---------|----------| | ||
| | `ActivitySource.StartActivity` returns null | Source name doesn't match any `AddSource()` — names must match exactly | | ||
| | Traces not appearing in exporter | Check OTLP endpoint: gRPC uses port 4317, HTTP uses 4318 | | ||
|
danmoseley marked this conversation as resolved.
|
||
| | Missing HTTP client spans | Ensure `AddHttpClientInstrumentation()` is registered; it works for both `IHttpClientFactory`/DI and `new HttpClient()` (use `IHttpClientFactory` for lifetime management) | | ||
| | High cardinality tags | Don't use user IDs, request IDs, or UUIDs as metric tags — explodes storage | | ||
| | OTLP gRPC vs HTTP mismatch | Default is gRPC (port 4317); if collector only accepts HTTP, set `OtlpExportProtocol.HttpProtobuf` | | ||
| | `Meter` / `ActivitySource` lifecycle | `ActivitySource` should be static; create `Meter` via `IMeterFactory` from DI (not `new Meter()`) for proper lifetime management and testability | | ||
67 changes: 67 additions & 0 deletions
67
tests/dotnet-aspnet/configuring-opentelemetry-dotnet/eval.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| scenarios: | ||
| - name: "Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core" | ||
| prompt: | | ||
| I'm adding OpenTelemetry to my ASP.NET Core 8 API. I need: | ||
| 1. Distributed tracing with OTLP export to a collector | ||
| 2. A custom span around my OrderService.ProcessOrder method | ||
| 3. A custom counter metric for orders processed | ||
| Show me the complete Program.cs setup and the service class. Don't create files. | ||
| assertions: | ||
| - type: "output_matches" | ||
| pattern: "(OpenTelemetry\\.Extensions\\.Hosting|AddOpenTelemetry)" | ||
| - type: "output_matches" | ||
| pattern: "(ActivitySource|StartActivity)" | ||
| - type: "output_matches" | ||
| pattern: "(AddOtlpExporter|UseOtlpExporter|OtlpExporter)" | ||
| - type: "output_not_contains" | ||
| value: "OpenTelemetry.Exporter.Console" | ||
| rubric: | ||
| - "Included OpenTelemetry.Extensions.Hosting and OpenTelemetry.Instrumentation.Http in the required NuGet packages" | ||
| - "Configured both .WithTracing() and .WithMetrics() in the same AddOpenTelemetry() call" | ||
| - "Registered custom ActivitySource names via AddSource() and showed that the ActivitySource name in the service class must match exactly" | ||
| - "Used IMeterFactory via dependency injection to create the Meter instead of a static Meter constructor" | ||
| - "Did not include gratuitous custom activities, logs, or metrics beyond what was specifically asked for" | ||
| - "Used a service name appropriate to the application, not a generic placeholder like MyOrderService" | ||
| reject_tools: ["bash", "edit"] | ||
| timeout: 180 | ||
|
|
||
| - name: "Configure all three OpenTelemetry signals with correct OTLP export" | ||
| prompt: | | ||
| I need to set up all three OpenTelemetry signals (traces, metrics, logs) in my | ||
| ASP.NET Core 8 app, all exporting via OTLP to my collector. | ||
| What NuGet packages do I need, and show me the complete Program.cs configuration? | ||
| Don't create files. | ||
| assertions: | ||
| - type: "output_matches" | ||
| pattern: "(AddOpenTelemetry|WithTracing|WithMetrics)" | ||
| - type: "output_matches" | ||
| pattern: "(AddOtlpExporter|UseOtlpExporter|OtlpExporter)" | ||
| - type: "output_matches" | ||
| pattern: "(Logging|WithLogging|AddOpenTelemetry|TraceId|SpanId)" | ||
| - type: "output_not_contains" | ||
| value: "OpenTelemetry.Exporter.Console" | ||
| rubric: | ||
|
mrsharm marked this conversation as resolved.
|
||
| - "Configured all three signals: tracing (WithTracing), metrics (WithMetrics), and logging" | ||
| - "Used a unified OTLP exporter configuration rather than repeating exporter setup per signal" | ||
| - "Ensured logs carry the same service identity as traces and metrics" | ||
| - "Listed the complete set of required NuGet packages including the OTLP exporter package" | ||
|
mrsharm marked this conversation as resolved.
|
||
| reject_tools: ["bash", "edit"] | ||
| timeout: 180 | ||
|
|
||
| - name: "Propagate trace context across a message queue" | ||
| prompt: | | ||
| I have two ASP.NET Core 8 services communicating via RabbitMQ. Service A publishes | ||
| an order event and Service B consumes it. Traces show up fine within each service | ||
| but I can't see the end-to-end distributed trace across the queue boundary. | ||
| How do I propagate the trace context through message headers? Don't create files. | ||
| assertions: | ||
| - type: "output_matches" | ||
| pattern: "(Propagat|TextMapPropagator|Inject|Extract)" | ||
| - type: "output_matches" | ||
| pattern: "(ActivitySource|StartActivity)" | ||
| rubric: | ||
| - "Showed how to inject trace context into message headers on the sending side using a TextMapPropagator" | ||
| - "Showed how to extract trace context from message headers on the receiving side and link it as a parent" | ||
| - "Used the extracted context as the parent when starting a new Activity on the consumer so spans connect into one distributed trace" | ||
| reject_tools: ["bash", "edit"] | ||
| timeout: 180 | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.