Add configuring-opentelemetry-dotnet skill by mrsharm · Pull Request #268 · dotnet/skills

mrsharm · 2026-03-06T17:23:37Z

Adds the configuring-opentelemetry-dotnet skill for setting up OpenTelemetry in ASP.NET Core.

Note: Replaces #91 (migrated from skills-old repo to new plugins/ structure).

What the Skill Teaches

The base model gets several OpenTelemetry patterns wrong or incomplete. This skill addresses five key gaps:

Correct NuGet package set — The base model often omits OpenTelemetry.Instrumentation.Http or installs the bare OpenTelemetry package instead of OpenTelemetry.Extensions.Hosting. The skill lists the exact packages needed for each signal.
ActivitySource/Meter name registration — Custom ActivitySource and Meter names must match AddSource()/AddMeter() exactly or telemetry is silently dropped. The base model rarely warns about this, which is the Initial documentation and validation workflow #1 debugging issue.
IMeterFactory via DI — The base model typically creates a static new Meter(). The skill teaches the modern IMeterFactory pattern via dependency injection for proper lifetime management.
Logging provider resource gap — ConfigureResource() on the tracing/metrics pipeline does NOT propagate to the logging provider. The skill teaches setting SetResourceBuilder explicitly on the logging provider so logs carry the same service name.
Non-HTTP context propagation — Trace context is automatic for HTTP calls, but the base model struggles with manual propagation across message queues using TextMapPropagator.Inject/Extract.

Eval Results

Scenario	Baseline	With Skill	Δ	Overfit	Verdict
Set up tracing+metrics with custom spans	3.2/5	5.0/5	+56%	0.09	✅
Configure all three signals with OTLP	4.0/5	5.0/5	+25%	0.09	✅
Propagate trace context across message queue	3.2/5	5.0/5	+56%	0.09	✅

Overall: +25.6% | Model: claude-opus-4.6 | Judge: claude-opus-4.6

mrsharm · 2026-03-06T17:26:26Z

Migration Note

This PR replaces #91 which was opened from mrsharm/skills-old. The skill and eval files have been migrated to the new plugins/ directory structure:

src/dotnet/skills/configuring-opentelemetry-dotnet/ → plugins/dotnet/skills/configuring-opentelemetry-dotnet/
src/dotnet/tests/configuring-opentelemetry-dotnet/ → tests/dotnet/configuring-opentelemetry-dotnet/

All prior review feedback from #91 still applies — please see that PR for the full discussion history.

Copilot

Pull request overview

Adds a new .NET skill, configuring-opentelemetry-dotnet, intended to guide agents through setting up OpenTelemetry (tracing/metrics/logging) in ASP.NET Core, with accompanying evaluation scenarios and CODEOWNERS entries.

Changes:

Added configuring-opentelemetry-dotnet skill documentation under plugins/dotnet/skills/.
Added evaluation scenarios under tests/dotnet/configuring-opentelemetry-dotnet/.
Registered code ownership for the new skill and test directory.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
plugins/dotnet/skills/configuring-opentelemetry-dotnet/SKILL.md	Introduces the new OpenTelemetry setup skill content and code snippets.
tests/dotnet/configuring-opentelemetry-dotnet/eval.yaml	Adds eval scenarios to validate skill activation and avoid false positives.
.github/CODEOWNERS	Assigns ownership for the new skill and its evals.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

mrsharm · 2026-03-06T17:43:16Z

Feedback carried over from #91

PR #91 — Add configuring-opentelemetry-dotnet skill (open)

18 review threads — Reviewers: copilot-pull-request-reviewer, tarekgh, noahfalk

Reviewer	Feedback
copilot	Missing `using OpenTelemetry.Trace;` for `SetStatus()`/`RecordException()`
copilot	File wrapped in ```skill code block — metadata won't parse
copilot	`return order;` references undefined variable
tarekgh	Does `SetDbStatementForText` exist in latest SqlClient instrumentation?
tarekgh	Does it need to reference `OpenTelemetry.Instrumentation.Runtime` package?
tarekgh	Traces don't configure endpoint but metrics do?
tarekgh	Missing `using` directives
tarekgh	Is `GetQueueDepth` just demonstrating the idea?
tarekgh	Is HttpClient instrumentation accurate for clients not from `IHttpClientFactory`?
noahfalk	Fix package list (suggestion provided)
noahfalk	Fix description (suggestion provided)
noahfalk	SQL instrumentation should be clearly marked optional
noahfalk	Runtime metrics should be marked optional
noahfalk	Custom tracing spans should also be optional (but more useful)
noahfalk	Use `IMeterFactory` instead of static `Meter` per official guidance
noahfalk	What about logs/metrics verification?
noahfalk	`IMeterFactory` again for custom metrics section
noahfalk	Eval prompts should be simpler/more generalized (e.g. "Please enable telemetry for my app")

danmoseley · 2026-03-07T08:10:50Z

Now we have new plugin factoring, moved the skill from plugins/dotnet/ to plugins/dotnet-diag/. OpenTelemetry is an observability/diagnostics concern and fits better alongside the other diagnostic skills (trace collection, performance analysis, etc.) in the dotnet-diag plugin.

ManishJayaswal · 2026-03-10T02:11:37Z

@BrennanConroy @adityamandaleeka - please review
@mrsharm - you may want to update the branch. This skill should also be under ASP plugin.

danmoseley · 2026-03-24T01:47:35Z

@mrsharm this seems ASP.NET specific? If so should probably go in the dotnet-aspnet plugin not dotnet-diag ?

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

danmoseley · 2026-03-28T05:36:55Z

/evaluate

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-03-28T05:44:38Z

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
configuring-opentelemetry-dotnet	Add OpenTelemetry tracing and metrics to an ASP.NET Core API	1.3/5 ⏰ → 1.3/5 ⏰	✅ configuring-opentelemetry-dotnet; tools: skill, edit / ✅ configuring-opentelemetry-dotnet; tools: skill, edit	✅ 0.11	✅
configuring-opentelemetry-dotnet	OpenTelemetry skill should not activate for simple logging question	5.0/5 → 5.0/5	⚠️ NOT ACTIVATED / ⚠️ NOT ACTIVATED	✅ 0.11	❌ [1]

[1] (Plugin) Quality unchanged but weighted score is -1.3% due to: time (10.0s → 14.1s)

⏰ timeout — run(s) hit the (120s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

📖 See InvestigatingResults.md for how to diagnose failures. Additional debugging guidance may be provided by your workflow.

🔍 Full results — includes quality and agent details

To investigate failures, paste this to your AI coding agent:

For PR 268 in dotnet/skills, download eval artifacts with gh run download 23678561832 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/503f4b305192b45401c6e94346fda3bda8722bc3/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ViktorHofer · 2026-03-31T16:13:38Z

/evaluate

ViktorHofer

Infra looks good. Don't merge until you have an area expert sign-off.

github-actions · 2026-03-31T16:16:37Z

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
configuring-opentelemetry-dotnet	Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core	3.3/5 → 5.0/5 🟢	✅ configuring-opentelemetry-dotnet; tools: skill, report_intent	✅ 0.09	✅
configuring-opentelemetry-dotnet	Configure all three OpenTelemetry signals with correct OTLP endpoints	4.3/5 → 5.0/5 🟢	✅ configuring-opentelemetry-dotnet; tools: report_intent, skill / ✅ configuring-opentelemetry-dotnet; tools: skill, report_intent	✅ 0.09	✅ [1]
configuring-opentelemetry-dotnet	Propagate trace context across a message queue	3.0/5 → 5.0/5 🟢	✅ configuring-opentelemetry-dotnet; tools: report_intent, skill / ✅ configuring-opentelemetry-dotnet; tools: skill	✅ 0.09	✅

[1] ⚠️ High run-to-run variance (CV=1.84) — consider re-running with --runs 5

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

tarekgh · 2026-03-31T20:22:08Z

CC @noahfalk @rajkumar-rangaraj @JamesNK

tarekgh

Added some questions/suggestions. Also, CC'ed some more people who will be good if can take a look too. LGTM otherwise!

mrsharm · 2026-04-06T10:46:03Z

/evaluate

mrsharm · 2026-04-06T10:47:10Z

@noahfalk / @tarekgh - addressed a bunch of feedback. PTAL - I'd like to merge this if there are no immediate issues and we can address any secondary problems in a separate PR.

github-actions · 2026-04-06T10:50:15Z

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
configuring-opentelemetry-dotnet	Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core	4.0/5 → 4.7/5 🟢	✅ configuring-opentelemetry-dotnet; tools: skill	✅ 0.13	✅ [1]
configuring-opentelemetry-dotnet	Configure all three OpenTelemetry signals with correct OTLP export	3.0/5 → 5.0/5 🟢	✅ configuring-opentelemetry-dotnet; tools: skill, report_intent	✅ 0.13	✅
configuring-opentelemetry-dotnet	Propagate trace context across a message queue	4.0/5 → 5.0/5 🟢	✅ configuring-opentelemetry-dotnet; tools: report_intent, skill / ✅ configuring-opentelemetry-dotnet; tools: skill, report_intent	✅ 0.13	✅ [2]

[1] ⚠️ High run-to-run variance (CV=1.45) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=1.39) — consider re-running with --runs 5

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions

mrsharm · 2026-04-06T17:38:04Z

/evaluate

mrsharm · 2026-04-06T17:38:34Z

@ViktorHofer - if things good look to you, should be good to merge -- approved by the area expert.

github-actions · 2026-04-06T17:42:29Z

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
configuring-opentelemetry-dotnet	Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core	3.7/5 → 4.7/5 🟢	✅ configuring-opentelemetry-dotnet; tools: skill, report_intent / ✅ configuring-opentelemetry-dotnet; tools: skill	✅ 0.13	✅ [1]
configuring-opentelemetry-dotnet	Configure all three OpenTelemetry signals with correct OTLP export	3.0/5 → 5.0/5 🟢	✅ configuring-opentelemetry-dotnet; tools: skill / ✅ configuring-opentelemetry-dotnet; tools: skill, report_intent	✅ 0.13	✅
configuring-opentelemetry-dotnet	Propagate trace context across a message queue	4.0/5 → 5.0/5 🟢	✅ configuring-opentelemetry-dotnet; tools: skill	✅ 0.13	✅

[1] ⚠️ High run-to-run variance (CV=1.94) — consider re-running with --runs 5

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions

* Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss) Teaches OpenTelemetry SDK configuration in ASP.NET Core: package selection, AddOpenTelemetry with tracing/metrics/logging, custom ActivitySource and Meter creation, OTLP exporter setup, and context propagation. Eval results: +8.1% improvement (threshold: 10%, near-miss) Includes eval.yaml with OTel setup scenario + negative test. * Sharpen eval.yaml: add custom-spans-invisible pain-point and name-matching rubric emphasis * Migrate configuring-opentelemetry-dotnet to plugins/ directory structure * Move OpenTelemetry skill to new dotnet-aspnet plugin, address review feedback - Move skill from plugins/dotnet-diag to new plugins/dotnet-aspnet plugin - Create dotnet-aspnet plugin.json - Fix SKILL.md frontmatter: remove code fence wrapper so YAML metadata parses - Fix undefined 'order' variable in ProcessOrderAsync example - Move SqlClient/Runtime instrumentation to clearly marked optional sections - Add missing using directives (System.Diagnostics, OpenTelemetry.Context.Propagation) - Handle null Activity.Current in context propagation snippet - Use IMeterFactory via DI instead of static Meter per official guidance - Add OpenTelemetry.Instrumentation.Http to eval rubric package list - Simplify eval prompt to be more natural/generalized - Update CODEOWNERS: add dotnet-aspnet plugin entries, remove old dotnet-diag entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Configure metrics OTLP exporter endpoint explicitly to match tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address round 2 review: logging endpoint, marketplace, RecordException using, ActivitySource in propagation, HttpClient pitfall - Configure logging OTLP exporter endpoint explicitly for consistency - Add dotnet-aspnet to both marketplace.json files - Add 'using OpenTelemetry.Trace' for RecordException extension method - Declare ActivitySource in context propagation snippet - Fix HttpClient instrumentation pitfall (works with new HttpClient() too) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify logging package source and add ActivitySource registration reminder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add @dotnet/aspnet team to skill CODEOWNERS entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dotnet-aspnet to README table, make propagation ActivitySource static Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add expect_activation:false, IMeterFactory using, clarify OTLP exporter scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OtlpExporter using directive, clarify OTLP package covers logging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify Jaeger is traces-only, document logging package provenance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval: increase timeout, reject tools, rewrite prompt as explanation-only - Scenario 1: rewrite prompt to ask for code explanation (not file modification), add reject_tools to prevent bash/edit overhead, increase timeout 120s->360s, update rubric wording - Scenario 2: add reject_tools to eliminate tool-overhead noise Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve OTel skill evals and address review feedback - Remove redundant per-skill CODEOWNERS lines (ViktorHofer feedback) - Add SetResourceBuilder + ParseStateValues to SKILL.md Step 3 (logging provider gap) - Rewrite eval scenarios: 2 targeted positive scenarios testing real skill gaps - Remove noisy negative activation scenario (high variance, -21.5% drag) - Eval results: +14.2% improvement (threshold: 10%), overfitting 0.10 (low) Scenario 1: 3.0/5 -> 5.0/5, Scenario 2: 3.8/5 -> 5.0/5 * Add propagate-trace-context scenario to OTel evals (+25.6%) Third scenario tests trace context propagation across message queues, where the base model scores 3.2/5 and skill-enhanced scores 5.0/5. Removed metrics and debug scenarios that baseline already aced (5.0/5) which only added token cost without measurable improvement. Final: 3 scenarios, +25.6% improvement, overfitting 0.09 (low) * Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by L86-87) * Use @dotnet/aspnet team alias in CODEOWNERS for wider reach * Addressed feedback --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com>

mrsharm requested review from a team and Copilot March 6, 2026 17:23

mrsharm requested review from dbreshears and timheuer as code owners March 6, 2026 17:23

Copilot started reviewing on behalf of mrsharm March 6, 2026 17:24 View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

danmoseley force-pushed the musharm/configuring-opentelemetry-dotnet-skill branch from d2d71e4 to 1edd695 Compare March 7, 2026 08:10

danmoseley added the skill label Mar 9, 2026

github-actions Bot mentioned this pull request Mar 21, 2026

🏥 Repository Health Dashboard #288

Closed

danroth27 added the area-aspnet label Mar 23, 2026

Copilot AI review requested due to automatic review settings March 28, 2026 05:23

Copilot started reviewing on behalf of danmoseley March 28, 2026 05:23 View session

Copilot AI reviewed Mar 28, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 28, 2026 05:35

Copilot started reviewing on behalf of danmoseley March 28, 2026 05:36 View session

Copilot AI reviewed Mar 28, 2026

View reviewed changes

danmoseley requested a review from Copilot March 28, 2026 05:44

Copilot started reviewing on behalf of danmoseley March 28, 2026 05:44 View session

Copilot AI reviewed Mar 28, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

danmoseley requested a review from Copilot March 28, 2026 05:52

Copilot started reviewing on behalf of danmoseley March 28, 2026 05:53 View session

Copilot AI reviewed Mar 28, 2026

View reviewed changes

Comment thread .github/CODEOWNERS Outdated

danmoseley requested a review from Copilot March 28, 2026 05:57

ViktorHofer approved these changes Mar 31, 2026

View reviewed changes

tarekgh reviewed Mar 31, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

tarekgh reviewed Mar 31, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

tarekgh reviewed Mar 31, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

tarekgh reviewed Mar 31, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

tarekgh reviewed Mar 31, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

tarekgh approved these changes Mar 31, 2026

View reviewed changes

noahfalk reviewed Apr 1, 2026

View reviewed changes

Addressed feedback

e58d054

github-actions Bot added a commit that referenced this pull request Apr 6, 2026

Update session data (PR #268)

1b15c44

mrsharm requested review from noahfalk and tarekgh April 6, 2026 10:55

tarekgh reviewed Apr 6, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

tarekgh reviewed Apr 6, 2026

View reviewed changes

Comment thread plugins/dotnet-aspnet/skills/configuring-opentelemetry-dotnet/SKILL.md Outdated

tarekgh approved these changes Apr 6, 2026

View reviewed changes

mrsharm enabled auto-merge (squash) April 6, 2026 16:26

Addressed feedback

d191a20

github-actions Bot added a commit that referenced this pull request Apr 6, 2026

Update session data (PR #268)

e1aecb2

noahfalk approved these changes Apr 7, 2026

View reviewed changes

mrsharm merged commit 665c39e into dotnet:main Apr 7, 2026
32 checks passed

Uh oh!

Conversation

mrsharm commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What the Skill Teaches

Eval Results

Uh oh!

mrsharm commented Mar 6, 2026

Migration Note

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrsharm commented Mar 6, 2026

Feedback carried over from #91

PR #91 — Add configuring-opentelemetry-dotnet skill (open)

Uh oh!

danmoseley commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ManishJayaswal commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danmoseley commented Mar 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danmoseley commented Mar 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Mar 28, 2026

Skill Validation Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

ViktorHofer commented Mar 31, 2026

Uh oh!

ViktorHofer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 31, 2026

Skill Validation Results

Uh oh!

Uh oh!

tarekgh commented Mar 31, 2026

Uh oh!

Uh oh!

mrsharm commented Mar 6, 2026 •

edited

Loading

danmoseley commented Mar 7, 2026 •

edited

Loading

ManishJayaswal commented Mar 10, 2026 •

edited

Loading

ViktorHofer left a comment •

edited

Loading