Add evaluation troubleshooting guide for AI agents by danmoseley · Pull Request #440 · dotnet/skills

danmoseley · 2026-03-25T02:55:56Z

Add evaluation troubleshooting guide for AI agents

Note

This PR was generated with Copilot assistance.

Problem

When skill evaluation produces failures, PR authors have no clear path to diagnose the cause. The eval results table shows scores and verdicts, but doesn't explain what went wrong or how to fix it. Investigating requires manually downloading artifacts from workflow logs, reverse-engineering the results.json schema, and pattern-matching against undocumented failure modes.

Solution

Add eng/skill-validator/InvestigatingResults.md — a troubleshooting guide designed primarily for AI coding agents to read and follow. When any skill fails evaluation, the PR comment now includes a ready-to-use prompt that the PR author can copy-paste to their AI agent. The agent downloads the artifacts, reads the guide, analyzes the results, and suggests fixes.

What's in the guide

results.json schema documentation — verdict, scenario, breakdown, and run metrics fields with descriptions
Scoring formula — breakdown field weights (quality 0.40, judgment 0.30, completion 0.15, etc.) so agents can interpret why scores are negative
7 common failure patterns with symptoms, causes, and fixes:
1. Timeout with empty output
2. Baseline already bad
3. High variance across runs
4. Quality unchanged but weighted score negative
5. Skill not activated
6. Rubric penalizes valid alternatives
7. Judge regressions on close calls
Priority ordering for when multiple patterns apply (fix timeouts first, then activation, then baseline, etc.)
gh run download instructions for artifact retrieval

Changes

File	Change
`eng/skill-validator/InvestigatingResults.md`	New troubleshooting guide
`eng/skill-validator/README.md`	Link to guide from Output section
`CONTRIBUTING.md`	Link to guide from CI evaluation section
`eng/skill-validator/src/Evaluate/Reporter.cs`	Show doc link in PR comment when any skill fails
`.github/workflows/evaluation.yml`	Add copy-paste agent prompt with run ID when failures exist

Validation

Tested the guide by launching 4 fresh AI agents (clean context, no prior analysis) against eval artifacts from 2 different PRs (PR #317 and PR #428). Across 19 failing scenarios:

95% correct pattern identification (18/19)
100% correct fix suggestions (19/19)
0 hallucinations
All agents rated the guide 4/5 for cold-start usability

Add InvestigatingResults.md with: - How to find and download evaluation artifacts from workflow runs - Full results.json schema documentation - Common failure patterns (timeouts, empty output, variance, rubric issues, skill not activated) with diagnosis steps and fixes - Example analysis script for AI agents to parse results Link the doc from: - Reporter.cs: show a link in the PR comment when any skill fails - skill-validator README.md: Output section - CONTRIBUTING.md: CI evaluation section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-25T02:56:07Z

Note

This PR is from a fork and modifies infrastructure files (eng/ or .github/).

Changes to infrastructure typically need to be submitted from a branch in dotnet/skills (not a fork) so that CI workflows run with the correct permissions and secrets.

Please consider recreating this PR from an upstream branch. If you don't have push access to dotnet/skills, ask a maintainer to push your branch for you.

Copilot

Pull request overview

Adds a troubleshooting guide to help diagnose skill evaluation failures and surfaces it from the evaluation outputs/PR comment, so authors (and AI agents) have a documented path to interpret results.json and act on common failure modes.

Changes:

Add a new InvestigatingResults.md guide documenting results.json, scoring, and common failure patterns.
Link the guide from skill-validator docs (README.md) and repo contributor docs (CONTRIBUTING.md).
Surface the guide link (and an AI copy/paste investigation prompt) in evaluation PR comments when failures occur.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`eng/skill-validator/src/Evaluate/Reporter.cs`	Adds a failure-only note in the generated markdown summary pointing to the troubleshooting guide.
`eng/skill-validator/README.md`	Links to the new troubleshooting guide from the output documentation.
`eng/skill-validator/InvestigatingResults.md`	New end-to-end guide for downloading artifacts and interpreting failures via `results.json`.
`CONTRIBUTING.md`	Adds a link to the troubleshooting guide from the CI evaluation section.
`.github/workflows/evaluation.yml`	Appends an AI-agent copy/paste investigation prompt to the consolidated PR comment when failures are detected.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-25T02:59:23Z

+| Field | Description |
+|-------|-------------|
+| `model` | Model used for agent runs |
+| `judgeModel` | Model used for judging |
+| `timestamp` | When the run started |
+| `verdicts[]` | Array of per-skill results |
+


The markdown tables in this guide start with || (e.g., || Field | Description |), which renders as an extra empty column in standard GitHub Markdown. Remove the extra leading | so the tables render correctly (and apply consistently to the other tables in this file).

Copilot · 2026-03-25T02:59:23Z

+Each scenario contains three runs and their comparison:
+
+| Field | Description |
+|-------|-------------|
+| `scenarioName` | Human-readable scenario name |
+| `baseline` | Run without the skill |
+| `skilledIsolated` | Run with only this skill loaded |
+| `skilledPlugin` | Run with the full plugin loaded |


This section says each scenario contains three runs, but skilledPlugin is optional (only present when plugin runs are enabled). Consider rewording to "two runs (baseline + isolated) plus an optional plugin run" to match the actual results.json structure.

Suggested change

Each scenario contains three runs and their comparison:

| Field | Description |

|-------|-------------|

| `scenarioName` | Human-readable scenario name |

| `baseline` | Run without the skill |

| `skilledIsolated` | Run with only this skill loaded |

| `skilledPlugin` | Run with the full plugin loaded |

Each scenario includes two runs (baseline + isolated) plus an optional plugin run, and their comparison:

| Field | Description |

|-------|-------------|

| `scenarioName` | Human-readable scenario name |

| `baseline` | Run without the skill |

| `skilledIsolated` | Run with only this skill loaded |

| `skilledPlugin` | Run with the full plugin loaded (present only when plugin runs are enabled) |

Copilot · 2026-03-25T02:59:24Z


+        bool anyFailure = verdicts.Any(v => !v.Passed);
+        if (anyFailure)
+            sb.AppendLine("\n> 📖 See [InvestigatingResults.md](https://github.com/dotnet/skills/blob/main/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures — or use the copy-paste prompt below.");


This adds a hard-coded link to dotnet/skills/blob/main/... and mentions a "copy-paste prompt below". When the markdown summary is generated outside the PR-comment workflow (e.g., local runs, or other consumers of summary.md), there may be no prompt below; and the hard-coded repo/branch makes the link stale if the default branch or repository differs. Consider linking more robustly (e.g., a repo-relative path, or injecting repo+ref from the workflow) and rewording so it doesn’t assume the prompt is present.

Suggested change

sb.AppendLine("\n> 📖 See [InvestigatingResults.md](https://github.com/dotnet/skills/blob/main/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures — or use the copy-paste prompt below.");

sb.AppendLine("\n> 📖 See [InvestigatingResults.md](/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures. Additional debugging guidance may be provided by your workflow.");

Copilot · 2026-03-25T02:59:24Z

+                echo ""
+                echo "> **To investigate failures**, paste this to your AI coding agent:"
+                echo ">"
+                echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --dir /tmp/eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/main/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._"


The troubleshooting prompt fetches the guide from a hard-coded .../main/... URL. Using the repository’s default branch (or a specific ref/sha associated with the run) would make this more robust and avoid breaking if the default branch name changes.

Suggested change

echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --dir /tmp/eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/main/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._"

echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --dir /tmp/eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/${{ github.sha }}/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._"

danmoseley · 2026-03-25T03:08:46Z

Superseded by #441 (same branch pushed to upstream so CI runs with correct permissions). Review feedback addressed there.

danmoseley requested review from JanKrivanek, ViktorHofer, dbreshears and timheuer as code owners March 25, 2026 02:55

Copilot AI review requested due to automatic review settings March 25, 2026 02:55

Copilot started reviewing on behalf of danmoseley March 25, 2026 02:56 View session

danmoseley mentioned this pull request Mar 25, 2026

Add skills for C# MCP Server Development #317

Merged

Copilot AI reviewed Mar 25, 2026

View reviewed changes

Update InvestigatingResults.md

4a2d6ce

danmoseley mentioned this pull request Mar 25, 2026

Add evaluation troubleshooting guide for AI agents #441

Merged

danmoseley closed this Mar 25, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add evaluation troubleshooting guide for AI agents - #440

Add evaluation troubleshooting guide for AI agents#440
danmoseley wants to merge 2 commits into
dotnet:mainfrom
danmoseley:danmose/eval-troubleshooting

danmoseley commented Mar 25, 2026

Uh oh!

github-actions Bot commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

danmoseley commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	sb.AppendLine("\n> 📖 See [InvestigatingResults.md](https://github.com/dotnet/skills/blob/main/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures — or use the copy-paste prompt below.");
	sb.AppendLine("\n> 📖 See [InvestigatingResults.md](/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures. Additional debugging guidance may be provided by your workflow.");

Uh oh!

Conversation

danmoseley commented Mar 25, 2026

Add evaluation troubleshooting guide for AI agents

Problem

Solution

What's in the guide

Changes

Validation

Uh oh!

github-actions Bot commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

danmoseley commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants