Add evaluation troubleshooting guide for AI agents

.github/workflows/evaluation.yml

@@ -642,6 +642,14 @@ jobs:
               cat summary-body.md
               echo ""
               echo "[Full results]($RUN_URL)"
+              # If any skill failed, add a copy-paste prompt for AI-assisted investigation
+              if grep -q '❌' summary-body.md; then
+                RUN_ID="${{ github.run_id }}"
+                echo ""
+                echo "> **To investigate failures**, paste this to your AI coding agent:"
+                echo ">"
+                echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --dir /tmp/eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/main/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._"
+              fi
             } > consolidated-comment.md
 
             cat consolidated-comment.md >> $GITHUB_STEP_SUMMARY

CONTRIBUTING.md

@@ -288,6 +288,8 @@ dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate -
 
 Tests run automatically on pull requests that modify files under `plugins/`. The evaluation workflow discovers changed plugins and runs the skill-validator for each one. Results are posted as a PR comment and uploaded as build artifacts.
 
+If a scenario fails or regresses, see [Investigating Results](eng/skill-validator/InvestigatingResults.md) for how to download artifacts, interpret `results.json`, and diagnose common failure patterns.
+
 ## Writing style
 
 - Be concise and specific.

eng/skill-validator/InvestigatingResults.md

@@ -0,0 +1,264 @@
+# Investigating Evaluation Results
+
+This guide is intended primarily for AI agents investigating skill evaluation failures, though humans will find it useful too. It documents the `results.json` schema, common failure patterns, and recommended fixes.
+
+## Using this guide with an AI agent
+
+This document is designed to be read by AI coding agents. When a skill evaluation has failures, the PR comment includes a ready-to-use prompt — just copy and paste it to your AI agent. The agent will download the artifacts, read this guide, analyze the results, and suggest fixes.
+
+If you need to run the investigation manually, follow the [Quick start](#quick-start) below.
+
+## Quick start
+
+1. **Download the results artifact:** `gh run download <run-id> --repo dotnet/skills --dir <path>`
+2. **Read `summary.md` first** for a quick overview of which scenarios passed/failed
+3. **Read `results.json`** for the full metrics, agent output, assertions, and judge reasoning
+4. **Identify the failure pattern** using the categories below — most failures match multiple patterns; fix them in priority order (timeouts first, then activation, then quality/rubric issues)
+5. **Apply the fix** and re-run with `/evaluate`
+
+## Finding the artifacts
+
+### Via CLI (recommended for AI agents)
+
+Extract the workflow run ID from the **Full results** link in the PR eval comment (e.g., `https://github.com/dotnet/skills/actions/runs/23520818616` → `23520818616`), then:
+
+```bash
+gh run download <run-id> --repo dotnet/skills --dir /tmp/eval-results
+```
+
+This downloads all artifacts into subdirectories, each containing `results.json` and `summary.md`.
+
+### Via browser
+
+From the PR comment, click the **Full results** link to open the GitHub Actions workflow run. Then:
+
+1. Click on any job (e.g., `evaluate (mcp-csharp-debug)`)
+2. Expand the **Upload results** step
+3. Find the `Artifact download URL` in the log output
+4. Download and extract
+
+Alternatively, scroll to the bottom of the workflow run summary page and download from the **Artifacts** section.
+
+## Understanding `results.json`
+
+Each file contains a top-level object with:
+
+| Field | Description |
+|-------|-------------|
+| `model` | Model used for agent runs |
+| `judgeModel` | Model used for judging |
+| `timestamp` | When the run started |
+| `verdicts[]` | Array of per-skill results |
+
+### Verdict structure
+
+Each verdict contains:
+
+| Field | Description |
+|-------|-------------|
+| `skillName` | Name of the skill being evaluated |
+| `passed` | Overall pass/fail |
+| `scenarios[]` | Array of per-scenario comparisons |
+| `overfittingResult` | Overfitting analysis (if enabled) |
+
+### Scenario structure
+
+Each scenario contains three runs and their comparison:
+
+| Field | Description |
+|-------|-------------|
+| `scenarioName` | Human-readable scenario name |
+| `baseline` | Run without the skill |
+| `skilledIsolated` | Run with only this skill loaded |
+| `skilledPlugin` | Run with the full plugin loaded |
+| `timedOut` | Whether any run hit the timeout |
+| `isolatedImprovementScore` | Weighted improvement (isolated vs baseline) |
+| `pluginImprovementScore` | Weighted improvement (plugin vs baseline) |
+| `isolatedBreakdown` | Per-metric contribution to the score (see below) |
+| `pluginBreakdown` | Per-metric contribution to the score (see below) |
+| `pairwiseResult` | Judge's rubric-by-rubric comparison |
+| `perRunScores` | Individual run scores (shows variance) |
+
+### Breakdown fields
+
+The `isolatedBreakdown` and `pluginBreakdown` objects show how each metric contributed to the improvement score. Each field is a raw delta (not yet weighted). The final score is computed as a weighted sum:
+
+| Field | Weight | Range | Meaning |
+|-------|--------|-------|---------|
+| `qualityImprovement` | 0.40 | [-1, 1] | Rubric-based quality delta |
+| `overallJudgmentImprovement` | 0.30 | [-1, 1] | Holistic judge assessment delta |
+| `taskCompletionImprovement` | 0.15 | {-1, 0, 1} | Did assertions pass? |
+| `tokenReduction` | 0.05 | [-1, 1] | Positive = fewer tokens (more efficient) |
+| `errorReduction` | 0.05 | [-1, 1] | Positive = fewer errors |
+| `toolCallReduction` | 0.025 | [-1, 1] | Positive = fewer tool calls |
+| `timeReduction` | 0.025 | [-1, 1] | Positive = faster |
+
+A `tokenReduction` of -1.0 means the skilled run used ≥2× the baseline's tokens. This is common when a skill is loaded (the skill content itself consumes tokens) but is only -0.05 in the final score, so it rarely causes failure on its own.
+
+### Run metrics
+
+Each of `baseline`, `skilledIsolated`, and `skilledPlugin` contains a `metrics` object:
+
+| Field | Description |
+|-------|-------------|
+| `timedOut` | Whether this run hit the timeout |
+| `wallTimeMs` | Total wall-clock time |
+| `taskCompleted` | Whether assertions passed |
+| `tokenEstimate` | Total tokens used |
+| `turnCount` | Number of agent turns |
+| `toolCallCount` | Number of tool calls |
+| `toolCallBreakdown` | Tool call counts by tool name |
+| `errorCount` | Number of errors during the run |
+| `assertionResults[]` | Per-assertion pass/fail with messages |
+| `agentOutput` | The agent's final text output |
+
+## Common failure patterns
+
+### 1. Timeout with empty output
+
+**Symptoms:**
+- `timedOut: true`
+- `agentOutput` is empty or just `\n\n`
+- All assertions fail
+- `toolCallBreakdown` shows `bash` usage
+
+**Cause:** The model spent its entire time budget running shell commands (e.g., `dotnet new`, `dotnet add package`, exploring NuGet contents) and never produced user-facing text.
+
+**Fixes:**
+- **Increase `timeout`** in `eval.yaml` — 180s is often not enough for scenarios that involve code generation. Try 360s.
+- **Restructure the prompt** to discourage bash exploration (e.g., "Show me the code" rather than "Create a project")
+- **Add `reject_tools: ["bash"]`** if the scenario should be answerable without shell commands
+
+### 2. Baseline already bad
+
+**Symptoms:**
+- Baseline scores are very low (1.0–2.0/5)
+- Skilled scores are also low
+- Quality improvement shows 0 or negative
+
+**Cause:** The question is too hard for the model even without the skill. The skill can't fix what the model can't do.
+
+**Fixes:**
+- Simplify the scenario prompt
+- Verify the baseline is working by examining `baseline.metrics.agentOutput`
+- Consider whether the scenario is testing the right thing
+
+### 3. High variance across runs
+
+**Symptoms:**
+- `perRunScores` contains both positive and negative values (e.g., `[0.07, -0.85, 0.04]`)
+- A spread greater than ~0.3 between min and max scores suggests problematic variance
+- Results flip between passing and failing across eval runs
+- Isolated and plugin scores disagree
+
+**Cause:** LLM non-determinism. The model takes different strategies on different runs.
+
+**Fixes:**
+- **Increase `--runs`** for more statistical stability (5 is the default; consider 7–10 for noisy scenarios)
+- **Tighten the prompt** to reduce the space of valid strategies
+- **Add `setup.files`** to give the model concrete files to work with rather than letting it scaffold from scratch
+
+### 4. Quality unchanged but weighted score negative
+
+**Symptoms:**
+- Footnote says "Quality unchanged but weighted score is -X% due to: judgment, tokens, tool calls"
+- The skilled output is roughly as good as baseline
+
+**Cause:** The skill adds token overhead (the skill content itself uses tokens) but doesn't improve quality enough to offset it.
+
+**Fixes:**
+- **Improve the skill content** to produce clearly better output for this scenario
+- **Reduce skill size** — shorter skills have less token overhead
+- **Check if the rubric matches** what the skill actually teaches
+
+### 5. Skill not activated
+
+**Symptoms:**
+- Skills Loaded column shows `⚠️ NOT ACTIVATED`
+- Skilled run has near-zero tokens (e.g., <100), 0 turns, 0 tools
+- The `turnCount` being 0 is the clearest signal — a small token count with 0 turns indicates the skill was loaded but the agent never ran
+
+**Cause:** The agent runtime didn't select the skill for this prompt. The skill's frontmatter `description` didn't match.
+
+**Fixes:**
+- Update the skill's `description` in SKILL.md frontmatter to better match the scenario prompt
+- Make sure the description includes keywords from the scenario
+
+### 6. Rubric penalizes valid alternatives
+
+**Symptoms:**
+- Pairwise judge picks baseline over skill
+- Both outputs are correct but use different approaches
+- `pairwiseResult.rubricResults` shows the rubric criterion is too narrow
+
+**Cause:** The rubric item favors one specific approach (e.g., step-by-step UI walkthrough) over an equally valid alternative (e.g., single CLI command).
+
+**Fixes:**
+- **Broaden the rubric** to explicitly accept multiple valid approaches
+- Example: Instead of `"Shows step-by-step UI configuration"`, use `"Explains how to connect — either as a single CLI command or via the UI configuration"`
+
+### 7. Judge regressions on close calls
+
+**Symptoms:**
+- `overallJudgmentImprovement` is -0.4 even though quality scores are similar
+- Pairwise judge is inconsistent between position-swapped runs
+
+**Cause:** When outputs are nearly equal, the judge's position bias can dominate. The position-swap mitigation defaults to "tie" on inconsistency, but the weighted scoring still penalizes.
+
+**Fixes:**
+- This is usually noise — re-run the eval to see if it persists
+- If it consistently happens, improve the skill to produce clearly differentiated output
+
+## When multiple patterns apply
+
+Most failing scenarios match 2–3 patterns simultaneously (e.g., timeout + token overhead + high variance). Fix them in this priority order:
+
+1. **Timeouts (#1)** — if the model can't finish, nothing else matters. Increase timeout first.
+2. **Skill not activated (#5)** — if the skill never loaded, fix the description before tuning anything else.
+3. **Baseline already bad (#2)** — if the baseline scores ≤2.0/5, the scenario may need simplification regardless of the skill.
+4. **High variance (#3)** — if `perRunScores` are unstable, a single eval run is unreliable. Re-run before concluding the skill is broken.
+5. **Rubric/judgment issues (#6, #7)** — once the runs are stable, tune the rubric.
+6. **Token overhead (#4)** — only optimize if quality is already good but the weighted score is marginally negative.
+
+## Analyzing results with an AI agent
+
+The `results.json` file is designed to be machine-readable. An AI agent can:
+
+1. **Parse the JSON** and extract metrics for each scenario
+2. **Compare baseline vs skilled** metrics to identify regressions
+3. **Read `agentOutput`** to see what the model actually produced
+4. **Check `assertionResults`** to see which assertions failed
+5. **Read `pairwiseResult.rubricResults`** for the judge's per-criterion reasoning
+6. **Examine `perRunScores`** to assess variance
+7. **Look at `toolCallBreakdown`** to understand what the model spent time on
+8. **Cross-reference `isolatedBreakdown`** to see which metrics drove the score
+
+### Example analysis script
+
+```python
+import json
+
+def analyze(path):
+    with open(path) as f:
+        data = json.load(f)
+    for verdict in data['verdicts']:
+        for scenario in verdict['scenarios']:
+            name = scenario['scenarioName']
+            bl = scenario['baseline']['metrics']
+            sk = scenario['skilledIsolated']['metrics']
+            print(f"--- {name} ---")
+            print(f"  Baseline: timedOut={bl['timedOut']}, output={len(bl.get('agentOutput',''))} chars")
+            print(f"  Skilled:  timedOut={sk['timedOut']}, output={len(sk.get('agentOutput',''))} chars")
+            print(f"  Improvement: {scenario.get('isolatedImprovementScore', 0):.1%}")
+            for a in bl.get('assertionResults', []):
+                status = 'PASS' if a['passed'] else 'FAIL'
+                print(f"  Baseline assertion [{status}]: {a['message']}")
+
+analyze('results.json')
+```
+
+## See also
+
+- [skill-validator README](README.md) — CLI usage, eval file format, scoring weights
+- [Overfitting detection](OverfittingDetection.md) — how overfitting scores are computed
+- [CONTRIBUTING.md](../../CONTRIBUTING.md) — writing eval files and running tests locally

eng/skill-validator/README.md

@@ -155,6 +155,8 @@ Results are displayed in the console with color-coded scores and metric deltas.
 - `junit` — `results.xml` with JUnit XML test results
 - `markdown` — `summary.md` with a results table, plus per-skill directories with per-scenario judge reports
 
+See [Investigating Results](InvestigatingResults.md) for how to diagnose poor scores, download artifacts, and interpret `results.json`.
+
 ### Consolidating results across matrix jobs
 
 When evaluating multiple plugins in parallel CI matrix jobs, use the `consolidate` subcommand to merge individual `results.json` files into a single markdown summary:

eng/skill-validator/src/Evaluate/Reporter.cs

@@ -609,6 +609,10 @@ public static string GenerateMarkdownSummary(
 
         sb.AppendLine($"\nModel: {model ?? "unknown"} | Judge: {judgeModel ?? "unknown"}");
 
+        bool anyFailure = verdicts.Any(v => !v.Passed);
+        if (anyFailure)
+            sb.AppendLine("\n> 📖 See [InvestigatingResults.md](https://github.com/dotnet/skills/blob/main/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures — or use the copy-paste prompt below.");
+
         return sb.ToString();
     }