Add evaluation troubleshooting guide for AI agents - #441
Merged
Conversation
Add InvestigatingResults.md with: - How to find and download evaluation artifacts from workflow runs - Full results.json schema documentation - Common failure patterns (timeouts, empty output, variance, rubric issues, skill not activated) with diagnosis steps and fixes - Example analysis script for AI agents to parse results Link the doc from: - Reporter.cs: show a link in the PR comment when any skill fails - skill-validator README.md: Output section - CONTRIBUTING.md: CI evaluation section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
danmoseley
requested review from
JanKrivanek,
ViktorHofer,
dbreshears and
timheuer
as code owners
March 25, 2026 03:07
…docs - Reporter.cs: Use repo-relative link instead of hard-coded GitHub URL - evaluation.yml: Use github.sha instead of hard-coded 'main' branch - InvestigatingResults.md: Clarify skilledPlugin run is optional Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a troubleshooting guide and integrates it into the skill evaluation UX so PR authors (and AI agents) can more easily diagnose evaluation failures by downloading artifacts and interpreting results.json.
Changes:
- Add
eng/skill-validator/InvestigatingResults.mddocumentingresults.json, scoring, and common failure patterns. - Link the guide from
eng/skill-validator/README.mdandCONTRIBUTING.md. - Update evaluation reporting to include a guide link and (on failures) an AI-agent copy/paste investigation prompt in the PR comment.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| eng/skill-validator/src/Evaluate/Reporter.cs | Adds a failure-only note linking to the troubleshooting guide in the generated markdown summary. |
| eng/skill-validator/README.md | Links to the new troubleshooting guide from the Output section. |
| eng/skill-validator/InvestigatingResults.md | New guide explaining artifact retrieval, results.json schema, scoring, and common failure patterns. |
| CONTRIBUTING.md | Links to the troubleshooting guide from the CI evaluation section. |
| .github/workflows/evaluation.yml | Adds an investigation prompt to the consolidated PR comment when failures are detected. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
danmoseley
added
the
DO NOT MERGE
This change is intentionally trivial and will be reverted before merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Do not merge yet -- making a temp skill edit so we can try it on this PR |
- Reporter.cs: Remove leading / from relative link - evaluation.yml: Use jq on results.json instead of grepping for emoji - evaluation.yml: Use needs.gate.outputs.head_sha for revision-pinned URL - InvestigatingResults.md: Mark pluginImprovementScore/pluginBreakdown as optional - InvestigatingResults.md: Fix skill-not-activated symptoms to use skillActivation fields instead of turnCount Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
/evaluate |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Use skillActivationIsolated/skillActivationPlugin instead of generic skillActivation - Clarify timestamp is when results were written, not run start Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Skill Validation Results
[1] (Isolated) Quality unchanged but weighted score is -6.6% due to: tokens (24684 → 49262), tool calls (2 → 3)
Model: claude-opus-4.6 | Judge: claude-opus-4.6
|
This reverts commit 83bd7f7.
Contributor
Author
|
@ViktorHofer or @JanKrivanek can you review? I realized I can't validate much here because it always uses the validator that's in main. I think we have to review and merge this and iterate if we need to. If this works I expect it will make PR's materially more actionable. I want to drain our PR backlog. |
ViktorHofer
approved these changes
Mar 25, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
danmoseley
enabled auto-merge (squash)
March 25, 2026 15:04
Member
|
/evaluate |
Contributor
Skill Validation Results
[1] (Isolated) Quality unchanged but weighted score is -18.2% due to: judgment, tokens (24225 → 49401), tool calls (2 → 3)
Model: claude-opus-4.6 | Judge: claude-opus-4.6
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add evaluation troubleshooting guide for AI agents
Note
This PR was generated with Copilot assistance.
Replaces #440 (moved from fork to upstream branch so CI runs with correct permissions).
Problem
When skill evaluation produces failures, PR authors have no clear path to diagnose the cause. The eval results table shows scores and verdicts, but doesn't explain what went wrong or how to fix it. Investigating requires manually downloading artifacts from workflow logs, reverse-engineering the
results.jsonschema, and pattern-matching against undocumented failure modes.Solution
Add
eng/skill-validator/InvestigatingResults.md— a troubleshooting guide designed primarily for AI coding agents to read and follow. When any skill fails evaluation, the PR comment now includes a ready-to-use prompt that the PR author can copy-paste to their AI agent. The agent downloads the artifacts, reads the guide, analyzes the results, and suggests fixes.What's in the guide
results.jsonschema documentation — verdict, scenario, breakdown, and run metrics fields with descriptionsgh run downloadinstructions for artifact retrievalChanges
eng/skill-validator/InvestigatingResults.mdeng/skill-validator/README.mdCONTRIBUTING.mdeng/skill-validator/src/Evaluate/Reporter.cs.github/workflows/evaluation.ymlValidation
Tested the guide by launching 4 fresh AI agents (clean context, no prior analysis) against eval artifacts from 2 different PRs (PR #317 and PR #428). Across 19 failing scenarios: