dotnet-test: fix writing-mstest-tests eval timeout and skill activation - #863
Conversation
…enario The 'Use comparison assertions for boundary testing' scenario chronically hit its 240s wall-clock timeout. Raise it to 360s to match the other code-generation scenarios in this eval file, per InvestigatingResults.md timeout guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate |
There was a problem hiding this comment.
Pull request overview
Raises the per-scenario evaluation timeout for a flaky/slow-running MSTest code-generation scenario in the dotnet-test plugin’s writing-mstest-tests test suite to reduce wall-clock timeouts during evaluation runs.
Changes:
- Increased the
timeoutfor the “Use comparison assertions for boundary testing” scenario from 240s to 360s.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/writing-mstest-tests/eval.yaml | Bumps a single scenario timeout to 360s to better accommodate slower code-generation runs and avoid assertion failures caused by timeouts. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 1/1 changed files
- Comments generated: 0
Skill Validation Results
[1] (Plugin) Quality unchanged but weighted score is -2.3% due to: tokens (12838 → 17538) Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
…ed-args scenarios Three writing-mstest-tests eval scenarios showed plugin-arm skill non-activation because their audit/review-style prompts routed to sibling skills (test-anti-patterns, assertion-quality). Tune the frontmatter description (budget-neutral, still <=1024 chars) to claim the scenario keywords: add 'review' (Modernize legacy test patterns) and 'swapped/reversed Assert.AreEqual args (Expected/Actual backwards)' matching the swapped-args prompt's symptom wording, while keeping every assertion keyword the other scenarios rely on. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Evangelink
changed the title
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
✅ Evaluation passed for |
Summary
Two fixes for the
dotnet-test/writing-mstest-testseval.1. Timeout (
eval.yaml)The "Use comparison assertions for boundary testing" scenario chronically hit its 240s wall-clock timeout. Raised
timeout: 240→360, matching the other code-generation scenarios in this file and followingeng/skill-validator/src/docs/InvestigatingResults.mdguidance.2. Skill activation (
SKILL.mddescription)Three scenarios showed plugin-arm skill non-activation:
Root cause is sibling-skill competition, not menu-budget truncation — the plugin arm loads only
dotnet-test(13.3k < 15k char budget), so descriptions aren't cut. The prompts use audit/review phrasing ("review these tests/assertions") that routes the model totest-anti-patterns/assertion-qualitywhen all 20 skills are present.Fix: tuned the frontmatter
description(perInvestigatingResults.md§5) to claim the scenario keywords — added "review" (Modernize scenario) and "swapped/reversed Assert.AreEqual args (Expected/Actual backwards)" matching the swapped-args prompt's symptom wording. Change is budget-neutral (still ≤ 1,024 chars, validated bycheck) and keeps every assertion keyword the other passing scenarios depend on.Validation
skill-validator check --plugin plugins/dotnet-test→ ✅ All checks passed (20 skills, 10 agents, 1 plugin).