dotnet-test: fix eval timeouts + reduce plugin skill-menu pressure for activation - #850
Conversation
Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout in the evaluation dashboard (skilled + plugin arms), producing empty output and failing all assertions. Increase per-scenario timeouts for the ones that timed out most often across recent runs: - writing-mstest-tests: 'Write unit tests for a service class', 'Write data-driven tests for a calculator', 'Use string assertions for format validation' 180->360; 'Use comparison assertions for boundary testing' 120->240 - coverage-analysis: 'Coverage plateau diagnosis', 'Project-wide coverage analysis with existing Cobertura data' 300->480 - test-gap-analysis: 'Decline request to write new tests from scratch' 120->300; 'Acknowledge well-tested code with few surviving mutations' 300->420 - run-tests: 'Filter xUnit v3 tests by trait on MTP', 'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240 Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the highest-priority failure mode; code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Skill Coverage Report
|
There was a problem hiding this comment.
Pull request overview
This PR adjusts evaluation configuration for the dotnet-test plugin by increasing per-scenario wall-clock timeout values in selected eval suites, targeting scenarios that frequently time out and therefore produce empty output and cascading assertion failures.
Changes:
- Increased timeouts for 4
writing-mstest-testsscenarios (up to 360s / 240s). - Increased timeouts for 2
coverage-analysisscenarios (to 480s). - Increased timeouts for 2
test-gap-analysisscenarios and 2run-testsscenarios (up to 420s / 300s / 240s).
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/writing-mstest-tests/eval.yaml | Raises timeouts for 4 MSTest-writing scenarios to reduce chronic wall-clock timeouts. |
| tests/dotnet-test/test-gap-analysis/eval.yaml | Raises timeouts for 2 gap-analysis scenarios to avoid empty-output runs. |
| tests/dotnet-test/run-tests/eval.yaml | Raises timeouts for 2 xUnit v3 filtering scenarios (MTP/query filter) to improve completion rates. |
| tests/dotnet-test/coverage-analysis/eval.yaml | Raises timeouts for 2 coverage-analysis scenarios to accommodate heavier analysis steps. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 0
|
/evaluate |
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Skill-activation investigation of the dotnet-test plugin arm (using the PR #850 eval session transcripts) showed the dominant failure mode is 'self-serve': in the crowded 16-skill plugin menu the model answers directly and loads no skill, rather than a sibling stealing activation or a budget cutoff. #800/#801/#835 already applied the standard remedies (trigger-phrasing, sibling deflection, DMI on the reference skills, additional_required_skills), so the remaining lever is reducing menu pressure. find-untested-sources is an agent-orchestrated helper: it is invoked by name from code-testing-researcher and the code-testing pipeline, is not listed in the plugin README's skill tables, and has no eval — yet it was still consuming the model-facing skill-menu budget and adding routing noise. Mark it disable-model-invocation: true (it stays invocable by name). This drops the plugin's rendered skill-menu cost from 14,981 to 14,261 of the 15,000-char budget (16 -> 15 model-invocable skills), giving the alphabetically-later user-facing test skills more headroom. skill-validator check passes (all 21 skills, 10 agents, 1 plugin). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Evangelink
changed the title
|
/evaluate |
Skill Validation Results
[1] (Plugin) Quality unchanged but weighted score is -2.1% due to: tokens (25247 → 34588) Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
|
✅ Evaluation passed for |
…ion assertions The 'Use proper collection assertions' scenario regressed hard in both eval arms (isolated -0.31, plugin -0.38) even though the skill activated and produced correct output. Root cause is a judge knowledge gap, not a skill bug: the pairwise judge (claude-opus-4.6) marked the skill's Assert.IsNotEmpty, Assert.HasCount, and Assert.ContainsSingle as 'fabricated / non-existent MSTest methods' and favored the baseline's Assert.AreEqual(3, x.Count) plus the genuinely-fake CollectionAssert.That.IsNotEmpty. Those three ARE real MSTest 3.8+ assertion APIs (Microsoft.Testing modern assertions; verified against Microsoft Learn and the MSTest 3.8 release notes), and are exactly what the writing-mstest-tests skill teaches — so the skill's output was more correct than the baseline it lost to. Clarify the scenario rubric to name these real MSTest 3.8+ methods as valid answers and add an explicit grading note that Assert.IsNotEmpty/IsEmpty/HasCount/ ContainsSingle/Contains/DoesNotContain must not be penalized as non-existent. This is the doc-recommended remedy for 'rubric penalizes valid alternatives' (eng/skill-validator/src/docs/InvestigatingResults.md, pattern #6); the skill content is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
pr-state/ready-for-eval
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
github-actions
Bot
added
waiting-on-review
…alue Three scenarios scored baseline 5.0/5 in the PR #850 eval run (results.json judgeResult.overallScore) — the model already aced them without the skill, so the skilled arm had no headroom and their NOT-ACTIVATED marks dragged the verdicts without reflecting real skill value. Per InvestigatingResults.md pattern #8 (baseline already good), make each harder around something the skill actually teaches, so the baseline struggles and the skill differentiates: - writing-mstest-tests / 'Fix swapped Assert.AreEqual arguments': the old prompt named the fix ('fix swapped Assert.AreEqual arguments') — an answer key — and had only two trivially-swapped assertions. Rephrase to describe the *symptom* (backwards Expected/Actual in failure messages) without naming the cause, and add a third already-correct assertion the model must leave untouched. Now tests unaided diagnosis + discrimination (an output_not_matches guard fails the run if it swaps the correct one). Skill teaches this (MSTEST0017). - run-tests / 'Filter MSTest tests by category on VSTest': old prompt wanted a bare 'TestCategory=Integration'. Require excluding the PaymentServiceTests class too, forcing an '&' AND-combinator with a negation operator ('!~'/'!=') — distinct from the existing AND-only 'Combine multiple filter criteria' scenario. filter-syntax teaches these operators. - run-tests / 'Run tests on a specific TFM with TRX in a multi-TFM MTP project (SDK 9)': also require a custom TRX filename, so the answer must place TWO MTP extension flags (--report-trx --report-trx-filename net9-results.trx) after the single '--' separator while keeping the built-in --framework before it — the exact before/after-'--' rule run-tests teaches for SDK 8/9. All new assertion regexes compile as .NET regex and were checked to match a correct answer (and, for the not-matches guard, to reject a wrongly-swapped one). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate |
|
/evaluate |
Skill Validation Results
[1] (Plugin) Quality unchanged but weighted score is -2.8% due to: tokens (25244 → 34580), time (11.0s → 15.2s)
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
What
Two categories of
dotnet-testeval issues surfaced on the evaluation dashboard: timeouts and skill-activation failures.1. Timeout fixes (verified: YAML valid)
Raised
timeoutfor the 10 scenarios that most frequently hit the wall-clock limit (empty output → all assertions fail):2. Skill-activation investigation + de-crowd
I analyzed the PR #850 eval session transcripts (
dotnet/skills-data) to see what the model actually did in the failing plugin-arm runs. Finding: the dominant failure is self-serve — in the crowded 16-skill plugin menu the model answers directly and loads no skill (not a sibling stealing activation, not budget truncation). Example —run-tests / Run tests in a VSTest MSTest project: isolated arm invokes theskilltool; plugin arm just reads the csproj and printsdotnet test.The standard remedies were already applied in #800/#801/#835 (natural-language trigger phrasing, sibling
DO NOT USEredirects,disable-model-invocationon the 5 reference skills,additional_required_skills, prompt observability). The remaining deterministic lever is reducing menu pressure:find-untested-sourcesdisable-model-invocation: true. It's an agent-orchestrated helper (invoked by name fromcode-testing-researcherand the pipeline), absent from the README skill tables, and has no eval — yet it was consuming skill-menu budget and adding routing noise. It stays invocable by name.skill-validator checkpasses (21 skills, 10 agents, 1 plugin).