Improve dotnet-test eval coverage and efficiency - #917
Conversation
Address remaining high-confidence items from dotnet#899 by bounding the code-testing pipeline and adding eval coverage for grade-tests and find-untested-sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046
|
/evaluate |
There was a problem hiding this comment.
Pull request overview
This PR improves the dotnet-test plugin’s evaluation coverage and reduces agent/runtime cost by tightening the code-testing-agent RPI pipeline to use bounded discovery and reusable artifacts, and by adding new eval scenarios for find-untested-sources and cross-family (Vally) coverage for grade-tests.
Changes:
- Added new
eval.yaml+eval.vally.yamlcoverage forfind-untested-sources(including fixtures). - Added Vally coverage for
grade-tests. - Updated
code-testing-agentand its sub-agents to bound scope, avoid whole-repo inventories, reuse.testagent/*artifacts, and load deep examples only when needed.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/grade-tests/eval.vally.yaml | Adds Vally capability eval coverage for grade-tests. |
| tests/dotnet-test/find-untested-sources/eval.yaml | Adds skill-validator eval scenarios for find-untested-sources (C# pairing + non-C# rejection). |
| tests/dotnet-test/find-untested-sources/eval.vally.yaml | Adds Vally capability eval scenario for find-untested-sources. |
| tests/dotnet-test/find-untested-sources/fixtures/pairing-repo/tests/Store.Tests/Store.Tests.csproj | Adds test-project fixture for pairing analysis. |
| tests/dotnet-test/find-untested-sources/fixtures/pairing-repo/tests/Store.Tests/CustomerServiceTests.cs | Adds paired test fixture to ensure one source is detected as “tested”. |
| tests/dotnet-test/find-untested-sources/fixtures/pairing-repo/src/Store/Store.csproj | Adds production-project fixture for pairing analysis. |
| tests/dotnet-test/find-untested-sources/fixtures/pairing-repo/src/Store/OrderProcessor.cs | Adds intentionally-unpaired source fixture to be detected as “untested”. |
| tests/dotnet-test/find-untested-sources/fixtures/pairing-repo/src/Store/CustomerService.cs | Adds paired production source fixture. |
| plugins/dotnet-test/skills/code-testing-agent/SKILL.md | Updates the user-facing pipeline docs to emphasize bounded context and reuse of pairing artifacts. |
| plugins/dotnet-test/agents/code-testing-researcher.agent.md | Refactors researcher behavior toward bounded scope + cheaper discovery paths. |
| plugins/dotnet-test/agents/code-testing-planner.agent.md | Refines planner guidance to rely on research artifacts and avoid embedding templates. |
| plugins/dotnet-test/agents/code-testing-implementer.agent.md | Refines implementer guidance to read only what’s needed per phase and avoid redundant reads. |
| plugins/dotnet-test/agents/code-testing-generator.agent.md | Updates generator guidance to minimize extension/example loading and avoid rescanning workspaces. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 13/13 changed files
- Comments generated: 4
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98
|
/evaluate |
|
👋 @Evangelink — this PR has 5 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98
|
/evaluate |
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
plugins/dotnet-test/skills/find-untested-sources/SKILL.md:11
find-untested-sourcesno longer setsuser-invocable: false/disable-model-invocation: true, so it will now appear in the model-facing skill menu and be eligible for direct routing. That’s a behavioral change compared to other internal helper/reference skills in this plugin (e.g.,test-analysis-extensions,code-testing-extensions,filter-syntax,platform-detection) which are explicitly hidden to reduce routing noise and skill-menu budget pressure. If this skill is still primarily an internal pipeline helper, consider restoring the hidden/internal metadata; if it’s intentionally user-facing now, consider whether the broadened “MANDATORY” description could cause unwanted activations and keep the front-matter metadata consistent with the intended exposure.
---
name: find-untested-sources
description: >
MANDATORY for static requests to find, identify, or list untested source files
or modules, sources without tests, source-to-test pairing, test-gap worklists,
or suggested test locations. Invoke even for a tiny package; do not substitute
manual globbing. Uses Roslyn for C#/.NET and tree-sitter for Python, TS/JS, Go,
Java, Rust, and Ruby. DO NOT USE FOR: line/branch coverage, CRAP risk, or
grading existing tests.
license: MIT
---
- Files reviewed: 18/18 changed files
- Comments generated: 0 new
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Preserve the bounded test-generation and activation changes while taking the latest delegation syntax and fixture namespace imports from main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98
AbhitejJohn
left a comment
AbhitejJohn
left a comment
There was a problem hiding this comment.
Multi-model review (GPT-5.5 + Gemini 3.1 Pro + manual verification). Substance looks good; two items are worth resolving before merge. Details inline.
- Skill-menu budget: this PR takes dotnet-test from 14,671 to 14,943 of the 15,000 rendered-menu cap (57 chars headroom). Please rerun the skill-validator on a clean worktree since the PR notes it couldn't be verified on the final state.
- Polyglot vs. C#-only callers: the re-enabled polyglot description isn't matched by the orchestrator instructions, which still restrict
find-untested-sourcesto C#.
Verified clean: regex escaping in both eval formats, Python suggested_test_path (test_pricing.py), no dangling refs from the removed Examples section, generator user-invocable: false consistent with README, and additional_required_skills usage in grade-tests.
Reconcile dotnet#917's dotnet-test eval configs into the sole eval.yaml convention. main completed the Vally harness migration (single eval.yaml per skill, zero eval.vally.yaml). This branch still carried interim eval.vally.yaml files and old-schema eval.yaml edits, causing conflicts in three dotnet-test skills. Resolution (honoring both the PR's intent and main's invariant): - code-testing-agent: kept main's canonical eval.yaml plus the PR's prompt additions (standard test-generation workflow lines) via clean auto-merge. - find-untested-sources: adopted main's canonical eval.yaml and folded in the PR's tightened OrderProcessorTests path grader; deleted eval.vally.yaml. - grade-tests: kept main's redesigned MSTest + no-list scenarios and re-authored the PR's three polyglot coverage scenarios (pytest, Go table-driven, production-unavailable) in the canonical Vally schema. All eval.vally.yaml files removed repo-wide (invariant preserved); YAML validated and markdownlint clean on changed docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/evaluate |
📊 Skill Evaluation Results3 skill(s) evaluated — 2 improved, 1 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
/evaluate |
📊 Skill Evaluation Results3 skill(s) evaluated — 2 improved, 1 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps
|
|
/evaluate |
📊 Skill Evaluation Results3 skill(s) evaluated — 1 improved, 2 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps
|
|
👋 @Evangelink — this PR has 6 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
📊 Skill Evaluation Results3 skill(s) evaluated — 1 improved, 2 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Require auditable requirement evidence from generation runs and remove the disambiguation hint from the added Roslyn scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 78c33fd6-c8b7-4053-8257-1845129f79ff
|
/evaluate |
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
tests/dotnet-test/grade-tests/eval.yaml:71
- This scenario now relies on the “production code unavailable” policy (since the fixture omits the production module), but the objective graders don’t currently require the
Production-dependent behavior coverage: Unverifieddisclaimer. Adding a concrete check makes the eval more robust and keeps it aligned with the skill’s contract.
- Files reviewed: 67/67 changed files
- Comments generated: 1
📊 Skill Evaluation Results3 skill(s) evaluated — 2 improved, 1 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
State that production code is unavailable and require the standard Unverified disclaimer for production-dependent behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 78c33fd6-c8b7-4053-8257-1845129f79ff
|
/evaluate |
📊 Skill Evaluation Results3 skill(s) evaluated — 1 improved, 2 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
/evaluate |
📊 Skill Evaluation Results3 skill(s) evaluated — 3 improved, 0 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Summary
Addresses remaining high-confidence work from #899:
eval.yamlcoverage for code-testing-agent, find-untested-sources, and grade-testsThe four remaining coverage gaps are non-user-invocable reference skills and need dependency-level evaluation design rather than direct-activation tests.
Checks
skill-validator check --plugin ./plugins/dotnet-testpasses for all 20 skills and 10 agentsRefs #899