Consolidate dotnet-test-frameworks into test-analysis-extensions - #851
Conversation
Remove the orphaned dotnet-test-frameworks reference skill, whose content was a duplicate subset of test-analysis-extensions/extensions/dotnet.md. Nothing actually loaded it. Redirect the remaining references and drop its eval tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate |
There was a problem hiding this comment.
Pull request overview
This PR removes the unused dotnet-test-frameworks reference skill from the dotnet-test plugin and consolidates the remaining reference guidance into the existing test-analysis-extensions .NET extension file, reclaiming skill budget without losing functionality.
Changes:
- Deleted the orphaned
dotnet-test-frameworksskill and its associated eval test suite. - Removed the now-dead cross-reference in
test-analysis-extensions/extensions/dotnet.mdand thedotnet-testplugin README. - Updated
exp-test-maintainabilitydocumentation to point attest-analysis-extensionsinstead of the deleted skill.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/dotnet-test-frameworks/eval.yaml | Removes eval scenarios for the deleted dotnet-test-frameworks skill. |
| tests/dotnet-test/dotnet-test-frameworks/eval.vally.yaml | Removes vally eval configuration for the deleted skill. |
| plugins/dotnet-test/skills/test-analysis-extensions/extensions/dotnet.md | Drops the obsolete “See also” note pointing to the removed skill. |
| plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md | Deletes the unused reference skill content. |
| plugins/dotnet-test/README.md | Removes the deleted skill from the plugin’s reference-data list. |
| plugins/dotnet-experimental/skills/exp-test-maintainability/SKILL.md | Updates guidance to reference test-analysis-extensions (but currently points to a different plugin’s skill). |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 6/6 changed files
- Comments generated: 1
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
Address PR review: exp-test-maintainability (dotnet-experimental plugin) must not point at test-analysis-extensions (dotnet-test plugin) — inline the .NET framework markers instead. Also strengthen the description/When-to-Use triggers so the 'each new case needs a whole new method / suggest a better structure' scenario reliably activates the skill instead of routing to a test-writing skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
…tained markers The description broadening regressed activation (3/4 -> 0/4 across a variance-dominated single eval run, CV up to 924%). Description is the activation lever, so restore the original wording and keep only the review-comment fix (inlined .NET framework markers in Step 1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate --runs 5 |
|
👋 @Evangelink — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
Skill Validation Results
[1] (Isolated) Quality unchanged but weighted score is -15.0% due to: judgment, quality Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a better structure', which the base model answers directly without loading the skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture pattern used by the activating scenarios (3 & 4): move the repetitive InputValidatorTests into a Validation.Tests fixture and ask the model to analyze the project. Mirror the change in eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate --runs 5 |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
|
github-actions
Bot
added
waiting-on-review
|
✅ Evaluation passed for |
Scenario 2 also pasted a self-contained snippet inline (well-maintained Auth.Tests) and asked to 'review these tests', which the base model handles without loading the skill. Convert to the project-fixture pattern used by the activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in eval.vally.yaml. All four scenarios now reference a project on disk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate --runs 5 |
All four scenarios scored NOT ACTIVATED / no-headroom (#8 in InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the skill only adds token/tool overhead with no offsetting quality gain. Match the convention of the sibling analysis skill test-anti-patterns: reject bash/edit/create on every scenario. This skill is analysis-only ('do not modify any files'), so the constraint is a no-op for correct behavior but levels the playing field between baseline and skilled runs — scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced overhead. Applied to both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate --runs 5 |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps |
…ministic - Import System.Security.Authentication for AuthenticationException - Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a deterministic one anchored on a fixed clock (FixedNow), and drive the injected FakeClock from that fixed time. This removes the flakiness and makes the FakeClock injection meaningful — a better exemplar for the 'well-maintained' scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/evaluate --runs 5 |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps |
Summary
Removes the orphaned
dotnet-test-frameworksreference skill from thedotnet-testplugin. Its content was a duplicate subset oftest-analysis-extensions/extensions/dotnet.md(the canonical superset, which additionally carries capability tags, trait attributes, and calibration notes), and a repo-wide search confirmed nothing actually loaded it — its own description claimed it was "loaded by test-anti-patterns," but that skill loadstest-analysis-extensions/extensions/dotnet.mdinstead.This reclaims skill budget with no functional loss. The plugin goes from 21 → 20 skills.
Changes
plugins/dotnet-test/skills/dotnet-test-frameworks/(the skill)tests/dotnet-test/dotnet-test-frameworks/(its eval tests, which targeted only that skill)plugins/dotnet-test/README.md— removed the reference-data table rowtest-analysis-extensions/extensions/dotnet.md— removed the now-dead "See also" notedotnet-experimental/.../exp-test-maintainability/SKILL.md— redirected its framework-markers pointer totest-analysis-extensions'dotnet.mdplugin.jsonneeded no change (it globs./skills/).