Fix writing-mstest-tests activation by deflecting sibling skills

[Evangelink]

Problem

Six writing-mstest-tests eval scenarios were failing with skill-activation issues in the plugin arm:

• Write unit tests for a service class
• Fix swapped Assert.AreEqual arguments
• Modernize legacy test patterns
• Use proper type assertions instead of casts
• Use DynamicData with ValueTuples over object arrays
• Use string assertions for format validation

Root cause

This is not a skill-menu budget-truncation problem. The dotnet-test plugin's rendered model-invocable menu is ~14.7K chars (< the 15,000 SKILL_CHAR_BUDGET), so writing-mstest-tests keeps its full description in the plugin menu, and that description already contains the relevant keywords.

The real cause is routing competition: these MSTest-specific prompts also match sibling skills, and the router was selecting the sibling instead — so writing-mstest-tests specifically wasn't in detectedSkills, yielding activated: false for the plugin arm.

Two model-invocable siblings matched but had no explicit deflection:

• assertion-quality — ended with a vague fixing assertions, capturing the type/string-assertion scenarios.
• test-anti-patterns — broad "audit/review tests, swapped assertions" with no MSTest carve-out, capturing the swapped-Assert.AreEqual and modernize-legacy scenarios.

(code-testing-agent already defers MSTest writing/attributes to writing-mstest-tests.)

Fix

Add explicit DO NOT USE FOR … (use writing-mstest-tests) deflections to both siblings' frontmatter descriptions:

• assertion-quality: replaced fixing assertions with writing, fixing, or modernizing MSTest tests, assertions, or attributes (use writing-mstest-tests).
• test-anti-patterns: added fixing or modernizing MSTest tests, assertions, attributes, or lifecycle (use writing-mstest-tests).

Both siblings' own eval suites are audit/evaluate-only (assertion-quality even has a "Decline request to write new tests" scenario), so these deflections reinforce rather than conflict with their existing coverage.

Validation

• skill-validator check --plugin plugins/dotnet-test passes ✅ — aggregate rendered menu 14,907 / 15,000; both edited descriptions remain under the 1,024-char cap.
• The live LLM eval was not runnable in this environment (no GITHUB_TOKEN); validation was done deterministically via check and budget math.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	dotnet-test	assertion-quality	22/22	100%
✅	dotnet-test	test-anti-patterns	21/21	100%

[Copilot]

Pull request overview

This PR updates dotnet-test plugin skill metadata to reduce routing competition that prevents writing-mstest-tests from activating in MSTest-focused scenarios.

Changes:

• Added an explicit MSTest deflection to test-anti-patterns frontmatter to route MSTest fix/modernization requests to writing-mstest-tests.
• Updated assertion-quality frontmatter to deflect MSTest writing/fixing/modernization requests to writing-mstest-tests.

Show a summary per file

File	Description
plugins/dotnet-test/skills/test-anti-patterns/SKILL.md	Adds MSTest-specific “DO NOT USE FOR” routing deflection in the skill frontmatter description.
plugins/dotnet-test/skills/assertion-quality/SKILL.md	Replaces/extends the frontmatter deflection text to steer MSTest writing/fixing/modernization away from this skill.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 2/2 changed files
• Comments generated: 2

[Evangelink]

/evaluate

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
test-anti-patterns	Detect mixed severity anti-patterns in repository service tests	5.0/5 → 5.0/5	✅ test-anti-patterns; tools: skill	✅ 0.20	❌ [1]
test-anti-patterns	Detect flakiness indicators and test coupling	5.0/5 → 5.0/5	✅ test-anti-patterns; tools: skill	✅ 0.20	❌ [2]
test-anti-patterns	Detect duplicated tests and magic values	4.0/5 → 5.0/5 🟢	✅ test-anti-patterns; tools: skill	✅ 0.20	❌ [3]
test-anti-patterns	Recognize well-written tests without inventing false positives	4.3/5 → 4.0/5 🔴	✅ test-anti-patterns; tools: skill	✅ 0.20	❌ [4]
test-anti-patterns	Detect coverage-touching pattern across a service facade	5.0/5 → 5.0/5	✅ test-anti-patterns; tools: skill	✅ 0.20	❌ [5]
test-anti-patterns	Detect self-referential assertions in round-trip and identity tests	4.3/5 → 5.0/5 🟢	✅ test-anti-patterns; tools: skill	✅ 0.20	✅ [6]
test-anti-patterns	Polyglot: detect anti-patterns in a Python pytest suite	4.7/5 → 5.0/5 🟢	⚠️ NOT ACTIVATED	✅ 0.20	❌ [7]
assertion-quality	Identify low assertion diversity in equality-dominated test suite	3.0/5 → 5.0/5 🟢	✅ assertion-quality; tools: skill	🟡 0.25	✅
assertion-quality	Flag assertion-free tests and trivial-only assertions	4.0/5 → 4.0/5	✅ assertion-quality; tools: skill, glob	🟡 0.25	❌ [8]
assertion-quality	Recognize well-diversified assertion usage	3.7/5 → 5.0/5 🟢	✅ assertion-quality; tools: skill, glob / ✅ assertion-quality; tools: skill	🟡 0.25	✅ [9]
assertion-quality	Identify self-referential assertions in identity and round-trip tests	3.7/5 → 4.0/5 🟢	✅ assertion-quality; tools: skill, glob	🟡 0.25	✅ [10]
assertion-quality	Decline request to write new tests from scratch	4.0/5 → 4.0/5	ℹ️ not activated (expected)	🟡 0.25	❌ [11]
assertion-quality	Polyglot: evaluate shallow assertions in a Jest/TypeScript OrderService suite	5.0/5 → 5.0/5	✅ assertion-quality; tools: skill, glob / ✅ assertion-quality; tools: skill, glob, bash, stop_bash	🟡 0.25	❌ [12]

[1] ⚠️ High run-to-run variance (CV=2922%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -6.2% due to: tokens (27684 → 56782), time (26.2s → 35.3s), tool calls (3 → 4)
[2] ⚠️ High run-to-run variance (CV=63%) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=285%) — consider re-running with --runs 5. (Isolated) Quality improved but weighted score is -12.8% due to: judgment, tokens (40292 → 71405), time (30.1s → 41.5s), tool calls (3 → 4)
[4] ⚠️ High run-to-run variance (CV=103%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -3.2% due to: tokens (26399 → 65150), tool calls (2 → 3), time (19.5s → 27.2s)
[5] ⚠️ High run-to-run variance (CV=210%) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -8.8% due to: tokens (28337 → 53529), quality, tool calls (3 → 4)
[6] ⚠️ High run-to-run variance (CV=102%) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=235%) — consider re-running with --runs 5. (Isolated) Quality improved but weighted score is -1.5% due to: quality
[8] (Plugin) Quality unchanged but weighted score is -7.8% due to: tokens (26490 → 92506), tool calls (2 → 5), time (21.1s → 42.8s)
[9] ⚠️ High run-to-run variance (CV=68%) — consider re-running with --runs 5
[10] ⚠️ High run-to-run variance (CV=195%) — consider re-running with --runs 5
[11] ⚠️ High run-to-run variance (CV=207%) — consider re-running with --runs 5
[12] (Plugin) Quality unchanged but weighted score is -23.8% due to: completion (✓ → ✗), tokens (27138 → 127486), tool calls (3 → 7), time (22.6s → 57.8s)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 835 in dotnet/skills, download eval artifacts with gh run download 28229176797 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/b9982a6b391d1343623eb8d4bbc6b1d1e1b024bb/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions