Skip to content

Fix test-anti-patterns plugin activation by disambiguating competing descriptions - #552

Merged
Evangelink merged 7 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/anti-patterns
Apr 21, 2026
Merged

Fix test-anti-patterns plugin activation by disambiguating competing descriptions#552
Evangelink merged 7 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/anti-patterns

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Problem

The test-anti-patterns skill was intermittently failing plugin activation checks on CI for these scenarios:

  • Detect mixed severity anti-patterns in repository service tests
  • Detect flakiness indicators and test coupling
  • Detect duplicated tests and magic values

In the plugin run (all skills + all agents loaded), the model sometimes chose competing targets instead of test-anti-patterns:

  1. The test-quality-auditor agent description ("audit a test suite, find weak tests, check test health") overlapped with test review prompts
  2. The writing-mstest-tests skill description included "review MSTest tests" and "common anti-patterns"

Fix

Three description changes to disambiguate:

  1. test-quality-auditor.agent.md — Narrowed to "multi-skill audit pipelines for comprehensive workspace-level assessment". Added explicit "Do NOT use for reviewing a single test file, class, or inline code snippet."

  2. writing-mstest-tests/SKILL.md — Removed "review" and "anti-patterns" from description. Added explicit "Do NOT use for reviewing or auditing existing test quality — use test-anti-patterns instead."

  3. test-anti-patterns/SKILL.md — Expanded trigger phrases to better match eval prompts: "investigate flaky tests, find test repetition or duplication, tests pass but don't actually verify anything." Listed more anti-pattern types in the description.

Validation

  • check --plugin ./plugins/dotnet-test passes
  • Before fix: 1-3 of 4 scenarios failed plugin activation (flaky)
  • After fix: 4/4 scenarios pass with full activation in both isolated and plugin mode (exit code 0)

…descriptions

The test-anti-patterns skill was not reliably activated in plugin mode
because sibling descriptions overlapped:

- test-quality-auditor agent: narrowed to multi-skill audit pipelines
  only, explicitly excluding single-file reviews
- writing-mstest-tests skill: removed 'review' and 'anti-patterns' from
  description, added explicit exclusion for test quality auditing
- test-anti-patterns skill: expanded trigger phrases to better match
  review prompts (flaky tests, duplication, tests that pass without
  verifying)
Copilot AI review requested due to automatic review settings April 20, 2026 10:39
@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

@Evangelink
Evangelink enabled auto-merge (squash) April 20, 2026 10:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates skill/agent metadata in the dotnet-test plugin to reduce intermittent plugin activation flakiness by disambiguating overlapping descriptions, ensuring prompts route to test-anti-patterns instead of competing targets.

Changes:

  • Narrowed test-quality-auditor agent description to emphasize multi-skill, workspace-level audit pipelines (and explicitly exclude single-file/snippet reviews).
  • Refocused writing-mstest-tests skill description on authoring/implementing MSTest tests and explicitly deferred anti-pattern auditing to test-anti-patterns.
  • Expanded test-anti-patterns skill description with additional trigger phrases and anti-pattern types to better match eval prompts.
Show a summary per file
File Description
plugins/dotnet-test/skills/writing-mstest-tests/SKILL.md Updates skill frontmatter description to avoid competing with anti-pattern review routing.
plugins/dotnet-test/skills/test-anti-patterns/SKILL.md Broadens frontmatter description to better capture anti-pattern/flakiness/duplication prompts.
plugins/dotnet-test/agents/test-quality-auditor.agent.md Tightens agent description to broad, multi-skill suite audits (not single-file reviews).

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 3/3 changed files
  • Comments generated: 1

Comment thread plugins/dotnet-test/skills/writing-mstest-tests/SKILL.md Outdated
github-actions Bot added a commit that referenced this pull request Apr 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
test-anti-patterns Detect mixed severity anti-patterns in repository service tests 5.0/5 → 5.0/5 ✅ test-anti-patterns; tools: skill, report_intent / ⚠️ NOT ACTIVATED ✅ 0.08 [1]
test-anti-patterns Detect flakiness indicators and test coupling 3.0/5 → 4.0/5 🟢 ✅ test-anti-patterns; tools: skill, report_intent / ⚠️ NOT ACTIVATED ✅ 0.08 [2]
test-anti-patterns Detect duplicated tests and magic values 3.0/5 → 4.7/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.08
test-anti-patterns Recognize well-written tests without inventing false positives 2.3/5 → 5.0/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill / ✅ test-anti-patterns; tools: skill, report_intent ✅ 0.08
writing-mstest-tests Write unit tests for a service class 4.0/5 → 4.0/5 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; code-testing-agent; tools: skill, task 🟡 0.24 [3]
writing-mstest-tests Write data-driven tests for a calculator 5.0/5 → 4.7/5 🔴 ✅ writing-mstest-tests; tools: skill 🟡 0.24 [4]
writing-mstest-tests Write async tests with cancellation 2.7/5 → 5.0/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.24
writing-mstest-tests Fix swapped Assert.AreEqual arguments 5.0/5 → 5.0/5 ✅ writing-mstest-tests; tools: report_intent, skill / ⚠️ NOT ACTIVATED 🟡 0.24 [5]
writing-mstest-tests Modernize legacy test patterns 4.3/5 → 4.7/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.24 [6]
writing-mstest-tests Replace ExpectedException with Assert.Throws 3.0/5 → 4.0/5 🟢 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.24
writing-mstest-tests Use proper collection assertions 3.0/5 → 2.0/5 🔴 ✅ writing-mstest-tests; tools: report_intent, skill / ✅ writing-mstest-tests; tools: skill 🟡 0.24 [7]
writing-mstest-tests Use proper type assertions instead of casts 2.3/5 → 3.3/5 🟢 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.24 [8]
writing-mstest-tests Set up test lifecycle correctly 2.7/5 → 4.3/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.24
writing-mstest-tests Use DynamicData with ValueTuples over object arrays 2.0/5 → 3.0/5 🟢 ⚠️ NOT ACTIVATED 🟡 0.24 [9]

[1] ⚠️ High run-to-run variance (CV=4.30) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=1.11) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=1.69) — consider re-running with --runs 5
[4] ⚠️ High run-to-run variance (CV=1.95) — consider re-running with --runs 5
[5] ⚠️ High run-to-run variance (CV=0.96) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -2.3% due to: tokens (12408 → 17131)
[6] ⚠️ High run-to-run variance (CV=8.46) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=0.71) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=1.10) — consider re-running with --runs 5
[9] ⚠️ High run-to-run variance (CV=0.98) — consider re-running with --runs 5

timeout — run(s) hit the (180s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions

@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

github-actions Bot added a commit that referenced this pull request Apr 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
test-anti-patterns Detect mixed severity anti-patterns in repository service tests 5.0/5 → 5.0/5 ✅ test-anti-patterns; tools: skill, report_intent, bash / ⚠️ NOT ACTIVATED ✅ 0.06 [1]
test-anti-patterns Detect flakiness indicators and test coupling 3.0/5 → 4.7/5 🟢 ✅ test-anti-patterns; tools: skill, report_intent / ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.06
test-anti-patterns Detect duplicated tests and magic values 3.7/5 → 5.0/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.06
test-anti-patterns Recognize well-written tests without inventing false positives 2.3/5 → 5.0/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.06
writing-mstest-tests Write unit tests for a service class 4.0/5 → 4.0/5 ✅ writing-mstest-tests; tools: skill / ✅ code-testing-agent; writing-mstest-tests; tools: skill, task, read_agent 🟡 0.26 [2]
writing-mstest-tests Write data-driven tests for a calculator 5.0/5 → 3.7/5 ⏰ 🔴 ✅ writing-mstest-tests; tools: skill 🟡 0.26 [3]
writing-mstest-tests Write async tests with cancellation 2.7/5 → 5.0/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.26
writing-mstest-tests Fix swapped Assert.AreEqual arguments 5.0/5 → 5.0/5 ✅ writing-mstest-tests; tools: skill, report_intent / ⚠️ NOT ACTIVATED 🟡 0.26 [4]
writing-mstest-tests Modernize legacy test patterns 4.0/5 → 5.0/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.26
writing-mstest-tests Replace ExpectedException with Assert.Throws 3.0/5 → 4.0/5 🟢 ✅ writing-mstest-tests; tools: skill, report_intent / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.26 [5]
writing-mstest-tests Use proper collection assertions 3.0/5 → 3.0/5 ✅ writing-mstest-tests; tools: report_intent, skill / ✅ writing-mstest-tests; tools: skill 🟡 0.26 [6]
writing-mstest-tests Use proper type assertions instead of casts 3.0/5 → 3.3/5 🟢 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.26 [7]
writing-mstest-tests Set up test lifecycle correctly 3.0/5 → 4.3/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.26
writing-mstest-tests Use DynamicData with ValueTuples over object arrays 2.7/5 → 3.0/5 🟢 ⚠️ NOT ACTIVATED / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.26 [8]

[1] ⚠️ High run-to-run variance (CV=38.58) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -3.3% due to: tokens (21133 → 36832)
[2] ⚠️ High run-to-run variance (CV=0.92) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=0.69) — consider re-running with --runs 5
[4] ⚠️ High run-to-run variance (CV=2.60) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -0.0% due to: tokens (12393 → 17097)
[5] ⚠️ High run-to-run variance (CV=0.60) — consider re-running with --runs 5
[6] ⚠️ High run-to-run variance (CV=0.69) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -17.0% due to: judgment, tokens (12635 → 28929), tool calls (0 → 2)
[7] ⚠️ High run-to-run variance (CV=6.70) — consider re-running with --runs 5. (Isolated) Quality improved but weighted score is -22.5% due to: judgment, quality, tokens (12433 → 17777)
[8] ⚠️ High run-to-run variance (CV=1.09) — consider re-running with --runs 5

timeout — run(s) hit the (180s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions

Copilot AI review requested due to automatic review settings April 20, 2026 12:46
@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 3/3 changed files
  • Comments generated: 2

Comment thread plugins/dotnet-test/skills/writing-mstest-tests/SKILL.md Outdated
Comment thread plugins/dotnet-test/skills/test-anti-patterns/SKILL.md Outdated
github-actions Bot added a commit that referenced this pull request Apr 20, 2026
github-actions Bot added a commit that referenced this pull request Apr 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
test-anti-patterns Detect mixed severity anti-patterns in repository service tests 5.0/5 → 5.0/5 ✅ test-anti-patterns; tools: skill, report_intent / ⚠️ NOT ACTIVATED ✅ 0.08 [1]
test-anti-patterns Detect flakiness indicators and test coupling 2.7/5 → 5.0/5 🟢 ✅ test-anti-patterns; tools: skill, report_intent / ⚠️ NOT ACTIVATED ✅ 0.08
test-anti-patterns Detect duplicated tests and magic values 3.3/5 → 5.0/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.08 [2]
test-anti-patterns Recognize well-written tests without inventing false positives 2.0/5 → 4.7/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.08
writing-mstest-tests Write unit tests for a service class 4.0/5 → 4.0/5 ✅ writing-mstest-tests; tools: skill, glob / ✅ code-testing-agent; writing-mstest-tests; tools: skill, task, read_agent 🟡 0.25 [3]
writing-mstest-tests Write data-driven tests for a calculator 5.0/5 → 4.3/5 🔴 ✅ writing-mstest-tests; tools: skill 🟡 0.25 [4]
writing-mstest-tests Write async tests with cancellation 2.3/5 → 5.0/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.25
writing-mstest-tests Fix swapped Assert.AreEqual arguments 5.0/5 → 5.0/5 ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.25 [5]
writing-mstest-tests Modernize legacy test patterns 4.0/5 → 4.3/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.25
writing-mstest-tests Replace ExpectedException with Assert.Throws 3.0/5 → 5.0/5 🟢 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.25
writing-mstest-tests Use proper collection assertions 3.3/5 → 2.0/5 🔴 ✅ writing-mstest-tests; tools: skill 🟡 0.25 [6]
writing-mstest-tests Use proper type assertions instead of casts 2.3/5 → 3.0/5 🟢 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.25 [7]
writing-mstest-tests Set up test lifecycle correctly 3.0/5 → 4.3/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.25
writing-mstest-tests Use DynamicData with ValueTuples over object arrays 2.3/5 → 4.0/5 🟢 ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.25 [8]

[1] ⚠️ High run-to-run variance (CV=180.48) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=0.52) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=1.50) — consider re-running with --runs 5
[4] ⚠️ High run-to-run variance (CV=7.38) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -2.8% due to: tokens (168361 → 374155), tool calls (18 → 24), time (105.1s → 136.1s)
[5] ⚠️ High run-to-run variance (CV=0.60) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -7.3% due to: tokens (12419 → 31373), tool calls (0 → 1), time (13.1s → 20.3s)
[6] ⚠️ High run-to-run variance (CV=2.18) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=0.50) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=0.69) — consider re-running with --runs 5

timeout — run(s) hit the (180s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions

Copilot AI review requested due to automatic review settings April 20, 2026 16:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 3/3 changed files
  • Comments generated: 2

Comment thread plugins/dotnet-test/skills/writing-mstest-tests/SKILL.md
Comment thread plugins/dotnet-test/skills/test-anti-patterns/SKILL.md Outdated
@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

github-actions Bot added a commit that referenced this pull request Apr 20, 2026
github-actions Bot added a commit that referenced this pull request Apr 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
test-anti-patterns Detect mixed severity anti-patterns in repository service tests 5.0/5 → 5.0/5 ✅ test-anti-patterns; tools: skill, report_intent / ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.07 [1]
test-anti-patterns Detect flakiness indicators and test coupling 3.0/5 → 4.3/5 🟢 ✅ test-anti-patterns; tools: skill, report_intent / ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.07 [2]
test-anti-patterns Detect duplicated tests and magic values 3.7/5 → 5.0/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill / ✅ test-anti-patterns; tools: skill, report_intent ✅ 0.07
test-anti-patterns Recognize well-written tests without inventing false positives 2.0/5 → 5.0/5 🟢 ✅ test-anti-patterns; tools: report_intent, skill ✅ 0.07
writing-mstest-tests Write unit tests for a service class 4.3/5 → 4.0/5 🔴 ✅ writing-mstest-tests; tools: skill, glob / ✅ writing-mstest-tests; tools: skill 🟡 0.26 [3]
writing-mstest-tests Write data-driven tests for a calculator 4.7/5 → 4.3/5 🔴 ✅ writing-mstest-tests; tools: skill 🟡 0.26 [4]
writing-mstest-tests Write async tests with cancellation 2.3/5 → 5.0/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.26
writing-mstest-tests Fix swapped Assert.AreEqual arguments 5.0/5 → 4.7/5 🔴 ✅ writing-mstest-tests; tools: report_intent, skill / ⚠️ NOT ACTIVATED 🟡 0.26 [5]
writing-mstest-tests Modernize legacy test patterns 4.7/5 → 4.3/5 🔴 ✅ writing-mstest-tests; tools: skill 🟡 0.26 [6]
writing-mstest-tests Replace ExpectedException with Assert.Throws 3.0/5 → 4.3/5 🟢 ✅ writing-mstest-tests; tools: skill, report_intent / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.26
writing-mstest-tests Use proper collection assertions 3.3/5 → 2.0/5 🔴 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.26 [7]
writing-mstest-tests Use proper type assertions instead of casts 2.7/5 → 3.0/5 🟢 ✅ writing-mstest-tests; tools: skill / ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.26 [8]
writing-mstest-tests Set up test lifecycle correctly 2.3/5 → 4.0/5 🟢 ✅ writing-mstest-tests; tools: skill 🟡 0.26 [9]
writing-mstest-tests Use DynamicData with ValueTuples over object arrays 2.0/5 → 5.0/5 🟢 ✅ writing-mstest-tests; tools: report_intent, skill 🟡 0.26

[1] ⚠️ High run-to-run variance (CV=2.75) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=1.15) — consider re-running with --runs 5
[3] ⚠️ High run-to-run variance (CV=1.68) — consider re-running with --runs 5
[4] ⚠️ High run-to-run variance (CV=4.98) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -2.7% due to: tokens (152894 → 422387), tool calls (17 → 24), time (98.6s → 136.8s)
[5] ⚠️ High run-to-run variance (CV=1.31) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -0.8% due to: tokens (12338 → 17261)
[6] ⚠️ High run-to-run variance (CV=1.24) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=1.97) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=3.24) — consider re-running with --runs 5
[9] ⚠️ High run-to-run variance (CV=0.64) — consider re-running with --runs 5

timeout — run(s) hit the (180s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions

@Evangelink
Evangelink merged commit 5826c42 into dotnet:main Apr 21, 2026
34 checks passed
@Evangelink
Evangelink deleted the dev/amauryleve/anti-patterns branch April 21, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants