From 02574b835308c0880aec5eabc555258c41554977 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Tue, 19 May 2026 11:38:21 +0200 Subject: [PATCH] test-anti-patterns / test-smell-detection: activation-only rewrite (Variant A) Both skills suffer plugin-mode activation failures: in plugin runs the model bypasses the skill tool entirely for ~5/6 of test-anti-patterns audit prompts and ~2/3 of test-smell-detection prompts (verified via /evaluate result JSON), answering directly from training instead. Root cause: the previous descriptions carried mutual-exclusion clauses (each skill told the model 'use the other one for smell audits'), which actively trained the agent to bypass both. This change keeps both skills but: * drops every cross-skill mutual exclusion from both descriptions; * leads with 'INVOKE THIS SKILL when...' mandate phrasing; * expands USE-FOR triggers with the exact eval-prompt phrasing ('audit my tests', 'audit for .NET test anti-patterns', 'test smell audit', 'formal smell audit', 'rank by severity', etc.); * shrinks DO NOT USE FOR to real boundaries (writing/running/migrating); * pulls in the prompt tweaks from PR #653 so eval prompts use the new trigger phrasing. Variant A of a 3-PR experiment to fix activation. Companion PRs: * Variant B = narrow specialist for test-smell-detection; * Variant C = full merge into test-anti-patterns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../skills/test-anti-patterns/SKILL.md | 27 ++++++++++--------- .../skills/test-smell-detection/SKILL.md | 16 ++++++++++- .../test-smell-detection/eval.vally.yaml | 15 ++++++----- .../test-smell-detection/eval.yaml | 15 ++++++----- 4 files changed, 46 insertions(+), 27 deletions(-) diff --git a/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md b/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md index 762da91553..9fbafa266e 100644 --- a/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md +++ b/plugins/dotnet-test/skills/test-anti-patterns/SKILL.md @@ -1,18 +1,21 @@ --- name: test-anti-patterns description: > - Detection-focused review of .NET test code for anti-patterns that - undermine reliability and diagnostic value. - USE FOR: audit test quality, review test code, find test anti-patterns, - tests pass but don't verify anything, flaky tests, ordering dependency, - duplicate tests, magic values, missing/no assertions, swallowed - exceptions, always-true assertions, over-mocking, test coupling, coverage - touching, coverage inflation. - DO NOT USE FOR: writing new tests (use writing-mstest-tests), direct - MSTest API rewrites or implementation-only fixes such as swapped - Assert.AreEqual argument order, running tests (use run-tests), migrating - between frameworks (use migration skills), deep formal audit based on - academic test smell taxonomy (use test-smell-detection). + Audits existing .NET test code (MSTest, xUnit, NUnit, TUnit) for + anti-patterns and quality issues that undermine reliability and diagnostic + value — produces a severity-ranked report (Critical / Warning / Info) with + concrete code-level fixes and acknowledgement of what the tests do well. + INVOKE THIS SKILL when the user asks to audit, review, rank, or find + problems in existing tests — including prompts about: "audit my tests", + "audit for .NET test anti-patterns", "test smell audit", "rank by + severity", "are these tests good", tests that pass but verify nothing, + no/missing assertions, swallowed exceptions, always-true / self-comparing + / self-referential / tautological assertions, broad exception types, + flakiness (Thread.Sleep, DateTime.Now), ordering dependency, shared + static state, reflection coupling, duplicated tests, magic values, + coverage touching, coverage inflation. + DO NOT USE FOR: writing new tests (use writing-mstest-tests); running + tests (use run-tests); framework migration (use migration skills). license: MIT --- diff --git a/plugins/dotnet-test/skills/test-smell-detection/SKILL.md b/plugins/dotnet-test/skills/test-smell-detection/SKILL.md index 148f85f6ea..38a6213f00 100644 --- a/plugins/dotnet-test/skills/test-smell-detection/SKILL.md +++ b/plugins/dotnet-test/skills/test-smell-detection/SKILL.md @@ -1,6 +1,20 @@ --- name: test-smell-detection -description: "Deep formal test smell audit based on academic research taxonomy (testsmells.org). Detects 19 categorized smell types — conditional logic, mystery guests, sensitive equality, eager tests, and more — with calibrated severity and research-backed remediation. Use for comprehensive test suite health assessments. For a quick pragmatic review, use test-anti-patterns instead. DO NOT USE FOR: writing new tests (use writing-mstest-tests), evaluating assertion quality specifically (use assertion-quality), or finding test duplication and boilerplate (use exp-test-maintainability)." +description: > + Formal academic test-smell audit for existing .NET tests using the + testsmells.org 19-smell taxonomy. Each finding maps to a named, citable + smell from the research literature with calibrated severity, integration- + test exception rules, and concrete remediations — and the same rubric runs + every time so reviews are comparable across suites. + INVOKE THIS SKILL whenever the user asks for a "formal smell audit", a + "test smell audit", a "test-smell taxonomy review", an objective + severity-ranked assessment of test design, an academic / research-backed + smell catalog audit, or a test review explicitly framed around "smell" / + "smells" rather than ad-hoc problems. Works with MSTest, xUnit, NUnit, + TUnit. + DO NOT USE FOR: writing new tests from scratch (use writing-mstest-tests); + running or executing tests (use run-tests); migrating between frameworks or + test platform versions (use migration skills). license: MIT --- diff --git a/tests/dotnet-test/test-smell-detection/eval.vally.yaml b/tests/dotnet-test/test-smell-detection/eval.vally.yaml index dfe31dc442..0c006b2e5f 100644 --- a/tests/dotnet-test/test-smell-detection/eval.vally.yaml +++ b/tests/dotnet-test/test-smell-detection/eval.vally.yaml @@ -7,8 +7,8 @@ stimuli: - name: Detect multiple test smells in order processing test suite prompt: | I'm worried about the quality of our OrderProcessor test suite. - The tests all pass but some of them feel off — can you review them - and tell me if there are any problematic patterns? + The tests all pass but some of them feel off — can you do a test + smell audit and rank any findings by severity? environment: files: - src: fixtures/smelly-tests/OrderService.Tests/OrderService.Tests.csproj @@ -42,9 +42,9 @@ stimuli: - Provided concrete fix suggestions showing how to rewrite at least some of the smelly tests - name: Recognize well-written tests with no significant smells prompt: | - Can you check my Calculator tests for any test design problems or - anti-patterns? I want to make sure they follow best practices before - we use them as a template for other test suites. + Can you do a formal test smell audit of my Calculator tests? I want + to make sure they're clean before we use them as a template for + other test suites. environment: files: - src: fixtures/clean-tests/Calculator.Tests/Calculator.Tests.csproj @@ -67,8 +67,9 @@ stimuli: - name: Recognize integration tests and avoid false positives for external resources prompt: | Our team has been arguing about whether our integration tests are - well-designed. Some people say they have too many smells. Can you - review them and give us an objective assessment? + well-designed. Some people say they have too many test smells. + Can you do a formal smell audit and give us an objective + severity-ranked assessment? environment: files: - src: fixtures/integration-tests/DataAccess.IntegrationTests/DataAccess.IntegrationTests.csproj diff --git a/tests/dotnet-test/test-smell-detection/eval.yaml b/tests/dotnet-test/test-smell-detection/eval.yaml index dc765010b9..ada42077a0 100644 --- a/tests/dotnet-test/test-smell-detection/eval.yaml +++ b/tests/dotnet-test/test-smell-detection/eval.yaml @@ -6,8 +6,8 @@ scenarios: - name: "Detect multiple test smells in order processing test suite" prompt: | I'm worried about the quality of our OrderProcessor test suite. - The tests all pass but some of them feel off — can you review them - and tell me if there are any problematic patterns? + The tests all pass but some of them feel off — can you do a test + smell audit and rank any findings by severity? setup: files: - path: "OrderService.Tests/OrderService.Tests.csproj" @@ -38,9 +38,9 @@ scenarios: - name: "Recognize well-written tests with no significant smells" prompt: | - Can you check my Calculator tests for any test design problems or - anti-patterns? I want to make sure they follow best practices before - we use them as a template for other test suites. + Can you do a formal test smell audit of my Calculator tests? I want + to make sure they're clean before we use them as a template for + other test suites. setup: files: - path: "Calculator.Tests/Calculator.Tests.csproj" @@ -66,8 +66,9 @@ scenarios: - name: "Recognize integration tests and avoid false positives for external resources" prompt: | Our team has been arguing about whether our integration tests are - well-designed. Some people say they have too many smells. Can you - review them and give us an objective assessment? + well-designed. Some people say they have too many test smells. + Can you do a formal smell audit and give us an objective + severity-ranked assessment? setup: files: - path: "DataAccess.IntegrationTests/DataAccess.IntegrationTests.csproj"