Add exp-test-smell-detection skill and evaluation

[Evangelink]

Adds a new experimental skill that detects test smells — bad programming practices in test code that reduce effectiveness and increase maintenance cost.

Skill design

The skill is structured in three layers:

• SKILL.md — Language-agnostic core detecting 10 smells organized by severity:
  • High: Conditional Test Logic, Mystery Guest, Sleepy Test, Assertion-Free Test
  • Medium: Eager Test, Magic Number Test, Sensitive Equality, Exception Handling in Tests
  • Low: General Fixture (Over-broad Setup), Ignored/Disabled Test
• extensions/dotnet.md — .NET-specific detection tables (assertion APIs for MSTest/xUnit/NUnit/TUnit, sleep patterns, skip annotations, exception handling idioms, integration test markers, setup/teardown methods)
• references/test-smell-catalog.md — Extended catalog of all 19 smells from testsmells.org research with real open-source examples, academic references, and prevalence data

Key calibration rules prevent false positives: integration tests aren't penalized for database usage, foreach-assert loops aren't flagged as conditional logic, contextually obvious count assertions aren't flagged as magic numbers.

Evaluation

4 scenarios with MSTest 4.1.0 fixtures:

Scenario	What it tests
Smelly suite	OrderProcessorTests.cs with 8+ distinct smells to detect
Clean suite	ScientificCalculatorTests.cs — must acknowledge good quality without inventing problems
Integration tests	Mix of legitimate integration patterns and real smells (Sleep, conditional logic, assertion-free) — must distinguish between them
Non-activation	Request to write new tests — skill should not activate

Based on

Smell taxonomy from testsmells.org (Peruma et al., ESEC/FSE 2020), severity thresholds from Spadini et al. (MSR 2020), and testing philosophy from Google's SWE Book (Chapter 11).

[Evangelink]

/evaluate

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
exp-test-smell-detection	Detect multiple test smells in order processing test suite	4.3/5 → 5.0/5 🟢	✅ exp-test-smell-detection; tools: skill	✅ 0.05	✅ [1]
exp-test-smell-detection	Recognize well-written tests with no significant smells	3.3/5 → 4.7/5 🟢	✅ exp-test-smell-detection; tools: skill	✅ 0.05	✅
exp-test-smell-detection	Recognize integration tests and avoid false positives for external resources	5.0/5 → 5.0/5	✅ exp-test-smell-detection; tools: skill	✅ 0.05	❌ [2]
exp-test-smell-detection	Decline request to write new tests from scratch	5.0/5 → 4.3/5 🔴	ℹ️ not activated (expected)	✅ 0.05	❌ [3]

[1] ⚠️ High run-to-run variance (CV=1.11) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=2.07) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -9.9% due to: tokens (43104 → 84985), tool calls (4 → 8), time (38.5s → 92.0s)
[3] ⚠️ High run-to-run variance (CV=5.05) — consider re-running with --runs 5

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

[Evangelink]

/evaluate

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
exp-test-smell-detection	Detect multiple test smells in order processing test suite	4.3/5 → 5.0/5 🟢	✅ exp-test-smell-detection; tools: skill	✅ 0.04	✅ [1]
exp-test-smell-detection	Recognize well-written tests with no significant smells	3.0/5 → 5.0/5 🟢	✅ exp-test-smell-detection; tools: skill	✅ 0.04	✅
exp-test-smell-detection	Recognize integration tests and avoid false positives for external resources	5.0/5 → 5.0/5	✅ exp-test-smell-detection; tools: skill	✅ 0.04	❌ [2]
exp-test-smell-detection	Decline request to write new tests from scratch	5.0/5 → 5.0/5	ℹ️ not activated (expected)	✅ 0.04	❌ [3]

[1] ⚠️ High run-to-run variance (CV=1.05) — consider re-running with --runs 5
[2] (Plugin) Quality unchanged but weighted score is -9.9% due to: tokens (38427 → 76016), tool calls (4 → 8), time (32.4s → 80.0s)
[3] ⚠️ High run-to-run variance (CV=1.19) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -27.0% due to: quality, judgment

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

[Copilot]

Pull request overview

Adds a new dotnet-experimental skill (exp-test-smell-detection) with supporting .NET-specific guidance, reference material, and a dedicated evaluation suite + fixtures to validate smell detection vs. false positives (including integration-test calibration and non-activation behavior).

Changes:

• Introduces the new skill definition (SKILL.md) plus a .NET detection extension and an extended reference catalog.
• Adds evaluation scenarios (eval.yaml) and MSTest fixture projects covering “smelly”, “clean”, “integration”, and non-activation prompts.
• Updates repo plumbing for the new content (CODEOWNERS + known-domains entry for testsmells.org).

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/dotnet-experimental/exp-test-smell-detection/fixtures/smelly-tests/OrderService.Tests/OrderService.Tests.csproj	Adds MSTest project fixture for the “smelly suite” scenario.
tests/dotnet-experimental/exp-test-smell-detection/fixtures/smelly-tests/OrderService.Tests/OrderProcessorTests.cs	Adds intentionally smelly unit tests used to validate smell detection outputs.
tests/dotnet-experimental/exp-test-smell-detection/fixtures/integration-tests/DataAccess.IntegrationTests/DataAccess.IntegrationTests.csproj	Adds MSTest project fixture for integration-test calibration scenario.
tests/dotnet-experimental/exp-test-smell-detection/fixtures/integration-tests/DataAccess.IntegrationTests/UserRepositoryIntegrationTests.cs	Adds integration-test fixture mixing legitimate patterns with intentional smells.
tests/dotnet-experimental/exp-test-smell-detection/fixtures/clean-tests/Calculator.Tests/Calculator.Tests.csproj	Adds MSTest project fixture for the “clean suite” scenario.
tests/dotnet-experimental/exp-test-smell-detection/fixtures/clean-tests/Calculator.Tests/ScientificCalculatorTests.cs	Adds a clean test suite fixture to ensure the skill doesn’t invent smells.
tests/dotnet-experimental/exp-test-smell-detection/eval.yaml	Adds 4 evaluation scenarios to validate activation and correct smell detection/calibration.
plugins/dotnet-experimental/skills/exp-test-smell-detection/SKILL.md	Defines the smell taxonomy, workflow, calibration rules, and reporting structure.
plugins/dotnet-experimental/skills/exp-test-smell-detection/extensions/dotnet.md	Documents .NET framework markers and detection patterns used by the skill.
plugins/dotnet-experimental/skills/exp-test-smell-detection/references/test-smell-catalog.md	Adds extended smell catalog/reference background beyond the core 10 smells.
eng/known-domains.txt	Allows linking to testsmells.org in skill/reference content.
.github/CODEOWNERS	Adds ownership entries for the new skill + its tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.