Skip to content

Add exp-test-gap-analysis skill for pseudo-mutation test gap detection - #490

Merged
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/test-find-bugs
Apr 1, 2026
Merged

Add exp-test-gap-analysis skill for pseudo-mutation test gap detection#490
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/test-find-bugs

Conversation

@Evangelink

Copy link
Copy Markdown
Member

New experimental skill that performs pseudo-mutation analysis on .NET production code to find gaps in existing test suites. Covers boundary, boolean/logic, return value, exception removal, arithmetic, and null-check mutation categories.

Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps, well-tested code recognition, non-activation) and fixture projects.

New experimental skill that performs pseudo-mutation analysis on .NET
production code to find gaps in existing test suites. Covers boundary,
boolean/logic, return value, exception removal, arithmetic, and
null-check mutation categories.

Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps,
well-tested code recognition, non-activation) and fixture projects.
@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
exp-test-gap-analysis Find boundary mutation gaps in tiered discount and shipping logic 4.7/5 → 5.0/5 🟢 ✅ exp-test-gap-analysis; tools: skill, edit / ✅ exp-test-gap-analysis; tools: skill, glob ✅ 0.06
exp-test-gap-analysis Find logic and null-check mutation gaps in access control code 3.7/5 → 4.7/5 🟢 ✅ exp-test-gap-analysis; tools: skill, glob / ✅ exp-test-gap-analysis; tools: skill, edit ✅ 0.06
exp-test-gap-analysis Acknowledge well-tested code with few surviving mutations 3.3/5 → 3.3/5 ✅ exp-test-gap-analysis; tools: skill ✅ 0.06 [1]
exp-test-gap-analysis Decline request to write new tests from scratch 3.7/5 ⏰ → 4.3/5 🟢 ℹ️ not activated (expected) ✅ 0.06

[1] (Plugin) Quality dropped but weighted score is +2.7% due to: tokens (355377 → 113804), tool calls (26 → 10), time (219.2s → 154.3s)

timeout — run(s) hit the (120s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

📖 See docs/InvestigatingResults.md for how to diagnose failures. Additional debugging guidance may be provided by your workflow.

🔍 Full Results - additional metrics and failure investigation steps

@Evangelink
Evangelink marked this pull request as ready for review April 1, 2026 14:45
Copilot AI review requested due to automatic review settings April 1, 2026 14:45
@Evangelink
Evangelink enabled auto-merge (squash) April 1, 2026 14:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new experimental .NET skill (exp-test-gap-analysis) and accompanying evaluation scenarios/fixtures to perform pseudo-mutation style reasoning to identify test suite gaps.

Changes:

  • Introduces exp-test-gap-analysis skill documentation describing mutation-point detection and gap reporting.
  • Adds eval.yaml with 4 scenarios plus fixture projects covering boundary gaps, logic gaps, and a “well-tested” baseline.
  • Updates CODEOWNERS to route ownership for the new skill and tests.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.

File Description
plugins/dotnet-experimental/skills/exp-test-gap-analysis/SKILL.md Defines the new skill’s purpose, workflow, and mutation categories.
tests/dotnet-experimental/exp-test-gap-analysis/eval.yaml Adds evaluation scenarios validating boundary/logic gap detection and “well-tested” recognition.
tests/dotnet-experimental/exp-test-gap-analysis/fixtures/** Adds .NET fixture projects + tests used by the eval scenarios.
.github/CODEOWNERS Assigns ownership for the new skill + test area.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/dotnet-experimental/exp-test-gap-analysis/eval.yaml
@Evangelink
Evangelink merged commit eb0f98e into main Apr 1, 2026
35 checks passed
@Evangelink
Evangelink deleted the dev/amauryleve/test-find-bugs branch April 1, 2026 19:19
sayedihashimi pushed a commit to sayedihashimi/skills that referenced this pull request Apr 20, 2026
dotnet#490)

New experimental skill that performs pseudo-mutation analysis on .NET
production code to find gaps in existing test suites. Covers boundary,
boolean/logic, return value, exception removal, arithmetic, and
null-check mutation categories.

Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps,
well-tested code recognition, non-activation) and fixture projects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants