From 45e8ab96859561d71cf61931abb2b890fdf9d954 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 23 Apr 2026 13:47:23 +0200 Subject: [PATCH 1/2] Fix dotnet-test-frameworks skill activation for evaluations The skill had user-invocable: false and a description saying 'DO NOT USE directly', which prevented the model from activating it during evaluation. All 6 eval scenarios default to expect_activation: true, causing SkillNotActivated failures on CI. - Remove user-invocable: false from both dotnet-test and experimental copies - Rewrite description with activation keywords matching eval prompts (assertion mapping, framework identification, skip annotations, exception assertions, lifecycle conversion, integration test markers) - Preserve note that other analysis skills also load this as reference data --- .../skills/exp-dotnet-test-frameworks/SKILL.md | 3 +-- plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md b/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md index a8f0cb4004..0d0d7bb8ec 100644 --- a/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md +++ b/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md @@ -1,7 +1,6 @@ --- name: exp-dotnet-test-frameworks -description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. DO NOT USE directly — loaded by test analysis skills (exp-test-smell-detection, exp-assertion-quality, exp-test-maintainability, exp-test-tagging) when they need framework-specific lookup tables." -user-invocable: false +description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. Use when identifying test frameworks, mapping assertion equivalents across MSTest/xUnit/NUnit/TUnit, converting lifecycle or setup/teardown methods between frameworks, looking up skip/ignore attributes, replacing try-catch with framework-native exception assertions, or identifying integration test markers and code patterns. Also loaded by test analysis skills (exp-test-smell-detection, exp-assertion-quality, exp-test-maintainability, exp-test-tagging) as framework-specific lookup tables." license: MIT --- diff --git a/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md b/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md index 8eaa2e47a5..a1754b1bda 100644 --- a/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md +++ b/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md @@ -1,7 +1,6 @@ --- name: dotnet-test-frameworks -description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. DO NOT USE directly — loaded by test analysis skills (test-anti-patterns, exp-test-smell-detection, exp-assertion-quality, exp-test-maintainability, exp-test-tagging) when they need framework-specific lookup tables." -user-invocable: false +description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. Use when identifying test frameworks, mapping assertion equivalents across MSTest/xUnit/NUnit/TUnit, converting lifecycle or setup/teardown methods between frameworks, looking up skip/ignore attributes, replacing try-catch with framework-native exception assertions, or identifying integration test markers and code patterns. Also loaded by test analysis skills (test-anti-patterns, exp-test-smell-detection, exp-assertion-quality, exp-test-maintainability, exp-test-tagging) as framework-specific lookup tables." license: MIT --- From 7968621d564ec2dad23f0cf222f7b1b8bae67b97 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 23 Apr 2026 14:24:39 +0200 Subject: [PATCH 2/2] Restore user-invocable: false, fix cross-refs, add expect_activation: false Address review comment: remove exp-* skill references from dotnet-test- frameworks description since those skills reference exp-dotnet-test- frameworks instead. Optimize eval results: this reference data skill's content is knowledge the model already has (baseline scores 5.0/5 on most scenarios). Direct activation adds token/time overhead without quality improvement: - Restore user-invocable: false (correct for reference data skill) - Shorten description to not trigger activation - Add expect_activation: false to all 6 scenarios The skill's value is realized when loaded by consumer skills (e.g. test-anti-patterns), not via direct user queries. --- .../skills/exp-dotnet-test-frameworks/SKILL.md | 3 ++- plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md | 3 ++- tests/dotnet-test/dotnet-test-frameworks/eval.yaml | 6 ++++++ 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md b/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md index 0d0d7bb8ec..fe4b2dde0f 100644 --- a/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md +++ b/plugins/dotnet-experimental/skills/exp-dotnet-test-frameworks/SKILL.md @@ -1,6 +1,7 @@ --- name: exp-dotnet-test-frameworks -description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. Use when identifying test frameworks, mapping assertion equivalents across MSTest/xUnit/NUnit/TUnit, converting lifecycle or setup/teardown methods between frameworks, looking up skip/ignore attributes, replacing try-catch with framework-native exception assertions, or identifying integration test markers and code patterns. Also loaded by test analysis skills (exp-test-smell-detection, exp-assertion-quality, exp-test-maintainability, exp-test-tagging) as framework-specific lookup tables." +description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. Loaded by test analysis skills (exp-test-smell-detection, exp-assertion-quality, exp-test-maintainability, exp-test-tagging) as framework-specific lookup tables." +user-invocable: false license: MIT --- diff --git a/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md b/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md index a1754b1bda..cfde1b7dce 100644 --- a/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md +++ b/plugins/dotnet-test/skills/dotnet-test-frameworks/SKILL.md @@ -1,6 +1,7 @@ --- name: dotnet-test-frameworks -description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. Use when identifying test frameworks, mapping assertion equivalents across MSTest/xUnit/NUnit/TUnit, converting lifecycle or setup/teardown methods between frameworks, looking up skip/ignore attributes, replacing try-catch with framework-native exception assertions, or identifying integration test markers and code patterns. Also loaded by test analysis skills (test-anti-patterns, exp-test-smell-detection, exp-assertion-quality, exp-test-maintainability, exp-test-tagging) as framework-specific lookup tables." +description: "Reference data for .NET test framework detection patterns, assertion APIs, skip annotations, setup/teardown methods, and common test smell indicators across MSTest, xUnit, NUnit, and TUnit. Loaded by test analysis skills (test-anti-patterns) as framework-specific lookup tables." +user-invocable: false license: MIT --- diff --git a/tests/dotnet-test/dotnet-test-frameworks/eval.yaml b/tests/dotnet-test/dotnet-test-frameworks/eval.yaml index d663cf58c0..a60d44bcbd 100644 --- a/tests/dotnet-test/dotnet-test-frameworks/eval.yaml +++ b/tests/dotnet-test/dotnet-test-frameworks/eval.yaml @@ -1,5 +1,6 @@ scenarios: - name: "Cross-framework assertion equivalence mapping" + expect_activation: false prompt: | I have these MSTest assertions in my test file and I need to know their equivalents in both xUnit and NUnit. Please provide a complete mapping for all six: @@ -40,6 +41,7 @@ scenarios: timeout: 120 - name: "Identify TUnit framework and its unique attributes" + expect_activation: false prompt: | I inherited a project with this test file and I'm not sure what testing framework it uses. Can you identify it and explain the key attributes? @@ -89,6 +91,7 @@ scenarios: timeout: 120 - name: "Replace try-catch with framework-native exception assertions" + expect_activation: false prompt: | My team lead says these tests should use proper exception assertions instead of try/catch. Can you refactor each one using its framework's native approach? @@ -163,6 +166,7 @@ scenarios: timeout: 120 - name: "Skip annotations across all four frameworks" + expect_activation: false prompt: | I maintain a solution with test projects using MSTest, xUnit, NUnit, and TUnit. I need to temporarily skip some tests in each project with a reason message explaining why. @@ -192,6 +196,7 @@ scenarios: timeout: 120 - name: "Convert NUnit lifecycle methods to xUnit equivalents" + expect_activation: false prompt: | I'm migrating this NUnit test class to xUnit. How should I convert the setup and teardown methods? @@ -252,6 +257,7 @@ scenarios: timeout: 120 - name: "Identify integration tests by markers and code patterns" + expect_activation: false prompt: | I have a test solution with hundreds of tests. I need to separate integration tests from unit tests for CI pipeline optimization. Here are samples from three projects — which ones