diff --git a/plugins/dotnet-test/skills/find-untested-sources/SKILL.md b/plugins/dotnet-test/skills/find-untested-sources/SKILL.md index 14cd1420d1..237ca19d29 100644 --- a/plugins/dotnet-test/skills/find-untested-sources/SKILL.md +++ b/plugins/dotnet-test/skills/find-untested-sources/SKILL.md @@ -10,6 +10,11 @@ description: > DO NOT USE FOR: line/branch coverage or CRAP risk (use coverage-analysis); whether existing tests are strong (use test-gap-analysis or assertion-quality). license: MIT +# Agent-orchestrated helper (invoked by name from code-testing-researcher and the +# code-testing pipeline); kept out of the model-facing skill menu so it does not +# consume the plugin's 15,000-char skill-menu budget or add routing noise that +# suppresses activation of the user-facing test skills. Still invocable by name. +disable-model-invocation: true --- # Find Untested Sources diff --git a/tests/dotnet-test/coverage-analysis/eval.yaml b/tests/dotnet-test/coverage-analysis/eval.yaml index 7ace7e158e..2a4b1b88de 100644 --- a/tests/dotnet-test/coverage-analysis/eval.yaml +++ b/tests/dotnet-test/coverage-analysis/eval.yaml @@ -29,7 +29,7 @@ scenarios: - "Identifies CalculateGpa as the highest-risk method (high complexity, low coverage)" - "Provides actionable recommendations for the riskiest methods" - "Generates or displays a coverage analysis report" - timeout: 300 + timeout: 480 - name: "Run coverage from scratch without existing data" prompt: "Analyze code coverage and risk hotspots for this project. I haven't run any coverage yet." @@ -88,4 +88,4 @@ scenarios: - "Quantifies the impact — covering CalculateGpa would raise overall coverage significantly" - "Uses CRAP scores to distinguish the risky gap (CalculateGpa) from trivial ones" - "Provides specific test recommendations for the uncovered method" - timeout: 300 + timeout: 480 diff --git a/tests/dotnet-test/run-tests/eval.yaml b/tests/dotnet-test/run-tests/eval.yaml index 21ae5f59e5..04481ea1ac 100644 --- a/tests/dotnet-test/run-tests/eval.yaml +++ b/tests/dotnet-test/run-tests/eval.yaml @@ -73,7 +73,7 @@ scenarios: timeout: 420 - name: "Run tests on a specific TFM with TRX in a multi-TFM MTP project (SDK 9)" - prompt: "This MSTest project uses Microsoft.Testing.Platform and targets both net8.0 and net9.0. I'm on .NET SDK 9. I want to run only the net9.0 tests and produce a TRX report. Show me the exact command." + prompt: "This MSTest project uses Microsoft.Testing.Platform and targets both net8.0 and net9.0. I'm on .NET SDK 9. I want to run only the net9.0 tests and produce a TRX report saved as net9-results.trx. Show me the exact command." setup: files: - path: "TestProject.csproj" @@ -85,12 +85,14 @@ scenarios: assertions: - type: "output_matches" pattern: "dotnet test[^\\r\\n]*--framework\\s+net9\\.0[^\\r\\n]*--\\s+--report-trx" + - type: "output_matches" + pattern: "--report-trx-filename\\s+\"?net9-results\\.trx" - type: "exit_success" rubric: - "Runs only the net9.0 target framework rather than both TFMs" - "Identifies that the project uses MTP (TestingPlatformDotnetTestSupport) with SDK 9 argument-separator behavior" - "Keeps '--framework net9.0' before the '--' separator (built-in dotnet test flag)" - - "Passes '--report-trx' after the '--' separator (MTP extension flag, required on SDK 9)" + - "Passes BOTH MTP extension flags after the single '--' separator: '--report-trx' and '--report-trx-filename net9-results.trx' — it does not split them across the separator or drop the '--report-trx-filename' extension" - "Avoids VSTest-style TRX syntax such as '--logger trx'" expect_tools: ["bash"] timeout: 480 @@ -100,7 +102,7 @@ scenarios: # ============================================================================ - name: "Filter MSTest tests by category on VSTest" - prompt: "I only want to run the integration tests using dotnet test. How do I filter by the Integration category?" + prompt: "I only want to run the Integration tests using dotnet test, but I need to exclude the ones in the PaymentServiceTests class. Show me the exact command." setup: additional_required_skills: ["filter-syntax"] files: @@ -113,12 +115,16 @@ scenarios: pattern: "--filter" - type: "output_matches" pattern: "TestCategory=Integration" + - type: "output_contains" + value: "&" + - type: "output_matches" + pattern: "(?:FullyQualifiedName|ClassName|Name)\\s*(?:!~|!=)[^\"&|]*PaymentService" - type: "exit_success" rubric: - - "Uses the VSTest --filter flag with the TestCategory property" - - "Uses the correct syntax: --filter \"TestCategory=Integration\"" - - "Does not use MTP-style filter flags like --filter-trait" - - "Does not suggest xUnit-style Trait filtering for an MSTest project" + - "Uses the VSTest --filter flag and keeps TestCategory=Integration to select the integration tests" + - "Combines the category filter with a class exclusion using the '&' (AND) combinator" + - "Excludes PaymentServiceTests with a negation operator — '!~' (does-not-contain) or '!=' (not-equals) against FullyQualifiedName/ClassName — rather than a positive match" + - "Does not run all tests and post-filter, and does not use MTP-only flags like --filter-not-class for this VSTest project" expect_tools: ["bash"] timeout: 120 @@ -193,7 +199,7 @@ scenarios: - "xUnit v3 on MTP requires framework-specific filter flags, not VSTest expression syntax" - "Correctly handles the '--' separator for SDK 9" expect_tools: ["bash"] - timeout: 120 + timeout: 240 - name: "Filter xUnit v3 tests by class pattern and trait using query filter language" prompt: "I have an xUnit v3 project with several test classes. I need to run only the Smoke tests from classes whose name contains 'Integration' — I want a single combined filter expression for dotnet test rather than separate flags." @@ -219,7 +225,7 @@ scenarios: - "The expression correctly targets only Smoke tests from integration test classes, not all Smoke tests or all integration tests" - "Used the path-segment syntax with wildcard segments and a trait qualifier like [Category=Smoke]" expect_tools: ["bash"] - timeout: 120 + timeout: 240 - name: "Filter TUnit tests by class using treenode-filter" prompt: "I want to run only the SmsNotificationTests from this TUnit project using dotnet test. What's the command?" diff --git a/tests/dotnet-test/test-gap-analysis/eval.yaml b/tests/dotnet-test/test-gap-analysis/eval.yaml index 440375f95d..7899b6ec36 100644 --- a/tests/dotnet-test/test-gap-analysis/eval.yaml +++ b/tests/dotnet-test/test-gap-analysis/eval.yaml @@ -103,7 +103,7 @@ scenarios: - "Acknowledged that both zero and negative quantities are tested for guard clauses" - "Acknowledged that RemoveStock verifies both the return value and the resulting stock level" - "If any gaps were found, they were presented as minor improvements rather than critical weaknesses" - timeout: 300 + timeout: 420 # ========================================================================== # Scenario 4: Non-activation — user wants to write new tests @@ -140,7 +140,7 @@ scenarios: rubric: - "Wrote test methods for the ShoppingCart class" - "Covered the AddItem and GetTotal methods" - timeout: 120 + timeout: 300 # ========================================================================== # Scenario 5: Report quality — exclude trivial/generated code, trace call diff --git a/tests/dotnet-test/writing-mstest-tests/eval.yaml b/tests/dotnet-test/writing-mstest-tests/eval.yaml index 35bb69ce41..d4eee87b5f 100644 --- a/tests/dotnet-test/writing-mstest-tests/eval.yaml +++ b/tests/dotnet-test/writing-mstest-tests/eval.yaml @@ -24,7 +24,7 @@ scenarios: - "Tests both success and failure paths (e.g., null input, empty items)" - "Uses Assert.ThrowsExactly or Assert.Throws for exception testing instead of [ExpectedException]" - "Uses Assert.AreEqual with expected as first argument and actual as second" - timeout: 180 + timeout: 360 - name: "Write data-driven tests for a calculator" prompt: "I have a PriceCalculator class with CalculateTotal and ApplyDiscount methods. Help me write data-driven MSTest tests using DataRow and DynamicData that cover multiple input combinations following modern MSTest best practices." @@ -44,7 +44,7 @@ scenarios: - "Seals the test class" - "Uses Assert.ThrowsExactly or Assert.Throws for testing invalid argument scenarios" - "Does not use manual foreach loops to iterate test data — uses proper data-driven attributes instead" - timeout: 180 + timeout: 360 - name: "Write async tests with cancellation" prompt: | @@ -90,7 +90,7 @@ scenarios: - name: "Fix swapped Assert.AreEqual arguments" prompt: | - Something seems off with my tests — I think I need to fix swapped Assert.AreEqual arguments. Can you review and fix the MSTest assertions? + When these MSTest tests fail, the failure messages look backwards — the value the code actually produced is reported as "Expected", and the literal I wrote is reported as "Actual". Review these assertions and fix whatever is causing that. Don't touch anything that is already correct. ```csharp [TestClass] @@ -109,18 +109,28 @@ scenarios: var result = Calculator.Multiply(42, 0); Assert.AreEqual(result, 0); } + + [TestMethod] + public void Subtract_Numbers_ReturnsDifference() + { + var result = Calculator.Subtract(10, 4); + Assert.AreEqual(6, result); + } } ``` reject_tools: ["bash", "edit", "create"] assertions: - type: output_matches - pattern: "Assert\\.AreEqual\\(5" + pattern: "Assert\\.AreEqual\\(5,\\s*result" - type: output_matches - pattern: "Assert\\.AreEqual\\(0" + pattern: "Assert\\.AreEqual\\(0,\\s*result" + - type: output_not_matches + pattern: "Assert\\.AreEqual\\(result,\\s*6\\)" rubric: - - "Identifies the swapped arguments in Assert.AreEqual — expected should be first, actual second" - - "Fixes to Assert.AreEqual(5, result) and Assert.AreEqual(0, result)" - - "Explains why argument order matters (failure messages show 'Expected: X, Actual: Y')" + - "Diagnoses the root cause unaided (the prompt does not name it): the first two assertions pass arguments in the wrong order — MSTest's signature is `Assert.AreEqual(expected, actual)`, so the literal must come first" + - "Fixes the two swapped assertions to `Assert.AreEqual(5, result)` and `Assert.AreEqual(0, result)`" + - "Leaves the already-correct `Assert.AreEqual(6, result)` in Subtract_Numbers_ReturnsDifference unchanged — does not swap a correct assertion" + - "Explains that argument order drives the 'Expected: X, Actual: Y' failure message, and may reference the MSTEST0017 analyzer that flags this" timeout: 120 - name: "Modernize legacy test patterns" @@ -205,9 +215,9 @@ scenarios: - type: output_matches pattern: "ContainsSingle|Assert\\.IsNotNull" rubric: - - "The non-empty validation no longer relies on a generic Assert.IsTrue boolean check — it uses a dedicated assertion that clearly reports 'collection was empty' on failure" - - "The exact-count validation uses a count-specific or equality assertion rather than wrapping the comparison in Assert.IsTrue" - - "The single-element lookup uses an assertion that produces a meaningful failure message when zero or multiple elements match, rather than throwing an unrelated exception" + - "The non-empty validation no longer relies on a generic Assert.IsTrue boolean check — it uses a dedicated assertion such as the real MSTest 3.8+ `Assert.IsNotEmpty(collection)` (or `Assert.AreNotEqual(0, x.Count)`) that clearly reports 'collection was empty' on failure. Grading note: `Assert.IsNotEmpty`, `Assert.IsEmpty`, `Assert.HasCount`, `Assert.ContainsSingle`, `Assert.Contains`, and `Assert.DoesNotContain` are real assertion methods introduced in MSTest 3.8+ (Microsoft.Testing) — do not penalize them as non-existent or fabricated." + - "The exact-count validation uses a count-specific assertion such as the real MSTest 3.8+ `Assert.HasCount(3, collection)` (or `Assert.AreEqual(3, x.Count)`) rather than wrapping the comparison in Assert.IsTrue" + - "The single-element lookup uses an assertion that produces a meaningful failure message when zero or multiple elements match — such as the real MSTest 3.8+ `Assert.ContainsSingle(...)` — rather than `Single()` followed by Assert.IsTrue, which throws an unrelated exception" - "The null check uses a null-specific assertion rather than Assert.IsTrue with a boolean null comparison" - "Explains why specialized assertions produce better failure diagnostics than generic Assert.IsTrue" timeout: 120 @@ -348,7 +358,7 @@ scenarios: - "Tests BuildGreeting validates the output starts with 'Dear' and ends with a comma" - "Tests GenerateSlug validates the slug pattern contains only lowercase letters, digits, and hyphens" - "Tests exception handling for null or invalid inputs" - timeout: 180 + timeout: 360 # ============================================================================ # Goal 6: Comparison assertions @@ -400,7 +410,7 @@ scenarios: - "Tests boundary conditions at grade thresholds (e.g., exactly 90 vs 89)" - "Tests that NormalizeScore caps the result at 100 when rawScore exceeds maxRaw" - "Seals the test class" - timeout: 120 + timeout: 240 # ============================================================================ # Goal 7: Collection, null, and reference assertions